The build passes, tests are green, it runs perfectly on every developer machine, and it fails immediately in production. The code is identical, so the difference is environmental. There is a short list of things that reliably differ, and checking them systematically is faster than reading the application code again.
01Configuration and secrets differ silently
Local development typically reads a dotenv file that has accumulated values over months. Production reads from a secret manager or orchestrator configuration. A variable present locally and absent in production produces either an immediate crash or, worse, a fallback to a default that appears to work until it corrupts something.
Validate configuration at startup and fail loudly. A process that refuses to start with a clear message naming the missing variable is far cheaper to diagnose than one that starts successfully and behaves incorrectly under specific conditions hours later.
| Difference | Symptom | Check |
|---|---|---|
| Missing environment variable | Crash at startup or wrong behaviour | Validate config on boot |
| Memory limit in the orchestrator | Container killed, exit code 137 | Compare limit to actual usage |
| Read-only file system | Write failures at runtime | Use a mounted volume for temp files |
| Case-sensitive file system | Module not found on Linux only | Match import casing exactly |
| Network policy or egress rules | Timeouts to external services | Test connectivity from inside the pod |
| UTC versus local time zone | Off-by-hours date logic | Store and compute in UTC |
02Resource limits do not exist on a laptop
A development machine has many gigabytes of memory and no enforced ceiling. A container has whatever limit was set, and exceeding it means the process is terminated by the kernel rather than receiving an error it can handle. Exit code 137 is the signature and it is frequently misread as an application crash.
Some runtimes compound this by sizing their heap from the host rather than the container limit, so the process happily allocates beyond what it is permitted before being killed. Setting the runtime's memory ceiling explicitly, below the container limit, resolves a category of failures that look random.
03Case sensitivity catches almost everyone once
macOS file systems are case-insensitive by default; production Linux containers are not. An import referring to a file with different capitalisation than the file on disk resolves locally and fails in the container with a module-not-found error that looks inexplicable given the file plainly exists.
The same class of problem appears with paths assembled from configuration, and with anything relying on directory ordering. Building in a Linux container locally removes the entire category rather than fixing instances of it.
04Test the artefact you deploy
Running the application with a development server and deploying a compiled container image means the thing tested is not the thing shipped. Build the image, run that image locally with production-like configuration, and the gap narrows to genuine infrastructure differences.
Compose the dependencies too. If production has a database, a cache and a message broker, running against those locally in containers surfaces connection, migration and startup-ordering problems before they reach an environment where diagnosis is slower and more visible.
| Practice | Removes |
|---|---|
| Run the same container image locally | Build and runtime differences |
| Validate configuration at startup | Silent missing-variable failures |
| Set explicit resource limits locally | Out-of-memory surprises |
| Use UTC everywhere | Time zone arithmetic bugs |
| Deploy to staging identical to production | Orchestrator and network policy issues |
05Make the failure legible
Much of the time lost is not spent fixing the problem but working out what the problem is. Structured logs that include the configuration values in use at startup, clear startup validation, and health checks that verify dependencies rather than returning a static response, all shorten that phase considerably.
The aim is not perfect parity, which is unattainable and not worth chasing. It is that the remaining differences are known, documented and few enough that when something fails only in production, the list of candidates is short.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer