The invoice arrives and it is roughly twice what it was. No major launch happened, headcount is flat, and traffic is up slightly but nowhere near double. This is a common pattern, and the cause is almost never a single expensive mistake. It is several small increases compounding, in categories the default console view does not make visible.
01Group the bill by usage type, not by service
The service-level view tells you EC2 costs more than it did, which is not actionable. Grouping by usage type separates compute hours from data transfer from storage operations, and that distinction usually identifies the cause immediately.
Work at daily granularity rather than monthly. A step change on a specific date points at a deployment or configuration change you can locate in version control; a gradual slope points at organic growth in storage or logging, which is a different problem with a different fix.
| Category | Why it grows | Typical fix |
|---|---|---|
| Cross-AZ data transfer | Chatty services in different zones | Co-locate or batch calls |
| NAT gateway processing | All egress routed through NAT | VPC endpoints for AWS services |
| CloudWatch logs ingestion | Debug logging left enabled | Sampling and retention limits |
| EBS snapshots | Automated, never expired | Lifecycle policy |
| Idle load balancers | Environments never torn down | Scheduled cleanup |
02Data transfer is the usual answer
Traffic between availability zones is billed in both directions. A service split across zones that makes a call per request generates continuous charges that scale with request volume, and nothing in the architecture diagram suggests a cost line. Teams find this after months because it never appears as a discrete item they decided on.
NAT gateways compound it. Every byte leaving a private subnet through a NAT gateway is charged for processing on top of transfer, so pulling container images or calling S3 without a VPC endpoint means paying twice for traffic that never leaves the region.
03Storage only ever grows
Nobody deletes data. Snapshots accumulate on daily schedules with no expiry, log groups default to retaining forever, and object storage fills with build artefacts nobody will read again. Each is individually trivial and collectively significant after a year.
Set retention at creation time. A log group created without a retention policy will still be storing debug output from two years ago, billed monthly, and the person who would have deleted it has usually left.
| Resource | Default | Reasonable setting |
|---|---|---|
| CloudWatch log group | Never expires | 14-30 days for application logs |
| EBS snapshot schedule | Retained indefinitely | 7-30 days with lifecycle rules |
| S3 build artefacts | Standard storage forever | Transition then expire |
| Old AMIs | Retained with backing snapshots | Keep last few generations |
04Non-production is often the larger half
Staging, QA and demo environments frequently run the same instance sizes as production and stay up around the clock while being used for roughly a quarter of the week. Scheduling them off outside working hours removes most of that cost with no engineering risk.
Environments created for a specific project and never destroyed are the other half. An ownership tag applied at creation, with anything untagged reviewed monthly, prevents the slow accumulation that makes the bill inexplicable a year later.
05Make it visible so it stops recurring
Cost anomaly detection with alerting on percentage change catches step increases within days rather than at invoice time. Combined with mandatory ownership tags, this converts an annual surprise into a routine notification somebody acts on while the cause is still obvious.
The organisational fix matters more than any single optimisation. When no team sees the cost of the infrastructure it provisions, the bill grows structurally, and a one-off cleanup buys perhaps six months before the same pattern reasserts itself.
Topics
Marcus Hale
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer