One account reports errors consistently. Every other customer is fine, the same code path works in testing, and nothing about the deployment is specific to them. Something about that tenant differs - almost always their data, their scale, or their configuration - and identifying which category it falls into narrows the search dramatically.
01Attach tenant identity to everything
Errors that cannot be attributed to an account are much harder to diagnose. Every log line, error report and trace should carry a tenant identifier so you can filter to that customer's activity and see the pattern rather than isolated events.
This also reveals scope quickly. Filtering by tenant shows whether the problem affects one user within the account or all of them, and whether it started at a specific time that correlates with a deployment or a change on their side.
| Category | Typical cause | Signature |
|---|---|---|
| Data shape | Unicode, very long values, nulls, unusual formats | Fails on specific records |
| Scale | Far more rows than any other tenant | Timeouts, memory exhaustion |
| Configuration | Feature flags, integrations, locale, time zone | Fails consistently for that tenant |
02Data shape is the most common cause
Assumptions about data hold until a customer violates them. Names containing characters outside the basic Latin set, addresses far longer than any test fixture, emoji in free text fields, empty strings where the code expects null, or dates before an epoch boundary all produce failures confined to whoever has that data.
Query for the extremes rather than reading records at random - the longest values, the earliest and latest dates, rows containing characters outside the expected range. The failing record is usually at one of those boundaries, and finding it takes minutes with the right query.
03Scale differences are rarely linear
A tenant with a hundred times more records than the median is not merely slower. They cross thresholds nobody else reaches: a query that was acceptable becomes a timeout, a list loaded entirely into memory exhausts the container, a report that ran in seconds takes minutes and hits a gateway limit.
Check the row counts per tenant for your main tables. If the affected account is at the top of that distribution by a wide margin, the problem is almost certainly a scale threshold and the fix is pagination, indexing or streaming rather than anything to do with correctness.
| Step | Reveals |
|---|---|
| Filter logs and traces by tenant | Timing, frequency and scope |
| Compare their row counts to the median | Scale threshold problems |
| Diff their configuration against a working tenant | Flags, locale, integrations |
| Query for boundary data values | Data shape violations |
| Reproduce with a copy of their data | Confirmation before fixing |
04Reproduce with their data, carefully
The fastest route to a fix is usually reproducing against a copy of the tenant's data in a non-production environment. Handle this properly: obtain approval, restrict access, anonymise personal data where the bug does not depend on it, and delete the copy afterwards.
Where the data cannot be copied for regulatory reasons, add targeted diagnostic logging around the failing path for that tenant only. It is slower than local reproduction but avoids moving sensitive records, and it is often sufficient.
05Turn the fix into a permanent test
Once identified, add the offending data shape to your fixtures - the long unicode name, the empty string, the extreme date. That is how the class of bug stops recurring rather than reappearing in a different code path six months later.
For scale problems, seed a test tenant sized like your largest customer and run performance checks against it. Testing exclusively against small, well-formed fixtures guarantees that your biggest and most valuable accounts encounter problems first.
Topics
Lena Voss
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer