The pattern is consistent: an organisation commits to an ambitious AI initiative, spends two quarters on it, and discovers the real work is finding out which of four customer tables is authoritative. The modelling was never the hard part. If your data is undocumented, inconsistent and scattered, no model will rescue that - and the foundational work is unglamorous enough that it rarely gets funded on its own.
01An honest readiness check
Before committing to an AI programme, answer these about the data it would depend on. Any 'no' is work that has to happen first, whether or not it appears in the project plan.
- Can you name the authoritative source for each key entity, without debate?
- Do you know how complete the important fields actually are?
- Can you reconstruct what a record looked like at a past date?
- Is there a documented definition of your core business metrics?
- Can a new engineer find and query the data without asking a specific person?
- Do you know which datasets contain personal data, and under what basis?
02History is the constraint people discover late
Most operational systems store current state and overwrite the past. That is correct for transactional workloads and fatal for machine learning, which needs to know what was true at the moment a decision was made - not what is true now.
Training on current-state data leaks future information into historical examples, producing models that perform brilliantly in evaluation and poorly in production. If you are not capturing change history today, start now regardless of whether an AI project is approved, because the missing history cannot be recovered later.
- Capture change events, not just current state.
- Timestamp everything, including when a record became known, not just when it changed.
- Retain enough history to cover a realistic training window.
- Beware derived fields recomputed retroactively - a classic leakage source.
03The stack that is usually enough
Mid-market companies routinely over-engineer this, buying a platform sized for a problem they do not have. The components below cover the large majority of needs, and each can be a managed service rather than a build.
| Layer | Purpose | Complexity |
|---|---|---|
| Ingestion | Move source data in reliably | Low - use managed connectors |
| Warehouse or lakehouse | Central queryable store | Low - managed |
| Transformation | Versioned, tested business logic | Medium - where the real work is |
| Catalogue and lineage | Find data, trace origin | Medium |
| Quality monitoring | Detect breakage early | Low, high value |
| Feature store | Serve consistent features | Only once models are in production |
04Quality monitoring earns its keep immediately
Data pipelines fail silently. An upstream schema change or a field that starts arriving null does not raise an exception - it quietly degrades everything downstream, and by the time someone notices, weeks of decisions have been made on wrong numbers.
Basic automated checks catch most of this cheaply: row counts within expected ranges, null rates on important fields, referential integrity, freshness. This is the highest return-on-effort component in the entire stack and it is consistently the last one implemented.
- Freshness: did the data arrive when expected?
- Volume: is the row count within a sane range?
- Null rate: has completeness changed on key fields?
- Schema: has an upstream field changed type or vanished?
- Distribution: has a numeric field shifted unexpectedly?
05Governance, before it becomes an audit
Knowing which datasets contain personal data, where they came from and who can access them stops being administrative housekeeping the moment a regulator or an enterprise customer asks. Retrofitting this across an established warehouse is substantially harder than building it in.
The minimum useful version is a catalogue with ownership and sensitivity classification, access control at the dataset level, and lineage sufficient to answer where a number came from. That is achievable in weeks and pays for itself the first time someone asks whether a model was trained on personal data.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.