Server deployments taught a generation of engineers that mistakes are cheap. Roll back, redeploy, move on. Firmware does not work that way. A device that fails to boot after an update is not a rollback, it is a field visit, and if the update reached the whole fleet it is a field visit to every customer you have. The engineering that prevents that happens before the first update ships.
01A/B partitions are the floor, not an optimisation
The device holds two complete system images. The update writes to the inactive slot, marks it as the boot target, and reboots. If the new image boots and successfully checks in, it is confirmed. If it does not, the bootloader falls back to the previous slot on the next reset.
The cost is doubling your storage requirement, which on a cost-sensitive device is a real conversation with hardware. It is a conversation worth having, because the alternative - writing over the running system and hoping - means any interrupted or faulty update is unrecoverable without physical access.
| Strategy | Storage cost | Failure mode | Recovery |
|---|---|---|---|
| In-place overwrite | 1x | Interrupted write bricks device | Physical access |
| A/B partitions | 2x | New image fails to boot | Automatic, on reset |
| A/B + recovery image | 2.5x | Both slots corrupted | Recovery image reflashes |
| Delta updates on A/B | 2x + patch | Bad patch, same as A/B | Automatic |
02The confirmation handshake is what makes rollback work
Booting is not the same as working. A device can boot cleanly and still have broken connectivity, a failed sensor driver or a crash loop in the application layer. If your definition of success is 'it booted', you will confirm updates that have bricked the device functionally.
Define success as a check-in to the platform after the application is genuinely running, with a watchdog that triggers rollback if that check-in does not arrive within a defined window. The device should have to prove it works, not merely prove it started.
03Sign everything, verify in the bootloader
Firmware images must be signed, and the signature must be verified by code the attacker cannot replace - which means the bootloader, ideally rooted in hardware. Verifying in the application layer is theatre: anything that can install unsigned firmware can also replace the thing doing the checking.
Key management is the part teams underestimate. Signing keys need to live in an HSM rather than a CI environment variable, key rotation has to be possible without physical access to devices already in the field, and a compromised key needs a revocation path. Decide all of this before the first unit ships, because retrofitting it to deployed hardware ranges from painful to impossible.
04Roll out in cohorts, halt automatically
Never ship to the whole fleet at once, regardless of how confident testing has made you. Internal devices first, then one percent, then ten, then the rest, with a soak period at each stage long enough for slow-burning problems - memory leaks, thermal issues, storage wear - to appear.
The halt condition must be automatic. A human watching a dashboard is not a control; by the time someone notices the failure rate climbing, the rollout has already reached thousands of devices. Define the threshold, wire it to the rollout system, and let it stop itself.
| Stage | Fleet share | Soak | Halt if |
|---|---|---|---|
| Internal | ~20 devices | 48h | Any failed confirmation |
| Canary | 1% | 72h | Failure rate > 0.5% |
| Early | 10% | 72h | Failure rate > 0.2% |
| Broad | 50% | 48h | Failure rate > 0.1% |
| Full | 100% | - | Failure rate > 0.1% |
05Assume the network dies mid-download
Field connectivity drops constantly, so downloads must be resumable and every image must carry a checksum verified before installation. A partial download that gets installed because nobody checked the length is a self-inflicted brick.
Bandwidth is also a real cost when the fleet is on cellular. Delta updates cut transfer dramatically but add complexity - the device must be on the exact expected base version, so you need a fallback to full image when it is not. For fleets in the thousands on metered connections, that complexity usually pays for itself quickly.
Topics
Marcus Hale
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer