A safe firmware update path for a fleet of field sensors

The devices sit in cabinets and on poles across a wide area, many of them in locations that take a half-day drive and a permit to reach. They shipped with an update mechanism that worked in the lab and had never been trusted in the field, so the fleet had been running the same firmware for nearly three years while defects accumulated, including one that lost readings during a specific weather condition. Sending an engineer to each device to update by hand had been costed and was roughly the price of replacing the fleet. The real problem was not building an updater; it was building one nobody would be afraid to use.

Senior embedded engineer, technical lead for this piece, on a team of five — two firmware, two on the server side, and a test engineer — over about fourteen months including the staged rollout. I wrote the firmware side of the update and the recovery behaviour, specified the server protocol, and ran the rollout with the operations manager. I was also the person who had to sit in front of a steering group and explain the risk, since the failure mode of a bad update is a device that is electronically fine and functionally dead until somebody drives to it.

I owned the on-device update logic: the dual-image arrangement so the new firmware is written to the inactive slot and only becomes the boot target after it proves itself, the verification of the image before anything switches, and the watchdog behaviour that reverts to the previous image if the new one fails to check in within a set period. I wrote the recovery path for a device that loses power mid-write, which is the case that actually happens and the reason this design exists. I also owned the field test programme with the test engineer, including deliberately interrupting updates in every way we could think of. The server-side distribution was specified by me and built by others.

Dual-image with automatic revert was the decision the whole design turns on, and it cost us memory we did not really have — it meant a smaller working area for the application and a specific feature was cut to fit. The alternative, an in-place update with a recovery mode, is what the original mechanism did and is cheaper in hardware terms; it also means a device that fails halfway is bricked until visited, which at our fleet size and travel cost is not a risk anyone should take. I also insisted that the revert be driven by the device rather than the server, so that a device which loses connectivity after an update reverts on its own rather than sitting broken waiting for an instruction it cannot receive.

The link to these devices is narrow and metered, so a firmware image had to be delivered in resumable chunks over potentially days, and we had to keep the transfer well under a monthly data allowance shared with the actual readings, which are the reason the device exists. Some sites have a permit process for physical access measured in weeks, so any plan that assumed a visit was not a plan. The regulator overseeing the operator required notification of changes affecting measurement, so each firmware release carried a documentation package and a lead time. And the whole rollout had to avoid the winter period when the readings matter most.

Four staged waves over five months: fifty devices, then four hundred, then the bulk, then the difficult sites. Around four thousand two hundred devices updated with eleven that required a physical visit, all eleven for pre-existing hardware faults the update surfaced rather than caused. The three-year-old defect is fixed across the fleet. The number that mattered internally was that the operator can now ship a firmware change in about three weeks rather than never, which has changed what the product team is willing to plan. Two devices did revert automatically in the second wave, exactly as designed, and that was the moment the steering group relaxed.

The test engineer built a rig that could cut power at any point in the write sequence and ran it a few thousand times, which found two genuine faults in my recovery path and is the reason I slept during the rollout. The two server-side engineers built the distribution and the per-device state tracking that let us run the waves. The operations manager chose the wave composition, deliberately putting the most awkward sites last, which was better sequencing than my instinct.