Updating firmware on fifty thousand ESP32s without bricking any

The first over-the-air update on a new product usually goes fine. It goes to a handful of boards on a desk, all of them on good Wi-Fi, all of them a few metres from the person who wrote the code. The update that hurts comes a year later, when the fleet has grown to tens of thousands of units sitting in basements, on rooftops and behind metal cabinets, and a small change to the network stack means a few percent of them never come back online. Nobody is going to drive out to a thousand sites to reflash a board by hand, so the whole update path has to assume that something will go wrong and still leave every device in a state it can recover from.

This is how I approach it on ESP32 based devices. Most of it carries over to other microcontrollers with a little translation.

Two slots and a way back

The foundation is the partition table. ESP-IDF supports two application slots, usually called ota_0 and ota_1, plus a small otadata partition that tells the bootloader which slot to run. The device always runs from one slot and downloads the new image into the other. If the download fails halfway, nothing has changed and the device carries on with the old firmware. Only once the new image is fully written and its checksum verified does the device switch the boot pointer and restart.

That covers a broken download. It does nothing for an image that downloads perfectly and then fails to work, which is the case that actually bricks devices in the field. For that you want the bootloader's rollback feature, enabled with CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE. With it switched on, a freshly installed image boots in a pending state. If the application confirms that it is healthy, the image becomes the new normal. If the device restarts before that confirmation happens, the bootloader treats the new image as failed and goes back to the previous slot on its own.

The whole design then hinges on what the application counts as healthy.

Deciding when an update has worked

I have seen firmware that marks itself valid on the first line of app_main. That protects against an image that crashes before it gets that far and against very little else. A device that boots happily and then can never reach the broker again is exactly as lost as one that crashes, and it is also the more likely failure after an update that touched Wi-Fi, TLS certificates or the modem driver.

What I prefer is to confirm the image only after the device has done the one thing it exists for. It connects to the network, completes the TLS handshake, reaches the MQTT broker and publishes a message that the backend acknowledges. Until then the image stays pending, and a task watchdog or a plain timer restarts the device if it takes too long, which sends it back to the old firmware.

static void confirm_update_if_pending(void)
{
    const esp_partition_t *running = esp_ota_get_running_partition();
    esp_ota_img_states_t state;

    if (esp_ota_get_state_partition(running, &state) != ESP_OK ||
        state != ESP_OTA_IMG_PENDING_VERIFY) {
        return;
    }

    // Called only after the first telemetry message has been acknowledged
    // by the backend, so the new image has proven it can do its job.
    esp_ota_mark_app_valid_cancel_rollback();
    ESP_LOGI(TAG, "update confirmed on %s", running->label);
}

The timeout deserves some thought. A device on a weak cellular link can take a couple of minutes to register on the network, so a thirty second limit would roll back perfectly good updates. I usually give it several minutes and let the device report which version it came back on, so the backend can tell a rollback apart from a device that simply went quiet.

Only run what you signed

Anything that can install new code over the air is also an attack surface, and a fleet that accepts unsigned images is one leaked credential away from running someone else's firmware. On the ESP32 family the answer is Secure Boot v2, which checks the signature of the bootloader and the application on every boot and of every image that arrives over the air, against a public key whose digest is burned into eFuses. On the original ESP32 it needs chip revision 3.0 or newer, which is worth checking before the first production order. Flash encryption goes alongside it so the firmware cannot simply be read off the chip.

Both of these are one-way decisions made at manufacturing time, which is why I like to settle them before the first production batch. Retrofitting secure boot onto devices that are already deployed ranges from awkward to impossible. The same goes for anti-rollback. ESP-IDF can store a security version in eFuses and refuse to boot anything older, which stops an attacker from installing an old image with a known vulnerability. It is worth enabling, and also worth using carefully, since every bump permanently burns a bit. On the ESP32 the field holds 32 of them, so I raise the security version only for releases that fix a real vulnerability.

Rolling out in waves

Even with rollback in place, I never send a new version to the whole fleet at once. A bad image that rolls back cleanly still costs every device a download, a restart and a few minutes offline, and some failures only show up hours later, after a memory leak has had time to grow or after the device has gone through a night of low temperatures.

The pattern that has served me well starts with a few dozen internal devices, moves to around one percent of the real fleet, then ten percent, then everyone. Each wave waits long enough to cover at least one full daily cycle of whatever the devices do. The backend watches a small set of numbers for the updated group against the rest of the fleet, mainly how many came back on the new version, how many rolled back, reconnect rates, free heap and the rate of crash reports. If any of them drift, the rollout stops by itself.

AWS IoT Jobs has most of this built in, with rollout rates that can grow over time and abort rules based on the share of devices that fail or time out. If you run your own broker, a small service that walks a device list in batches and checks a few metrics between batches does the same job, and it is not much code.

I also like to pick the waves deliberately. The first real group should include the oldest hardware revisions, the weakest connections and the sites in the coldest places, because those are where problems show up first. A canary group made only of healthy devices on good networks tells you very little.

The cost of a megabyte

On Wi-Fi nobody thinks about image size. On cellular it adds up quickly. Once an ESP32 application has a TLS stack, an MQTT client and the product logic in it, an image of a megabyte or more is normal. At one and a half megabytes, sending it to fifty thousand devices moves roughly seventy five gigabytes of data. On an IoT data plan priced per megabyte, a single release can cost more than a month of normal telemetry, and the fleet will still need several releases a year.

A few habits help here. Keeping the image lean pays off every time you update. Resumable downloads mean a device that loses its connection at ninety percent does not start again from zero. Recent ESP-IDF releases support this in esp_https_ota through the ota_resumption option, which picks up a failed download with an HTTP range request. Delta updates, where the device downloads only the difference between its current image and the new one, can shrink a small change to a fraction of the full size. They add complexity on both the build side and the device, and I would only reach for them once the data bill makes a clear case.

Battery powered devices need one more rule, which is to never start an update below a comfortable charge level. Writing flash and holding a modem connection for a few minutes draws far more current than the device's normal routine, and a battery that dies halfway through leaves the device on the old slot at best.

What the backend needs to know

Every device should report its running firmware version, the slot it booted from and whether the last boot followed a rollback. With that in place, the fleet's version spread becomes a chart you can look at, and questions like how many devices are still on the version from March get an answer in seconds. I also keep a record of which release each device was offered, when it downloaded it and when it confirmed it. When something goes wrong, that history is usually the fastest way to see a pattern, such as every failure sharing a hardware revision or a mobile operator.

Where I would start

If I had to pick the pieces that matter most for a new product, I would put the two slot layout with rollback, confirmation tied to a real round trip to the backend, and signed images at the top, because those are hard or impossible to add later. Staged rollouts, resumable downloads and fleet version reporting can all be built on top once the devices are out there.

If you are working out an update strategy for a fleet and want to talk it through, I am always happy to compare notes.