IoT Edge modules do not start after offline reboot – only edgeAgent runs

Question

IoT Edge modules do not start after offline reboot – only edgeAgent runs

shekoofy 5

We are using Azure IoT Edge 1.5 (aziot) on an ARM64 (i.MX8) Linux device.

Scenario

Device is online and connected to IoT Hub

Deployment is applied successfully

All modules (edgeHub + custom modules) are running

Device is shut down cleanly

Network is disconnected

Device is powered on while offline

Expected Previously running modules should start from cached deployment and images.

Actual

Only edgeAgent starts

No other modules are created or started

edgeAgent logs repeatedly show:

Empty edge agent config was received. Attempting to read config from backup

  `Unable to process module <name> add or update as the module identity could not be obtained`
  
     Repeated attempts to connect to IoT Hub (AMQP TCP / WebSocket)

Observations

Module images are present locally

/var/lib/aziot/identityd exists (device_info, modules, prev_state)

Device has no battery-backed RTC

RTC resets to 1970 on cold boot

  System time is corrected later via NTP, but after aziot services already started
  
  `edgeAgent` attempts to read backup config from `/tmp/edgeAgent/edgeAgent/backup.json`, which does not persist across reboots

Question Is this behavior expected by design, or is there a supported way to make IoT Edge reliably start modules offline after a reboot (for example on devices without an RTC)?We are using Azure IoT Edge 1.5 (aziot) on an ARM64 (i.MX8) Linux device.

Scenario

Device is online and connected to IoT Hub

Deployment is applied successfully

All modules (edgeHub + custom modules) are running

Device is shut down cleanly

Network is disconnected

Device is powered on while offline

Expected
Previously running modules should start from cached deployment and images.

Actual

Only edgeAgent starts

No other modules are created or started

edgeAgent logs repeatedly show:

Empty edge agent config was received. Attempting to read config from backup

  `Unable to process module <name> add or update as the module identity could not be obtained`
  
     Repeated attempts to connect to IoT Hub (AMQP TCP / WebSocket)

Observations

Module images are present locally

/var/lib/aziot/identityd exists (device_info, modules, prev_state)
This only happens in some devices, and all devices are generated identically on the same day

Question
Why does this happen? is there a supported way to make IoT Edge reliably start modules offline after a reboot?

0 comments

2 answers

Your answer

Answer 1

The behavior you're experiencing with Azure IoT Edge, where only the edgeAgent starts after an offline reboot, is likely due to the absence of a battery-backed RTC (Real-Time Clock) on your device. When the device is powered on without an RTC, the system time resets to 1970, which can lead to issues with the edgeAgent reading the backup configuration from /tmp/edgeAgent/edgeAgent/backup.json. This file does not persist across reboots, causing the edgeAgent to report an empty config and fail to start other modules.

The expected behavior is that previously running modules should start from cached deployment and images, but without the correct time, the edgeAgent cannot obtain the module identities from the IoT Hub, leading to the issues you're seeing.

To improve reliability in starting modules offline after a reboot, consider the following:

RTC Implementation: If possible, implement a battery-backed RTC to maintain the system time across reboots. This would help the edgeAgent correctly read the backup configuration.
Local Configuration: Ensure that your device can handle local configurations effectively, allowing it to start modules based on cached deployment information even when offline.
Deployment Strategy: Review your deployment strategy to ensure that it accounts for scenarios where devices may reboot without network connectivity.

Currently, there may not be a supported way to make IoT Edge reliably start modules offline after a reboot without addressing the RTC issue or ensuring the configuration persists across reboots.

References:

Solutions to common issues for Azure IoT Edge

Answer 2

Sander van de Velde | MVP 37,081 MVP

Hello @shekoofeh dehkhodaei,

welcome to this Azure Community forum.

Azure IoT Edge should be able to restart in an offline situation for months or even years.

The problem seen seems to be related to the clock as indicated.

This is because there are several certificates involved when running an Azure IoT Edge solution.

Apart from the TLS connection with the IoT Hub and the optional certificate for the IoT hub device identity registration, a certificate is used to secure the inter module communication.

If you have ruggedized your device, you have probably seen that Azure IoT edge comes standard with those certificated, valid for 60 days only.

I have never used an edge device without battery backup for the internal clock but having a correct time is essential. Can you consider adding battery support?

NTP is mentioned. Making use of a cloud based NTP server is counter productive in offline scenarios.

Can you run an NTP server on the local network?

Thinking out of the box but not endorsing this, you could try to replace the certificates with custom ones that are valid... Check this blog post (based on the older 1.1 version of Azure IoT Edge but the principal is still the same) for more information about providing custom certificates.

--

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

shekoofy 5 Reputation points

2026-01-09T11:08:00.3333333+00:00
Thanks for the detailed explanation.

We have now added a battery-backed RTC, and the device boots with a correct system time and RTC time:

Local time: Fri 2026-01-09 10:12:44 UTC RTC time: Fri 2026-01-09 10:12:44

However, the issue still occurs when the device boots offline.

Additional clarification:

edgeAgent does find and read the cached deployment (backup.json is present via a persistent bind mount)

Module images are present locally

edgeAgent parses the deployment and attempts to start modules

The failure consistently happens at module identity creation:
` ```` Unable to process module <name> add or update as the module identity could not be obtained`

This suggests that aziot-identityd is refusing to issue module identities while offline, even when:

the deployment is cached

the system clock is correct at boot

identity state exists under /var/lib/aziot/identityd

At the same time, most other devices with identical hardware, OS image, and IoT Edge version (produced the same day) are able to start modules offline correctly, which suggests this may depend on whether module identities were successfully refreshed at some earlier point.
Sander van de Velde | MVP 37,081 Reputation points MVP

2026-01-09T12:12:24.4+00:00

Hello @chic chic

this could be a bug so I recommend posting the same question in the GitHub repository

Azure IoT Edge is open source and the team behind it is actively updating the repo and releases.

They are able to go into the specifics of the technical implications of being offline.

Please cross reference the issue and this post.
shekoofy 5 Reputation points

2026-01-12T12:14:09.91+00:00
Thanks for your response;
We compared journalctl -fu aziot-identityd between a working edge board and a non-working one (only edgeAgent running), under the same offline conditions.

Key difference observed

On the working device, aziot-identityd successfully serves module-specific identity requests locally while offline, for example:

GET /identities/modules/IoTEdgeMetricsCollector?api-version=2020-09-01&type=aziot --> 200

These requests continue to succeed even though DNS resolution to IoT Hub fails, which allows edgeAgent to start modules offline.

On the non-working device, we consistently see:

GET /identities/modules?api-version=2020-09-01&type=aziot --> 404 !!! Hub client error

In this case, aziot-identityd appears unable to return cached module identities locally and instead attempts to reach IoT Hub. When offline, this fails, and module identities are never issued—so only edgeAgent runs.

Important context

Both devices:

Have correct system time and RTC

Have cached deployments Have cached module images Are intentionally offline Hardware, OS image, IoT Edge version, and production date are identical
Manas R Mohanty 17,270 Reputation points Moderator

2026-01-12T12:24:12.2666667+00:00

Hi chic chic

I am replicating this issue on "Esp32+arduino" setup and shall reach product group for insights post replication.

Thank you for your inputs and providing the needed investigation time.
shekoofy 5 Reputation points

2026-01-12T15:29:52.2533333+00:00

Thanks for your response
shekoofy 5 Reputation points

2026-01-13T14:39:59.1133333+00:00

Hi Manas Mohanty,
could you please let me know how you are trying to reproduce the issue?
Manas R Mohanty 17,270 Reputation points Moderator

2026-01-14T05:05:01.3+00:00

Hi shekoofy

But your concerns are correct. It will be hard to replicate consistently.

I was trying to stress test a Linux VM instead for minimal replication.

Next step

I have gone through above statements and trying to get clue from the device logs procured from "journalctl -fu aziot-identityd".

Follow up query

Could you share the difference on configuration of working and non-working devices.

Thank you.

Manas R Mohanty 17,270 Moderator

Hi shekoofy

IoT Edge relies on persisted runtime state + cached identities to operate offline after a reboot. If the identity cache is absent (404) or the agent’s backup plan does not persist, the runtime will try to reach IoT Hub and fail when offline.

We can probably temporarily save them.

Checking below modules with PG on work around for re-hydrating the idenity modules.

Shall update you once I hear from them. Feel free to modify and test them.


// $edgeAgent desired properties (excerpt)
{
  "$edgeAgent": {
    "properties.desired": {
      "schemaVersion": "1.1",
      "systemModules": {
        "edgeAgent": {
          "settings": {
            "image": "mcr.microsoft.com/azureiotedge-agent:1.5",
            "createOptions": {
              "HostConfig": {
                "Binds": [
                  "/var/lib/iotedge/edgeAgent:/tmp/edgeAgent"
                ]
              }
            }
          }
        },
        "edgeHub": {
          "settings": {
            "image": "mcr.microsoft.com/azureiotedge-hub:1.5",
            "createOptions": {
              "HostConfig": {
                "Binds": [
                  "/var/lib/iotedge/edgeHub:/tmp/edgeHub"
                ]
              }
            }
          }
        }
      }
    }
  }
}

Thank you.

shekoofy 5 Reputation points

2026-01-16T12:01:57.84+00:00
Hi Manas,

Thanks for the follow-up.

To answer your question about configuration differences first: there are no intentional configuration differences between the working and non-working devices. Hardware, OS image, IoT Edge version (1.5 / aziot), provisioning flow, deployment, and createOptions are identical. The devices were produced the same day and provisioned using the same process.

Regarding the suggested persistence of /tmp/edgeAgent: this is already in place on the non-working device. The edgeAgent container is currently running with a persistent bind mount:

/mnt/storage/edgeagent:/tmp/edgeAgent StorageFolder=/tmp/edgeAgent

backup.json persists across reboots, and edgeAgent successfully reads and parses the cached deployment. The failure does not occur at deployment retrieval.

The consistent failure occurs at module identity issuance. On the non-working device, aziot-identityd returns:

GET /identities/modules → 404 !!! Hub client error

followed by edgeAgent logging:

Unable to process module <name> add or update as the module identity could not be obtained

For comparison, on working devices under the same offline conditions, aziot-identityd serves module identities locally:

GET /identities/modules/<module> → 200

Happy to provide additional aziot-identityd logs or test any suggested workaround.

Thanks again for investigating this with the product group.
Manas R Mohanty 17,270 Reputation points Moderator

2026-01-20T07:05:48.8366667+00:00

Hi shekoofy

Please share the details of working/non-working devices along resource id of IOT hub in private message.

Thank you.

IoT Edge modules do not start after offline reboot – only edgeAgent runs

2 answers

Key difference observed

Important context

Your answer