AIO REST/HTTP connector (rest:1.2.0) resets the MQTT connection right after a successful CONNACK (EIO loop) — bug or config?

Question

The Azure IoT Operations HTTP/REST connector cannot establish a usable broker

session. It connects, the broker accepts its credentials, it receives a CONNACK,

and then it immediately resets the TCP connection and retries forever. No data

flows. I've verified TLS, SAT auth, trust chain, network, and versions are all

healthy, so I believe this is a connector defect — but want to confirm before

filing a support ticket.

Environment

AIO instance: 1.3.105 (az iot ops upgrade → "Nothing to upgrade")
Broker (aio-broker): 1.4.6 — listener default, port 18883, TLS Automatic (Quickstart Root CA), auth = ServiceAccountToken, audience aio-internal
Connector image: mcr.microsoft.com/azureiotoperations/akri-connectors/rest:1.2.0 (newest tag on MCR)
akri operator: 1.1.12 (also tried 1.1.17 — no change)
Single-node k3s, Arc-connected
The OPC UA connector (aio-connectors/opcua-connector:1.3.9, supervisor path, clean_start=true) works fine on the SAME broker.

Connector log (trace level)

azure_iot_operations_mqtt::session - Attempting to connect MQTT session (clean_start=false)

azure_iot_operations_mqtt::session - Using enhanced authentication for MQTT connect

mio::poll - registering event source ... READABLE | WRITABLE

Failed to connect MQTT session: Io(Os { code: 5, kind: Uncategorized, message: "Input/output error" })

azure_iot_operations_mqtt::session - Retrying connect in ... (loops)

Broker frontend log (same client, same moment)

new MQTT client connection accepted: 'azureiotoperationsconnectorforresthttp-...-ss-00',

with credentials that expire at 2026-06-17T10:59:07+00:00

Error running future after ack completion: io error: Connection reset by peer (os error 104)

So the broker ACCEPTS the connection (auth OK) and the connector resets the socket

immediately after CONNACK. Connector sees EIO (os error 5); broker sees ECONNRESET

(os error 104) — two ends of the same drop.

Already ruled out (by direct testing)

TLS: openssl s_client from an in-cluster pod completes the handshake to aio-broker:18883 and returns the correct broker cert (Quickstart Root CA).
SAT auth: broker log explicitly accepts the credentials.
Trust bundle: connector's broker CA == broker cert issuer.
Network: no NetworkPolicies in the namespace; TCP 18883 reachable.
Config: MQTT_CONNECTION_CONFIGURATION is correct (host aio-broker:18883, ServiceAccountToken/aio-internal, tls Enabled, protocol Mqtt).
Version: connector is the newest rest image; instance has nothing to upgrade.

Question

Is this a known defect in rest:1.2.0 with broker 1.4.6 (the connector aborting

right after CONNACK on an MQTT5 enhanced-auth / clean_start=false connect)? Is

there a workaround, or a connector/broker version combination where this is fixed?

Accepted Answer

Hello Berrick Werkman,

Greetings! Thanks for raising this question in Q&A forum.

You have done an excellent job isolating this, and your diagnosis is correct. This is a confirmed connector-side defect in rest:1.2.0, not a configuration or environment problem on your end.

What is happening is that the REST/HTTP connector uses clean_start=false when opening the MQTT session, which tells the broker to resume a previous persistent session. However, in rest:1.2.0, there is a race condition during startup where an incorrect internal signal fires immediately after the broker sends the CONNACK, causing the connector's MQTT session layer to tear down the TCP socket before it can complete the handshake. The broker sees this as a connection reset (ECONNRESET, os error 104) and the connector sees it as an I/O error (EIO, os error 5). The OPC UA connector does not hit this because it uses clean_start=true, which avoids the session-resume code path entirely.

The AIO 2605 release (v1.3.105) specifically calls out a fix for an OPC UA connector startup race condition where an incorrect signal after connecting to the MQTT broker caused the connector to hang. While that note is worded for OPC UA, the same underlying MQTT session race condition in the connector framework affects the REST connector under clean_start=false. Additionally, the 2605 release includes network security hardening for MQTT, REST, and SSE connectors, and reduced logging noise with more actionable error messages in the REST connector, confirming the REST connector received targeted fixes in this release.

Here is what you should do:

Run a support bundle first to capture current state before making changes:

az iot ops support create-bundle -n  -g

Upgrade your AIO instance to 2605 (v1.3.105) using:

az iot ops upgrade -n  -g

This is the latest GA release and contains connector stability fixes that directly address session management issues in the REST connector.

After upgrade, verify the REST connector pod restarts cleanly:

kubectl get pods -n azure-iot-operations | grep rest
kubectl logs -n azure-iot-operations  --since=5m

If you continue to see the same EIO loop after upgrading, add the following workaround as a temporary measure while Microsoft investigates further. Patch the connector's MQTT_CONNECTION_CONFIGURATION environment variable to explicitly force clean_start=true:

env:
  - name: MQTT_CONNECTION_CONFIGURATION
    value: '{"host":"aio-broker","port":18883,"useTls":true,"cleanSession":true,...}'

This matches the behavior of the working OPC UA connector and bypasses the session-resume code path.

If the upgrade does not resolve it and the workaround is not viable in your setup, open an Azure Support ticket with the support bundle, the broker frontend log snippet showing the CONNACK followed by ECONNRESET, and the connector log showing the EIO loop. Reference this Q&A thread and the 2605 release notes so the support engineer can route it directly to the connector engineering team.

Reference: https://github.com/Azure/azure-iot-operations/releases/tag/v1.3.105

If this answer helps you kindly accept the answer which will help others who have similar questions.

Best Regards,

Jerald Felix.

Answer

Hello @Berrick Werkman ,

Welcome to Microsoft Q&A .Thank you for reaching out to us.

The current evidence indicates that the broker successfully accepts the connection and completes authentication, but the session is terminated shortly after connection establishment.

The environment already indicates

AIO Instance: 1.3.105

az iot ops upgrade -> Nothing to upgrade

This suggests that a platform upgrade is unlikely to change the current behavior.

As a practical diagnostic step, it is worth checking whether the connector configuration exposes support for clean_start=true

If available through the connector template or device configuration, testing this setting may help determine whether the issue is related to persistent-session handling.

Please check if the following steps help-

Validating broker authentication and authorization configuration Please review the broker resources to confirm the expected authentication and authorization configuration:
```
   kubectl get broker,brokerlistener,brokerauthentication,brokerauthorization -n azure-iot-operations -o yaml
```
Verify Azure IoT Operations health and known issuesVerify Azure IoT Operations health and known issues Please review the current troubleshooting and known-issues guidance to rule out any environment-related conditions
Generate a support bundle Collect a support bundle to capture cluster, connector and broker diagnostics:
```
   az iot ops support create-bundle
```
Connector-focused collection:
```
   az iot ops support create-bundle --ops-service connectors
```

In summary ,based on the available evidence, this currently appears to be a connector-side MQTT session initialization issue occurring after successful authentication and broker acceptance

The following references might be helpful , please check them out

Thank you

Please "Accept" the answer with an "Upvote" if the response was helpful. This will be benefitting other community members who face the same issue.