Installation Microsoft Connected Cache - Longueur de clé générée : 44 chars au lieu de 88

Bocar KANE 0 Reputation points
2026-06-18T10:54:28.2666667+00:00

Our Microsoft Connected Cache ISP node has been Unhealthy since June 15, 2026 following a BGP routing change.

The provisionmcc.sh script consistently generates a truncated symmetric key of 44 characters instead of the expected 88 in /etc/aziot/config.toml, causing a persistent 401 authorization error on every DPS provisioning attempt.

We have already run uninstallmcc.sh + provisionmcc.sh multiple times with fresh registration keys, deleted and recreated the cache node — problem persists.

How to fix this issue?

Thanks

Azure IoT Edge
Azure IoT Edge

An Azure service that is used to deploy cloud workloads to run on internet of things (IoT) edge devices via standard containers.


2 answers

Sort by: Newest
  1. Anshika Varshney 15,625 Reputation points Microsoft External Staff Moderator
    2026-06-24T21:25:27.5633333+00:00

    Hello @Bocar KANE

    The symptoms point to a provisioning issue rather than a cache-node software corruption, especially since the problem persists after uninstall/reinstall and node recreation. A 44-character value in /etc/aziot/config.toml usually indicates the symmetric key is being truncated or transformed before it reaches the IoT Edge/Aziot configuration, which then causes the DPS 401 authorization failure.

    Suggested troubleshooting steps

    1. Verify network path and DNS after the BGP change. Confirm the node can reach all required Microsoft endpoints consistently from the new routing path, including the DPS and IoT Hub endpoints used during provisioning. A routing change can expose MTU, NAT, proxy, or asymmetric-return-path issues that break provisioning even when basic connectivity looks fine.
    2. Check whether the registration key is being copied or parsed incorrectly. Re-enter the key manually from the portal or source of truth and avoid copy/paste through terminals, scripts, or password managers that may trim characters, wrap lines, or alter encoding. Also confirm there are no hidden spaces, line breaks, or shell-expansion issues in the script input.
    3. Inspect the generated /etc/aziot/config.toml before provisioning completes. Compare the expected symmetric key length with what is written to the file and confirm whether truncation happens during script generation or only after provisioning starts. If the file already contains the shortened value, the issue is in the local generation step rather than DPS itself.
    4. Clear stale provisioning state before retrying. Remove any leftover IoT Edge/Aziot state, cached certificates, and prior MCC configuration artifacts, then rerun uninstallmcc.sh and provisionmcc.sh from a clean state. If possible, also reboot the host after cleanup to ensure no old services or files are reused.
    5. Validate time synchronization and system identity. Make sure the host clock is correct and NTP is working, because time drift can interfere with auth flows and certificate-based provisioning. Also confirm the server identity has not changed unexpectedly after recreation or routing changes.
    6. Test from an alternate network path or temporary bypass. If possible, move the node to a different egress path or temporarily bypass the changed BGP route to see whether provisioning succeeds. If it works on the alternate path, the issue is likely network-related rather than MCC-specific.
    7. Run the provisioning with verbose logging and capture the exact failure point. Collect the full output from provisionmcc.sh, IoT Edge/Aziot logs, and any related system logs to determine whether the key is truncated before the DPS call or whether the DPS call itself rejects an otherwise valid payload. That distinction will determine whether the fix belongs in the script, the network, or the backend registration.
    8. Please provide some below details:
    • Full provisionmcc.sh output with debugging enabled.
    • The exact length and redacted form of the symmetric key as written in /etc/aziot/config.toml.
    • Network diagnostics from the host, including DNS resolution and outbound connectivity to the required endpoints.
    • The timestamp when the node first became unhealthy, aligned to the BGP change window.

    If the key is already truncated in the generated config file, escalate as a provisioning-generation defect and attach the logs plus the exact repro steps. If the generated key is correct but DPS still returns 401, focus escalation on network path validation and endpoint reachability after the BGP routing change.

    I Hope this helps. Do let me know if you have any further queries.

    Thankyou!

    Was this answer helpful?

    0 comments No comments

  2. Jerald Felix 18,760 Reputation points Volunteer Moderator
    2026-06-22T12:11:39.8366667+00:00

    Hello Bocar KANE,

    Greetings! Thanks for raising this question in the Q&A forum.

    This is a very specific and technically precise issue you have described. The truncated 44-character symmetric key in config.toml instead of the expected 88-character key is the direct cause of the persistent 401 on every DPS provisioning attempt, because the key is being written incomplete and DPS rejects it. Let me explain the root cause and the correct path to resolution.

    Why this is happening

    The 44-character versus 88-character discrepancy is a classic sign of a base64 encoding truncation during the provisioning call that fetches the symmetric key from the MCC backend. The provisionmcc.sh script calls out to the Azure MCC provisioning service to retrieve the symmetric key that gets written into /etc/aziot/config.toml. If that outbound HTTPS call is partially intercepted, inspected, or NAT-translated by a network device in a way that truncates the response body, the script writes only the first half of the key it received 44 characters instead of 88 and then proceeds to configure aziot with this broken value.

    The BGP routing change on June 15 is the critical clue here. A BGP change that alters the egress path from your MCC server can cause traffic to flow through a different firewall, DPI appliance, or TLS inspection proxy that it was not previously traversing. If that device performs any response body modification, SSL inspection, or content filtering on HTTPS responses, the key retrieval call from provisionmcc.sh will receive a truncated payload. Since you have reproduced this consistently across multiple uninstall and reinstall cycles with fresh registration keys, the problem is almost certainly not in the key itself or the portal, but in the network path the provisioning script takes to reach the MCC backend APIs.

    Here are the steps to diagnose and resolve this:

    Step 1: Verify the exact network path from the MCC server post-BGP change

    On the MCC server, trace the route to the MCC provisioning endpoint to see which path is now being used after the BGP change:

    traceroute -T -p 443 management.azure.com
    traceroute -T -p 443 global.azure-devices-provisioning.net
    

    Compare this output with what the route looked like before June 15. If the BGP change has caused traffic to flow through a new transit or internal network device, that device is the likely culprit.

    Step 2: Check for TLS inspection or proxy on the new network path

    A network element that performs certificate validation or TLS inspection on the connection can interfere with the provisioning script's HTTPS calls, causing truncated or malformed responses. Run the following to check if the connection to the provisioning endpoint is being intercepted:

    curl -v https://global.azure-devices-provisioning.net 2>&1 | grep -E "issuer|subject|SSL"
    

    If the certificate issuer shown is your ISP's internal CA or a DPI vendor (e.g., Cisco, Palo Alto, Zscaler) rather than DigiCert or Microsoft, then TLS inspection is the cause of the truncation. The provisioning script does not support TLS-inspecting proxies.

    Step 3: Update Docker's DNS as the first immediate fix to try

    The MCC support documentation recommends this as the first step for connectivity issues after provisioning failures. Updating Docker's DNS can help resolve connectivity issues. Try updating to Google's public DNS resolver:

    nano /etc/docker/daemon.json
    

    Update the file to:

    {
      "log-driver": "json-file",
      "log-opts": {"max-size": "10m", "max-file": "3"},
      "dns": ["8.8.8.8", "8.8.4.4"]
    }
    

    Then restart Docker and rerun iotedge check:

    systemctl restart docker
    iotedge check --verbose
    

    Step 4: Manually inspect and correct the truncated key in config.toml

    While you investigate the root network cause, you can attempt a manual repair to verify whether a full-length key works. First, retrieve the correct full 88-character symmetric key from the Azure portal:

    1. Go to Azure Portal > Your MCC Resource > Settings > Cache Nodes
    2. Select your cache node and look for the Registration Key or Symmetric Key values
    3. Copy the full key value

    Then manually edit /etc/aziot/config.toml on the server:

    sudo nano /etc/aziot/config.toml
    

    Find the symmetric_key line and replace the 44-character truncated value with the full 88-character key. Save the file, then restart the IoT identity service:

    sudo aziotctl config apply
    sudo systemctl restart aziot-keyd aziot-certd aziot-identityd aziot-tpmd
    sudo iotedge system restart
    

    Then check the status:

    iotedge check --verbose
    

    If the node comes up healthy with the manually corrected key, this confirms the truncation is happening during the provisioning script's API call, not in the portal.

    Step 5: Use the Diagnose and Solve Problems tool in the portal

    If the issue persists, use the Diagnose and solve problems functionality within your Connected Cache resource in the Azure portal. You can find this on the left pane within your Connected Cache resource. Select Troubleshoot under the type of problem you are facing and follow the prompts.

    Step 6: Escalate to Microsoft via a support ticket

    Since this issue started with a specific BGP routing change and the provisioning script is behaving consistently incorrectly across multiple fresh registration keys, this is very likely a network-path issue specific to your ISP environment after the routing change not a general MCC bug. The MCC team at Microsoft needs to be involved to assist further.

    Raise a support ticket via Azure Portal > Help + Support > New support request, selecting Microsoft Connected Cache as the service. Include:

    • The date the issue started (June 15, 2026)
    • That the BGP routing change preceded the failure
    • The truncated 44-character key being written to config.toml
    • The iotedge check verbose output
    • Your cache node ID and the 401 error correlation from the DPS provisioning logs

    You can also reach the MCC ISP team directly at mccforispsupport@microsoft.com which is the dedicated support channel for ISP MCC deployments.

    If this answer helps you kindly accept the answer which will help others who have similar questions.

    Best Regards,

    Jerald Felix.

    Was this answer helpful?

    0 comments No comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.