Alerting on: New dead-lettered message detected in service bus

Hello @Vuong, K (Kevin) ,

one thing to note before you implement the first approach is that it fails in a way that's hard to notice.

DeadletteredMessages cannot be forwarded to a Log Analytics workspace. In the Service Bus monitoring data reference, the metric is listed as DS Export: No, meaning it is not exported by diagnostic settings even if AllMetrics is enabled. As a result, a query on AzureMetrics returns an empty result instead of an incorrect one, which cannot be distinguished from “no messages have been dead-lettered so far.” The warning remains silent, and the configuration appears to be functioning properly.

The same reference notes that dimensions are generally not included in exported metric data. EntityName is the only dimension for this metric, which is why filtering by the queue name would not narrow down the results even for exportable metrics.

A replacement that leaves the dead-letter queue untouched

The previous answer points out that a message must be completed to avoid repeated warnings. This applies to Receive, which locks the message. Peek does not lock the message, is explicitly non-destructive, works on dead-letter queues, iterates in enqueue order from the lowest to the highest sequence number, and accepts a starting sequence number. This results in exactly the behavior you originally asked about: one warning for each new message sent to the dead-letter queue, while the existing messages remain unchanged.

[Function("DeadLetterWatermark")]
public async Task Run([TimerTrigger("0 */5 * * * *")] TimerInfo timer)
{
    await using var receiver = _client.CreateReceiver(
        "orders",
        new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });

    long watermark = await _state.GetWatermarkAsync("orders");
    long highest = watermark;

    while (true)
    {
        var batch = await receiver.PeekMessagesAsync(
            maxMessages: 50,
            fromSequenceNumber: highest + 1);

        if (batch.Count == 0)
        {
            break;
        }

        foreach (var message in batch)
        {
            await _alerts.RaiseAsync(
                message.MessageId,
                message.DeadLetterSource);

            highest = Math.Max(highest, message.SequenceNumber);
        }
    }

    if (highest > watermark)
    {
        await _state.SetWatermarkAsync("orders", highest);
    }
}

Three details determine whether this will work in production:

• PeekMessagesAsync returns at most the requested number of messages and does not guarantee a minimum number. Therefore, the loop runs until an iteration returns an empty result. A single call would silently truncate a burst of messages.
• The watermark should be stored in persistent external storage, such as Azure Table Storage, with one entry per entity. A Function host restarts and scales down to zero, and any loss of in-memory state results in either repeated alerts or a missed time window.
• Pre-populate the watermark once with the highest currently available sequence number. Since your dead-letter queue is already full, the first pass would otherwise trigger alerts for the entire backlog.

Peek returns a detached snapshot and may still contain messages that have been consumed or have expired, because cleanup runs asynchronously. This is generally acceptable for a notification, but the status from the Peek should not be treated as an exact count.

If the messages may leave the queue

ForwardDeadLetteredMessagesTo on the queue or subscription forwards dead-lettered messages to a regular queue in the same namespace, where a normal trigger is fired once per message and multiple entities can share a handler. This becomes relevant as soon as the archiving and closing pattern described in the previous answer grows beyond a handful of queues. The Basic plan does not support this; the target entity must already exist in the same namespace, and chains are limited to four hops.

References:

• Monitoring data reference, including the export column and the note on dimensions: https://learn.microsoft.com/en-us/azure/service-bus-messaging/monitor-service-bus-reference
• Message browsing and peek semantics: https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-browsing
• Auto-forwarding: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-auto-forwarding

Drafted with help from Claude, disclosed per the Q&A AI usage policy. The export column and the dimension note were verified against the Service Bus monitoring data reference, the peek semantics against the message browsing page, and the auto-forwarding limits against its documentation. The code sample illustrates the pattern and was not run against a live namespace.

Hello @Vuong, K (Kevin) ,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

You are running into a well-known limitation of native Azure Monitor metric alerts. When you configure an alert on the "Count of dead-lettered messages in a Queue/Topic" metric (e.g., Greater than 0), Azure Monitor evaluates the alert as stateful. This means the alert transitions to a "Fired" state upon the first dead-lettered message and remains fired as long as the count is above 0. It will not send new notifications for subsequent messages that enter the queue.

To alert on newly dead-lettered messages while preserving the existing ones in the DLQ, you have two primary approaches:

Approach 1: Log Search Alert using KQL (Code-Free)

Instead of relying on the native metric threshold, you can stream your Service Bus metrics to a Log Analytics workspace and use a custom Kusto Query Language (KQL) script to detect a delta (increase) in the dead-letter count.

Steps:

1. Navigate to your Service Bus Namespace -> Diagnostic settings.
2. Click Add diagnostic setting and route AllMetrics to your Log Analytics workspace.
3. Once the metrics begin flowing, navigate to Logs and create a Log Search Alert using the following query:

AzureMetrics
| where ResourceProvider == "MICROSOFT.SERVICEBUS"
| where MetricName == "DeadletteredMessages"
// Optional: filter by your specific queue name
// | where Resource contains "YOUR_QUEUE_NAME"
| summarize CurrentCount = max(Maximum) by Resource, bin(TimeGenerated, 5m)
| sort by Resource, TimeGenerated asc
| serialize
| extend PrevCount = prev(CurrentCount, 1)
| extend PrevResource = prev(Resource, 1)
// Only calculate the difference if we are evaluating the same resource
| extend CountIncrease = iff(Resource == PrevResource, CurrentCount - PrevCount, 0.0)
| where CountIncrease > 0
| project TimeGenerated, Resource, CurrentCount, CountIncrease

Why this works: This query evaluates the DLQ depth in 5-minute increments. It compares the current count against the count from 5 minutes ago. If the difference is greater than 0, it means new messages have arrived, and the alert will fire regardless of how many messages were already sitting in the queue.

Approach 2: Active Monitoring via Logic Apps / Azure Functions

If you need immediate alerts or want your notification to include the exact message payload and the dead-letter reason, you cannot rely on metrics. You must actively listen to the DLQ.

Because reading a message without completing it causes the peek-lock to expire (resulting in an infinite loop of repeated alerts for the same message), you must actively remove the message from the DLQ. To preserve the messages for later debugging, the standard enterprise pattern is:

1. Create a Logic App or Azure Function triggered by the dead-letter queue (using the path queuename/$DeadLetterQueue).
2. Read the message properties, including the DeadLetterReason and DeadLetterErrorDescription.
3. Copy the message to a cheaper, persistent storage repository (e.g., Azure Blob Storage or a Cosmos DB container).
4. Send your custom notification (Email, Teams, Service Now, etc.).
5. Complete the message, removing it from the Service Bus DLQ.

References

• Azure Monitor Alerts - Alerts and State
• Create Log Search Alerts
• Overview of Service Bus dead-letter queues

I hope this helps guide your monitoring architecture. Please let me know if this resolves your issue