Azure Firewall log ingestion – Blob Storage update behavior and alternative ingestion options

Balamurugan, Susinthika 60 Reputation points
2026-09-12T13:26:28.8266667+00:00

Hi Microsoft Team,

We are currently evaluating Azure Blob Storage as an ingestion method for Azure Firewall logs and would like some clarification regarding the blob creation/update behavior and the available alternatives.

1. Azure Firewall logs written to Blob Storage

We understand that Azure Firewall diagnostic logs written to Blob Storage are organized into hourly PT1H.json blobs, and that during the current hour, events are appended to the same PT1H.json blob as they are received.

Could you please confirm the following:

  • Once a PT1H.json blob is created for an hour, will the same blob continue to be updated/appended throughout that hour?
  • After the hour is completed, is that blob finalized and no longer updated, with a new PT1H.json blob being created for the next hour?
  • When new events are appended to an existing PT1H.json blob during the same hour, does Azure Blob Storage generate an Event Grid event/notification for that update?
  • If so, which Event Grid event type is generated, and can it reliably be used to detect new data appended to the existing hourly blob?
  • Is there an Azure-supported mechanism to identify when the current hourly blob has received new data or when the blob is ready to be consumed?

We are particularly interested in understanding this behavior because an ingestion consumer needs to avoid reading the hourly blob while it is still being updated and ensure that all events for the hour are eventually processed.

2. Alternative ingestion methods

If Blob Storage is not the recommended approach for this use case, could you please confirm whether Azure Firewall logs can be forwarded/ingested through any of the following mechanisms instead?

  • Event Hubs
  • Azure Storage Queues
  • An Azure Firewall API or other supported API-based ingestion method
  • Any other Azure-supported mechanism that would allow us to reliably consume Azure Firewall logs incrementally

For each applicable option, could you please provide the relevant Microsoft documentation describing how Azure Firewall logs can be configured to use that destination and any relevant limitations or considerations?

In particular, we would appreciate guidance on which approach Microsoft recommends when the requirement is to reliably consume Azure Firewall logs as they become available, without missing or partially reading events from the hourly Blob Storage files.

Thank you.

Azure Blob Storage
Azure Blob Storage

An Azure service that stores unstructured data in the cloud as blobs.

0 comments
Answer accepted by question author
Allan Solomon Mejia 8,500 Reputation points
2026-09-12T19:16:44.8566667+00:00

Hello @Balamurugan, Susinthika

Your understanding of the PT1H.json layout is mostly correct, but one important detail is when an hourly blob is considered complete.

When Azure Monitor resource logs are archived to Storage, they use paths similar to:

insights-logs-<category>/resourceId=.../

y=YYYY/m=MM/d=DD/h=HH/m=00/PT1H.json

During the current hour, Azure Monitor appends events to the corresponding PT1H.json as it receives them.

However, don't assume the previous hour's blob becomes immutable immediately at HH:00. At the start of a new hour, existing logs can still be written to the previous hour's blob while new logs are written to the new hour's blob. Logs are organized by when Azure Monitor receives them, not necessarily when the underlying event occurred.

Therefore, there isn't a documented "hour complete" or "blob finalized" event that you would use as a reliable ingestion checkpoint.

Regarding Event Grid, avoid using Microsoft.Storage.BlobCreated as a record-level notification mechanism for these Azure Monitor archive blobs.

Event Grid's BlobCreated semantics are based on the Storage API operation used to create/replace/commit a blob. Microsoft documents it for operations such as PutBlob, PutBlockList, and CopyBlob; it isn't a contract that Azure Monitor will generate one Event Grid event for every diagnostic-log record appended to an existing PT1H.json.

Consequently, I wouldn't design an Azure Firewall log consumer around:

Azure Firewall > Diagnostic settings > Blob PT1H.json > Event Grid > Incremental consumer

if the requirement is to process every Firewall event reliably as it becomes available.

Event Hubs is the better fit for incremental ingestion

Azure Firewall diagnostic settings can stream supported resource logs directly to Azure Event Hubs. Azure Monitor explicitly supports Event Hubs as a diagnostic-settings destination for streaming platform logs to external consumers/SIEM systems.

The architecture would then be:

Azure Firewall > Azure Monitor Diagnostic Settings > Azure Event Hubs > Consumer group > Your ingestion application

This avoids having to determine whether an hourly blob has changed or whether you've already processed part of the file. Your consumer can maintain its Event Hubs partition offsets/checkpoints and process the stream incrementally.

You can also configure multiple diagnostic settings/destinations if you need different use cases, for example, Event Hubs for streaming ingestion and Storage for long-term archival. Azure Monitor diagnostic settings support Storage, Event Hubs, and Log Analytics destinations.

One Azure Firewall-specific caveat is that destination support can vary by log category. For example, Microsoft's current Azure Firewall monitoring reference states that DNS Flow Trace Logs aren't supported with Event Hub as a destination, although Storage and Log Analytics support that category.

Azure Storage Queues: these aren't a native Azure Monitor diagnostic-settings destination for Azure Firewall logs. You could build an intermediate component that reads another supported destination and writes messages to a Storage Queue, but there isn't a direct Azure Firewall diagnostic setting → Storage Queue configuration.

Azure Firewall API: I also wouldn't use the Azure Firewall management APIs as a replacement log-ingestion API. Firewall traffic/resource logs are exposed through Azure Monitor diagnostic logging; the management APIs are for management/control-plane operations, not an incremental feed of Firewall log records.

For your stated requirement: reliably consuming Azure Firewall events incrementally without repeatedly reading partially updated hourly blobs, I recommend Event Hubs, assuming the specific Firewall log categories you require support that destination.

Keep Blob Storage in parallel if you also require inexpensive long-term archive/audit retention:

User's image This separates the streaming and archival requirements rather than trying to turn the archive format into a streaming interface.

References:

Azure Monitor resource logs and PT1H.json behavior

Stream Azure monitoring data to Event Hubs

Azure Monitor diagnostic settings and supported destinations

Monitor Azure Firewall logs and metrics

Azure Firewall monitoring data reference

Azure Blob Storage events with Event Grid

=============================================================================

Help make this community better for everyone: If this answer helped or resolved your issue, please accept it or upvote it. If not, share more details in a comment so we can continue the discussion and find the right solution. Thank you.

Was this answer helpful?

1 person found this answer helpful.
0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    Comments have been turned off. Learn more

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.