An Azure service that stores unstructured data in the cloud as blobs.
Azure Firewall log ingestion – Blob Storage update behavior and alternative ingestion options
Hi Microsoft Team,
We are currently evaluating Azure Blob Storage as an ingestion method for Azure Firewall logs and would like some clarification regarding the blob creation/update behavior and the available alternatives.
1. Azure Firewall logs written to Blob Storage
We understand that Azure Firewall diagnostic logs written to Blob Storage are organized into hourly PT1H.json blobs, and that during the current hour, events are appended to the same PT1H.json blob as they are received.
Could you please confirm the following:
- Once a
PT1H.jsonblob is created for an hour, will the same blob continue to be updated/appended throughout that hour? - After the hour is completed, is that blob finalized and no longer updated, with a new
PT1H.jsonblob being created for the next hour? - When new events are appended to an existing
PT1H.jsonblob during the same hour, does Azure Blob Storage generate an Event Grid event/notification for that update? - If so, which Event Grid event type is generated, and can it reliably be used to detect new data appended to the existing hourly blob?
- Is there an Azure-supported mechanism to identify when the current hourly blob has received new data or when the blob is ready to be consumed?
We are particularly interested in understanding this behavior because an ingestion consumer needs to avoid reading the hourly blob while it is still being updated and ensure that all events for the hour are eventually processed.
2. Alternative ingestion methods
If Blob Storage is not the recommended approach for this use case, could you please confirm whether Azure Firewall logs can be forwarded/ingested through any of the following mechanisms instead?
- Event Hubs
- Azure Storage Queues
- An Azure Firewall API or other supported API-based ingestion method
- Any other Azure-supported mechanism that would allow us to reliably consume Azure Firewall logs incrementally
For each applicable option, could you please provide the relevant Microsoft documentation describing how Azure Firewall logs can be configured to use that destination and any relevant limitations or considerations?
In particular, we would appreciate guidance on which approach Microsoft recommends when the requirement is to reliably consume Azure Firewall logs as they become available, without missing or partially reading events from the hourly Blob Storage files.
Thank you.