An Azure service that stores unstructured data in the cloud as blobs.
Hello , Welcome to MS Q&A
.For 1 TB+ storage and potentially millions of blobs/versions, I would recommend using Azure Blob Inventory rather than making individual API calls for every blob.
Blob Inventory is Azure’s native capability for generating a scheduled inventory of blobs and blob versions, and it can output the results in CSV or Parquet. It can include fields such as:
- Blob name/path
- Version ID
- Current version indicator
- Size
- ETag
- Last modified
- Blob type
- Metadata/properties
The inventory can generate multiple output files for large datasets, along with a manifest that helps process all the generated files.
Microsoft docs: Azure Blob Inventory – Microsoft Learn
For the Blob URL, Storage Account and Container, these can be derived from the inventory context + blob name, so we don't need to call the API separately for every blob.
If we need a real-time/on-demand export instead of a scheduled inventory, the alternative is List Blobs with include=versions and continuation tokens. Azure returns up to 5,000 blobs per request, so the process needs to handle pagination until all records are retrieved.
Microsoft docs: Enumerating Blob Resources – Microsoft Learn
So, in short:
Millions of blobs/versions → Azure Blob Inventory → CSV/Parquet → Salesforce migration pipeline
This would be the most Azure-native and scalable approach. If the requirement is an immediate snapshot rather than scheduled inventory, then we should go with the List Blobs + pagination approach.
Pls check and let me know if any further ques
Thanks
Deepanshu