An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
One potential approach is to implement an Active-Passive Hot Standby configuration using Continuous Data Export and Geo-Redundant Storage. This should ensure that your data is replicated to the secondary region at the storage layer without requiring a full-scale compute cluster to be running 24/7. In this scenario, you maintain a secondary cluster at the minimum possible SKU and node count to handle metadata sync and critical small-scale queries, which significantly reduces the monthly burn compared to a mirrored production environment.
Data replication is managed by exporting your primary data to an Azure Storage account configured with GRS or RA-GRS. You can use the following command structure to set up a continuous export job that pushes your data to the secondary region's storage endpoint.
.create-or-alter continuous-export MyExportJob
over (MyTable)
to table MyExternalTable
with (intervalBetweenRuns=5m)
<| MyTable | where ingestion_time() > ago(5m)
In the event of a regional failure like the one recently seen in Central US, you would trigger a scale-up operation on the passive cluster. This can be automated using the Azure CLI to increase the capacity or change the VM size to match your production requirements only when the failover is active. This allows you to pay for the high-performance compute only during the duration of the incident.
az kusto cluster update --cluster-name "secondary-cluster" --resource-group "my-rg" --sku name="Standard_E16s_v4" capacity=10
Another way to further reduce costs is to utilize External Tables on the secondary cluster. Instead of ingesting all data twice, which doubles the ingestion cost, the secondary cluster can query the exported data directly from the GRS storage.
If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.
hth
Marcin