Data Purger role not honored by Log Analytics Purge API

Hello Jeremy Cahill,

Thanks for the very detailed write-up, the symptoms point to something that is almost always mis-diagnosed as an RBAC misconfiguration. The Log Analytics purge endpoint wraps the authorization decision inside a QueryValidationError payload, but the underlying check is still a real RBAC evaluation against the caller's bearer token claims, not against the current role-assignment state in ARM. That distinction is what usually causes this exact failure pattern.

Most likely root cause: stale / cached token on the user-assigned managed identity.

When you acquire a token via MSAL's ManagedIdentityApplicationBuilder for https://management.azure.com/.default, the token is cached (both by IMDS/the managed identity service and by MSAL's in-memory cache in the Function host). Tokens are typically valid for ~24 hours, and the IMDS endpoint will continue to hand back a cached token until it is close to expiry.

The purge service performs its RBAC check against the oid / claims in the presented token, not by re-querying ARM at request time. So if the Data Purger role was assigned after the token currently being used was first issued, the cached token does not reflect the new role and the purge backend correctly reports "does not have the role 'Data Purger'" even though the assignment is visibly correct in the portal.

This matches your symptoms exactly:

• The query operation (which uses a different RBAC-checked permission that was already on the token when it was first cached, e.g., via Monitoring Reader on App Insights) succeeds.
• The purge call (which requires Microsoft.OperationalInsights/workspaces/purge/action, granted by the Data Purger built-in role, role ID 150f5e0c-0603-4f03-8c7f-cf70034c4e90) fails.

Reference: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/personal-data-mgmt

• Removing and re-adding the assignment on the portal does not help because the running Function instance is still holding the same cached token.
• It is reproducible across multiple workspaces / identities because every one of those identities has the same "first-token-acquired-before-role-was-granted" condition.

Please have a look into below provided actions to confirm and fix:

1. Decode the bearer token your Function is sending (paste it on jwt.ms) and verify:
   • oid matches the user-assigned managed identity's Object (Principal) ID.
   • aud is https://management.azure.com/.
   • iat (issued-at) is before the time you (re)assigned the Data Purger role.
2. Force a fresh token acquisition. Easiest options:
   • Restart the Function App (Stop → Start, not just Restart from VS Code) so the MSAL in-process cache is cleared.
   • Or, in code, call the managed-identity token provider with forceRefresh: true (or use TokenRequestContext with ForceRefresh = true for Azure.Identity) so MSAL bypasses its cache and re-acquires a token from IMDS.
   • As a quick sanity check, re-deploy / swap the slot , this also forces a new worker process and a new token.
3. After the new token is issued, re-run the purge. The 400 / QueryValidationError should disappear.

Secondary things worth double-checking (in priority order):

Even after fixing the token cache, a small set of conditions can still cause QueryValidationError on AppTraces. Quickly validating them avoids a second round trip:

• Filter shape for the Properties column on AppTraces. The purge API allows a key on dynamic columns (this is how customDimensions/Properties is targeted), but only the documented operators are accepted on that combination. Run the equivalent Kusto in Logs first to confirm the predicate is well-formed and returns the expected rows,

     AppTraces
     | where TimeGenerated < datetime(<cutoff-datetime>)
     | where tostring(Properties["<custom-dimension-name>"]) == "<marker-value>"
  

  If the KQL does not return rows or the dynamic field name is mis-cased, the purge engine will reject the predicate.
• GDPR-compliance constraint. Per the official REST API reference, "Log Analytics only supports purge operations required for compliance with GDPR. The Log Analytics product team reserves the right to reject requests for purge operations that are not for the purpose of GDPR compliance." The validation layer can surface this as a QueryValidationError. If the use case is not GDPR-driven (it's cost / cleanup of telemetry), the recommended approach is to use table-level retention or the Delete Data API instead of purge.
• Confirm the role is the built-in Data Purger, not a custom role, and that the principalId on the assignment is the managed identity's object ID (not the App Registration / Enterprise App appId). Your Bicep uses principalType: 'ServicePrincipal', which is correct for a user-assigned MI, but it's worth double-checking the principalId parameter resolves to the MI's principalId, not its clientId.
• Throttling / batching. Purge is throttled at 50 requests/hour per workspace; predicates should be batched. Not a cause of QueryValidationError, but worth knowing if you start automating retries.

Please have a look into below recommended alternative if this isn't GDPR-driven:

If the purpose of this job is operational cleanup (e.g., trimming AppTraces), the supported and lower-risk path is:

• Delete Data API (tables/{table}/deleteData, api-version 2023-09-01) — requires only the Log Analytics Contributor built-in role, supports TimeGenerated and other filters, and is the documented "ideal for unplanned deletions of individual records" path.
• Or simply set table-level retention on AppTraces so old rows age out automatically.

Purge is intentionally constrained to GDPR because, as the docs call out, "Delete and purge operations are destructive and non-reversible! Use extreme caution in their execution."

References

• Workspace Purge - REST API (api-version 2025-07-01)
• Manage personal data in Azure Monitor Logs (permissions: Data Purger + Log Analytics Contributor)
• Delete Data API (recommended for non-GDPR deletions)
• Manage data retention in a Log Analytics workspace
• Microsoft.OperationalInsights/workspaces/purge/action – roles allowing this operation (Data Purger, role ID 150f5e0c-0603-4f03-8c7f-cf70034c4e90) [rbac-catalog.dev]

Could you do the following and share the result?

1. Restart the Function App and immediately re-run the purge (a single retry).
2. If it still fails, capture the new bearer token from the failing call, paste it on jwt.ms, and confirm the iat timestamp is after the most recent role assignment time on that workspace.

That single data point will confirm whether this is the cached-token scenario or whether we need to dig into the predicate-validation / GDPR-rejection path next.

Thanks,

Suchitra.