Managed Dapr 1.16.4 Scheduler not firing workflow start-jobs — all Dapr Workflows stuck in PENDING

Question

Managed Dapr 1.16.4 Scheduler not firing workflow start-jobs — all Dapr Workflows stuck in PENDING

Jorge Galrito 0

Summary All Dapr Workflow instances in this environment remain in runtimeStatus: PENDING and never transition to RUNNING. ScheduleNewWorkflowAsync is accepted (the app logs Scheduled workflow 'CaseBundleWorkflow' with instance ID '<id>') but the Scheduler never delivers the start job back to the sidecar, so execution never begins. This is environment-wide across all workflow instances (14+ requests stuck), not specific to one instance. Onset ~2026-07-15.

Business impact Production court-bundle/report generation is completely down. Officer-initiated bundle requests never produce output.

What we observe

New workflow → exists: true, isWorkflowRunning: false, runtimeStatus: 6 (PENDING), createdAt: 0001-01-01 indefinitely.
App/sidecar logs: repeated Scheduler stream connected for [JOB_TARGET_TYPE_JOB], Running actor reminder migration from state store to scheduler; during earlier multi-replica operation, remote actor moved, DaprBuiltInActorNotFoundRetries, Error communicating with placement: context canceled.
TerminateWorkflowAsync / purge calls hang indefinitely (never return) for affected instances.
Example instance: eeff1125-5466-432f-8404-f4515613aaf2 (accepted 12:24Z, never started).

Remediation already attempted (app-side), no effect

Truncated the Dapr Postgres actor/state store (416,082 → 0 rows).
Pinned the app to a single replica (min=max=1) to eliminate actor-placement churn.
Multiple rolling restarts of the revision.
Full scale-to-zero deregistration followed by fresh re-registration.

In every case a newly submitted workflow still stays in PENDING. Because the state store purge had no effect, the stuck triggers appear to reside in the Scheduler service, which we cannot access on managed ACA.

Suspected root cause (please confirm) Consistent with the Dapr Scheduler retention-reminder defect: when workflows are scheduled with stable/deterministic instance IDs and the same ID is later re-scheduled, the prior run's retention reminder fires indefinitely and never drains, accumulating in the Scheduler and saturating it so new workflows can't be dispatched (dapr_runtime_workflow_operation_count{operation=purge_workflow,status=failed} increments ~1/s per affected instance). Reference: dapr/dapr issue #10039; fix lands in Dapr 1.17.8 (which also introduced a separate regression re: missing dapr-callee-app-id, so a build with the retention fix and that resolved is required). This environment moved to the Scheduler-based reminder model in the ACA Dapr 1.16.4 rollout (Mar–Apr 2026).

Requests

Inspect/clear the accumulated, non-draining retention reminders in the Scheduler for env acaenvbv473og3b3ppm (or bounce the managed Scheduler / its etcd) to restore workflow job dispatch.
Advise timeline for moving this environment to a Dapr build containing the 1.17.8-class retention-reminder fix (without the #10039 regression).
Confirm whether any managed-Dapr rollout or Scheduler change reached uksouth on/around 2026-07-15.
Confirm the supported/GA status of Dapr Workflows on ACA and any recommended configuration to avoid Scheduler saturation.

Timeframe: first observed 2026-07-15 ~12:24 UTC; ongoing.

Jorge Galrito 0 Reputation points

2026-07-16T11:34:29.8566667+00:00
Hey @Pravallika KV

Thank you for your reply, this aligns with our own diagnosis, and we've implemented recommendations #1 (unique instance IDs; we've removed the one code path that re-used a deterministic ID) and #3 (periodic purge of completed workflows + reduced retention).

On the second recommendation ("upgrade the ACA Dapr runtime to a 1.17.8-class build") we need the concrete mechanism, because we can't find one:

There is no az containerapp env flag or subcommand to set the Dapr runtime version (only --dapr-connection-string and Dapr component management exist).

properties.daprConfiguration.version reports 1.16.4-msft.9. A PATCH setting it to 1.17.8 (api-version 2024-10-02-preview) returns success but is silently ignored, the value stays 1.16.4-msft.9 and no upgrade occurs.

There is no documentation describing a version-selection or upgrade path for managed Dapr on Container Apps.

So could you clarify one of the following:

The exact, documented mechanism by which a customer upgrades the managed Dapr runtime version on a Container Apps environment; or

Confirmation that the managed Dapr runtime version is platform-controlled and not customer-settable, in which case that recommendation is not something we can action, and the fix can only reach us via a Microsoft-driven rollout.

If it's (2): please provide the rollout timeline for a 1.17.8-class build (with the dapr-callee-app-id regression resolved) to the uksouth region, and whether an expedited or environment-scoped upgrade can be arranged. This is a production impact affecting UK police forces, and without a customer-accessible upgrade path, you are asking us to perform an action the platform doesn't expose.

Thank you
Pravallika KV 18,850 Reputation points Microsoft External Staff Moderator

2026-07-17T05:17:38.18+00:00

Hi @Jorge Galrito , I've reached out to you over private message, could you please check and fill the form provided.
Pravallika KV 18,850 Reputation points Microsoft External Staff Moderator

2026-07-21T06:56:39.8433333+00:00

Hi @Jorge Galrito , following up to see if you had a chance to check the above response. Please check and confirm.

2 answers

Your answer

Jorge Galrito 0 Reputation points

2026-07-16T11:34:29.8566667+00:00

Hey @Pravallika KV

Thank you for your reply, this aligns with our own diagnosis, and we've implemented recommendations #1 (unique instance IDs; we've removed the one code path that re-used a deterministic ID) and #3 (periodic purge of completed workflows + reduced retention).

On the second recommendation ("upgrade the ACA Dapr runtime to a 1.17.8-class build") we need the concrete mechanism, because we can't find one:

There is no az containerapp env flag or subcommand to set the Dapr runtime version (only --dapr-connection-string and Dapr component management exist).

properties.daprConfiguration.version reports 1.16.4-msft.9. A PATCH setting it to 1.17.8 (api-version 2024-10-02-preview) returns success but is silently ignored, the value stays 1.16.4-msft.9 and no upgrade occurs.

There is no documentation describing a version-selection or upgrade path for managed Dapr on Container Apps.

So could you clarify one of the following:

The exact, documented mechanism by which a customer upgrades the managed Dapr runtime version on a Container Apps environment; or

Confirmation that the managed Dapr runtime version is platform-controlled and not customer-settable, in which case that recommendation is not something we can action, and the fix can only reach us via a Microsoft-driven rollout.

If it's (2): please provide the rollout timeline for a 1.17.8-class build (with the dapr-callee-app-id regression resolved) to the uksouth region, and whether an expedited or environment-scoped upgrade can be arranged. This is a production impact affecting UK police forces, and without a customer-accessible upgrade path, you are asking us to perform an action the platform doesn't expose.

Thank you
Pravallika KV 18,850 Reputation points Microsoft External Staff Moderator

2026-07-17T05:17:38.18+00:00

Hi @Jorge Galrito , I've reached out to you over private message, could you please check and fill the form provided.
Pravallika KV 18,850 Reputation points Microsoft External Staff Moderator

2026-07-21T06:56:39.8433333+00:00

Hi @Jorge Galrito , following up to see if you had a chance to check the above response. Please check and confirm.

Answer 1

Hi @Jorge Galrito ,

Thanks for reaching out to Microsoft Q&A.

This sounds like a platform/runtime scheduling problem where Dapr Workflow schedules are being accepted, but the Scheduler never successfully dispatches the start job to the sidecar, leaving all workflows stuck in runtimeStatus: PENDING.

For Azure Container Apps Jobs, there are known platform behaviors where automated triggering can break after certain lifecycle actions (ex: suspend/resume), and the provided workaround in that context is to delete and recreate the job (with event-driven scaling metadata refreshed).
For monitoring, use Azure Monitor / Log Analytics (e.g., ContainerAppConsoleLogs_CL) to verify actual execution/log activity when portal/UI history may be incomplete or desynchronized.

But again: these do not map cleanly to Managed Dapr Workflows.

Try below:

Stop reusing deterministic workflow instance IDs.
Generate a unique ID per workflow execution (for example, append a GUID).
Upgrade ACA Dapr runtime to a version containing the Scheduler retention-reminder fix (1.17.8-class fix, with the dapr-callee-app-id regression resolved).
Reduce workflow retention and periodically purge completed workflows.

If the issue still persists, please provide the requested details over private message to investigate the issue further, we will check and provide further information.

Hope this helps!

If the resolution was helpful, kindly take a moment to click on User's image and click on Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Your option (2) is correct: the Dapr runtime version in Azure Container Apps is platform-controlled. Custom version selection is not supported, environments are upgraded automatically, and there is no documented customer command for forcing a specific build or requesting an environment-scoped upgrade. A PATCH that leaves the -msft version unchanged is therefore expected behavior.

There is another important support boundary here. The current Container Apps Dapr documentation does not list the Workflow API or the actor/workflow SDK packages as supported managed capabilities. PostgreSQL state is also a Tier 2 component, handled on a best-effort basis. Consequently, there is no supported configuration that can guarantee this Workflow/Scheduler scenario or a public rollout date for the 1.17.8-class fix.

Your evidence—fresh unique IDs remaining PENDING after the state store was emptied, plus terminate/purge calls hanging—does point past application code and toward the managed Scheduler path. Q&A cannot inspect or clear that Scheduler.

Open a severity-appropriate Azure support case and include the environment resource ID, UK South region, runtime version, first-failure UTC time, several stuck instance IDs, sidecar/placement logs, and the failing purge metric. Ask the service team to inspect Scheduler/etcd health and confirm the regional rollout state.

Avoid further destructive state-store resets. For immediate production resilience, move the workflow host to self-managed Dapr on AKS or another supported orchestrator until managed Workflow support and the required runtime fix are available.