Azure Durable Functions (TypeScript) - Non-Deterministic workflow detected after external event, but not after timer expiration

Hi team,

We're running into a non-determinism issue in an Azure Durable Functions orchestration written in TypeScript and haven't been able to identify the cause.

Error

"Non-Deterministic workflow detected: A previous execution of this orchestration scheduled a timer task with sequence number 19, but the current orchestration replay instead produced a ScheduleTaskOrchestratorAction action with this sequence number. Was a change made to the orchestrator code after this instance had already started running?"

Scenario

The orchestration waits for either:

• an external event (waitForExternalEvent)
• a timeout (createTimer)

using Task.any(). The timeout can be several months in the future, so the Azure Storage provider internally breaks the wait into multiple shorter timer segments, resulting in multiple TimerCreated events in the orchestration history.

When the timeout path is taken, the orchestration replays and continues successfully by executing the Activity_UpdateActivityInstance and subsequent activities without any non-determinism errors. In the attached history screenshot, this can be seen around the TimerFired event above EventId 9.

The issue only occurs when an external event is received. In this case:

1. Task.any() completes successfully.
2. resolveWinner() correctly identifies the winning event.
3. The activity immediately following the wait executes successfully.
4. After that, the orchestration fails with the non-determinism error. In the attached history screenshot, this happens at EventId 19. From the history, EventId 19 was first associated with the TimerCreated, but after receiving the event, it is linked to the Activity_UpdateActivityInstance (which is the activity immediately following the wait).

We have not modified the code so it can't be related to that.

Can you please help us understand what might be causing this replay mismatch, or point out anything suspicious in the attached history and code?

Code:

const decision = yield* waitForExternalEventTask(ipasContext, {
      metadata: {
         activityInstanceBusinessKey: 'WaitForExternalEvent:LegalRepAction1',
      },
      input: {
         eventNames: [EXTERNAL_EVENTS.AccountDiscontinued, EXTERNAL_EVENTS.PaymentAccountUpdated],
		 // timeoutAt is guaranteed deterministic 
         timeoutAt: timeoutAt.toJSDate().toISOString(),
      },
   })

export function* waitForExternalEventTask<E extends EventNames>(...): Generator<Task, WaitForExternalEventOutput<E[number]>, unknown> {
   ...

   yield* updateActivityInstanceCaller.waitForOne({
      input: {
         historyEntry: { status: 'WAITING' },
         activityInstanceId: activityInstance.id,
      },
   })

   orchestrationLogger.info(`Waiting for one of "${eventNames}" until ${timeoutAt.toISOString()}`)

   const eventTasks: Task[] = eventNames.map((name) => orchestrationContext.df.waitForExternalEvent(name))
   const forceTimeoutNowTask: Task = orchestrationContext.df.waitForExternalEvent(FORCE_TIMEOUT_NOW_EVENT_NAME)
   const timeoutTask: TimerTask = orchestrationContext.df.createTimer(timeoutAt)

   const winnerTask = (yield orchestrationContext.df.Task.any([
      ...eventTasks,
      timeoutTask,
      forceTimeoutNowTask,
   ])) as Task

   const respondedAt: Date = orchestrationContext.df.currentUtcDateTime
   const result: WaitForExternalEventOutput<E[number]> = resolveWinner(
      winnerTask,
      eventTasks,
      eventNames,
      forceTimeoutNowTask,
      timeoutTask,
      respondedAt
   )

   yield* updateActivityInstanceCaller.waitForOne({
      input: {
         historyEntry: { status: 'SUCCESS' },
         activityInstanceId: activityInstance.id,
         output: result,
      },
   })

   return result
}

function resolveWinner<E extends EventNames>(
   winnerTask: Task,
   eventTasks: Task[],
   eventNames: E,
   forceTimeoutNowTask: Task,
   timeoutTask: TimerTask,
   respondedAt: Date
): WaitForExternalEventOutput<E[number]> {
   // always clean up a timer that did not win
   if (!timeoutTask.isCompleted) {
      timeoutTask.cancel()
   }

   // Case 1: One of the user defined events received
   const winnerIndex: number = eventTasks.indexOf(winnerTask)
   if (winnerIndex !== -1) {
      const eventName: string = eventNames[winnerIndex]!

      return {
         reason: ExternalEventReason.EventReceived,
         eventName,
         respondedAt: respondedAt.toISOString(),
      }
   }

   // Case 2: Force timeout now event received
   if (winnerTask === forceTimeoutNowTask) {
      return {
         reason: ExternalEventReason.Timeout,
         eventName: FORCE_TIMEOUT_NOW_EVENT_NAME,
         respondedAt: respondedAt.toISOString(),
      }
   }

   // Case 3: Timeout
   if (winnerTask === timeoutTask) {
      return {
         reason: ExternalEventReason.Timeout,
         respondedAt: respondedAt.toISOString(),
      }
   }

   throw new Error(`Task.any returned an unrecognised winner while waiting for "${eventNames}"`)
}