Can the watermark go back in Azure Stream Analytics

Question

Can the watermark go back in Azure Stream Analytics

Dhruv Singla 250

The azure documentation for time handling in azure stream analytics states the following.

When there's any incoming event, the watermark is the largest event time Stream Analytics has seen so far minus the out-of-order tolerance window size.

When there's no incoming event, the watermark is the current estimated arrival time minus the late arrival tolerance window. The estimated arrival time is the time that has elapsed from the last time an input event was seen plus that input event's arrival time.

Suppose we have the following settings

out-of-order tolerance window is 2 minutes
late arrival tolerance window is 5 minutes

Now, suppose the following is sequence of events

Wall Clock	Event Arrives	WaterMark
10 : 00	Event 1	Max Event Time - Out of Order Window = 10:00 - 2 = 9:58
10 : 01	No Event	Estimated Arrival Time - Late Arrival Tolerance = (1 + 10:00) - 5 = 9:56
10 : 02	No Event	Estimated Arrival Time - Late Arrival Tolerance = (2 + 10:00) - 5 = 9:57

So, can the watermark go back as in the above example?

0 comments

2 answers

Your answer

Answer 1

Marcin Policht 109.3K MVP Volunteer Moderator

AFAIK, the watermark in Azure Stream Analytics is monotonic per input partition or substream, meaning it can only stay the same or move forward in time, but it cannot move backward. This is because both the max(event time seen so far) and the estimated arrival time are non-decreasing:

max(event time seen so far): This value can only stay the same or increase as new events arrive with later event times. It cannot decrease.
estimated arrival time: This value is calculated based on the last event's arrival time plus the elapsed wall-clock time since the event was processed. As time progresses, this value will only increase, as it is based on a non-decreasing wall-clock time.

Since the watermark is derived from these non-decreasing quantities, after applying the out-of-order and late arrival tolerance windows, the watermark itself will either stay the same or increase but will never decrease.

In your example, with the out-of-order tolerance and late arrival tolerance windows, the watermark will not go from 09:58 to 09:56. Instead, once the watermark reaches 09:58, it will either remain at 09:58 or move forward, but it will not move backward to 09:56.

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin

Dhruv Singla 250 Reputation points

2026-03-08T14:34:02.1033333+00:00

Hi

Thanks for the response.

If it can't go back, then what exactly determines the watermark? Is there any third formula? Like Max(existing watermark, calculated watermark)
Marcin Policht 109.3K Reputation points MVP Volunteer Moderator

2026-03-08T15:17:06.99+00:00
Yep - the watermark in Azure Stream Analytics is determined by the maximum of two values: the current watermark (the last watermark observed) and the newly calculated watermark based on the incoming event times and the time windows defined (out-of-order and late arrival tolerance).

When events arrive: The watermark is based on the event times seen so far. Specifically, it is the maximum of the event times minus the out-of-order tolerance. So, the watermark moves forward as more events come in. The formula in this case is:

watermark = max(event time seen so far) - out-of-order tolerance

When no events arrive: The watermark is calculated based on the estimated arrival time, which is the last event's arrival time plus the elapsed wall-clock time. The late arrival tolerance is then subtracted from this value. The formula here is:

watermark = estimated arrival time - late arrival tolerance

The estimated arrival time is calculated as:

estimated arrival time = last event's arrival time + time elapsed since last event

In both cases, the final watermark is determined by:

watermark = max(existing watermark, calculated watermark)

So, regardless of whether events are arriving or not, the watermark will always either stay the same or move forward, as it is calculated from a non-decreasing value (either the max event time or the estimated arrival time).

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin
Dhruv Singla 250 Reputation points

2026-03-09T05:38:09.1066667+00:00

Hi

Thanks for the response.

Can you give me link to any documentation that says the same? I just want to be double sure about it.

I can accept your comment an answer, I just want to be certain that this is the exact formula being followed. Because, the documentation should have obviously written this thing, if it was the case.
Marcin Policht 109.3K Reputation points MVP Volunteer Moderator

2026-03-09T14:35:23.7933333+00:00

Refer to https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-time-handling . You'll find there also references to https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/ and https://www.oreilly.com/radar/the-world-beyond-batch-streaming-102/

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin
Dhruv Singla 250 Reputation points

2026-03-09T19:16:45.9166667+00:00

Hi

Thanks for the links

Didn't find anything regarding your formula in the links.

Answer 2

Manoj Kumar Boyini 19,590 Microsoft External Staff Moderator

Hi Dhruv Singla

No—you won’t actually see the watermark move backwards in ASA. Even though the “idle” formula (current wall‐clock minus the late‐arrival window) can mathematically yield a lower timestamp than your last watermark, the engine always takes the max of “previous watermark” and “newly computed watermark.” In practice:

• At 10:00 you saw Watermark = 10:00 (max event time) – 2 min = 9:58.
• At 10:01 the idle formula gives 10:01 – 5 min = 9:56, but ASA will compare that with the existing 9:58 and keep 9:58.
• Likewise at 10:02 the formula says 9:57—still behind 9:58, so watermark remains 9:58 until enough time elapses to push (wall‐clock – lateTolerance) above 9:58 (that happens at 10:03).

Because of this “max(previous, computed)” behavior, your watermark is monotonic and never regresses.

References

Understand time handling in ASA (watermark definition & progression): https://docs.microsoft.com/azure/stream-analytics/stream-analytics-time-handling
Illustrated example of watermarks in ASA: https://docs.microsoft.com/azure/stream-analytics/stream-analytics-time-handling#illustrated-example-of-watermarks
Events dropped when arriving outside allowed time policy: https://docs.microsoft.com/azure/stream-analytics/stream-analytics-time-handling#how-time-progresses-in-azure-stream-analytics

Manoj Kumar Boyini 19,590 Reputation points Microsoft External Staff Moderator

2026-03-15T09:01:58.9733333+00:00

Hi Dhruv Singla

I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Dhruv Singla 250 Reputation points

2026-03-18T10:11:41.5266667+00:00

Hi Manoj
Based on my research on other questions, it looks like the formula doesn't have max. It is just that the late arrival formula triggers after late arrival tolerance window has passed. This doesn't let the watermark go back.
Smaran Thoomu 35,870 Reputation points Microsoft External Staff Moderator

2026-03-18T11:18:41.4866667+00:00
Hi Dhruv Singla Yes - your understanding is correct.

In Azure Stream Analytics, the watermark does not move backwards in practice. Even though the idle-time formula may mathematically produce a lower value, the engine will not regress the watermark.

What actually happens is:

When events are coming → watermark is based on event time – out-of-order window

When no events arrive → the late arrival logic applies, but it only advances watermark after enough time has passed

Because of this behavior, the watermark remains monotonic (only moves forward)

So in your example, the watermark will stay at 9:58 until the idle-time calculation produces a value greater than 9:58.

There is no scenario where ASA will move the watermark backwards during normal processing.

Hope this clarifies
Dhruv Singla 250 Reputation points

2026-03-19T06:51:24.9733333+00:00

Hi Smaran

As I said, I don't think the late arrival watermark formula produces a value prior to the existing watermark. This is because of the system only using this formula when late arrival window has crossed. Can you confirm this with internal staff?
Smaran Thoomu 35,870 Reputation points Microsoft External Staff Moderator

2026-03-23T13:32:03.99+00:00

Dhruv Singla Yes - your understanding is aligned with how Azure Stream Analytics behaves in practice.

The idle-time (late arrival) watermark logic is not continuously applied in a way that would reduce the current watermark. Instead, it effectively comes into play only after enough time has passed without new events, and only if the computed value would move the watermark forward.

Because of this, the watermark does not move backwards. The system ensures that time progression remains monotonic, and any internal calculation that results in a lower value is simply not used to update the watermark.

So in real execution, you won’t observe a regression - the watermark will remain at its current value until a new calculation produces a higher timestamp.

Hope this clarifies.