WinUI 3: How can I detect user interaction with highlighted external controls and advance spoken step-by-step guidance?

Perry Jones 45 Reputation points
2026-08-12T21:45:23.2666667+00:00

Title:

WinUI 3: How can I detect user interaction with highlighted external controls and advance spoken step-by-step guidance?

Question:

I’m developing a Windows desktop application using WinUI 3, Windows App SDK, .NET 8, and Microsoft UI Automation.

The application is designed as an interactive step-by-step helper for people who may not be comfortable using a computer.

The intended experience is:

  1. The helper verbally tells the user what to do next, for example: “Click the Windows Start button. I’ve highlighted it for you.”

The application locates the real external Windows control using UI Automation.

A transparent, click-through topmost overlay highlights that control.

The user performs the action themselves. The application must never click, type, or perform the action for them.

The application detects that the user completed the requested action.

The helper verbally acknowledges completion and automatically speaks the next instruction.

The next appropriate control is highlighted.

This continues until the guided task is finished.

Our highlighting implementation currently obtains the UI Automation element's bounding rectangle and positions a transparent overlay around it.

The overlay uses:

WS_EX_TRANSPARENT | WS_EX_LAYERED | WS_EX_TOOLWINDOW

and the XAML overlay content has:

IsHitTestVisible = false

so the user should remain able to interact normally with the underlying application.

The current guidance implementation, however, is not sufficiently interactive.

For external actions it currently polls UI Automation approximately every 750 ms and determines completion primarily by checking whether a target disappeared or whether another expected UI element appeared.

For a number of external controls, completion is currently manual. The user has to return to our application and choose “I did this” before the next step becomes available.

That defeats much of the purpose of the application. The intended user may need the helper to actually talk them through the process rather than repeatedly reading instructions, performing an action, returning to the helper, and manually confirming completion.

The application already has an offline speech/voice system. The issue is not generating speech itself. We need to correctly connect external user interaction, guidance state progression, highlighting, and speech.

What is the recommended Windows architecture for implementing this type of event-driven guided interaction?

Specifically:

Can Microsoft UI Automation reliably notify our application when the user activates a UI element that we previously located and highlighted?

For buttons and links supporting InvokePattern, should we subscribe to UIA_Invoke_InvokedEventId / InvokePattern.InvokedEvent on the resolved element?

How should we detect user actions on controls that use other UI Automation patterns, such as SelectionItem, Toggle, ExpandCollapse, or text entry?

For text-entry steps, what is the recommended way to determine that the user has entered the expected text without our application taking control of the external program?

Should we combine UI Automation events with state/property-change events and use polling only as a fallback?

How should UI Automation event handlers be managed when the target element disappears and a new window/control appears as a result of the user's action?

Are there known issues with using a transparent topmost WinUI 3 overlay in combination with UI Automation event monitoring?

Is there a recommended state-machine/event pattern for safely doing:

locate target → highlight target → speak instruction → wait for user action → detect completion → acknowledge verbally → advance → locate/highlight next target → speak next instruction

The important requirement is that our application remains guidance-only. It should observe enough Windows UI state to know when the user completed a step, but the actual mouse click, keyboard input, selection, or other action must always be performed by the user.

We would appreciate guidance on the correct Microsoft UI Automation/WinUI 3 approach for building this reliably on Windows 11.Title:

WinUI 3: How can I detect user interaction with highlighted external controls and advance spoken step-by-step guidance?

Question:

I’m developing a Windows desktop application using WinUI 3, Windows App SDK, .NET 8, and Microsoft UI Automation.

The application is designed as an interactive step-by-step helper for people who may not be comfortable using a computer.

The intended experience is:

The helper verbally tells the user what to do next, for example: “Click the Windows Start button. I’ve highlighted it for you.”

The application locates the real external Windows control using UI Automation.

A transparent, click-through topmost overlay highlights that control.

The user performs the action themselves. The application must never click, type, or perform the action for them.

The application detects that the user completed the requested action.

The helper verbally acknowledges completion and automatically speaks the next instruction.

The next appropriate control is highlighted.

This continues until the guided task is finished.

Our highlighting implementation currently obtains the UI Automation element's bounding rectangle and positions a transparent overlay around it.

The overlay uses:

WS_EX_TRANSPARENT | WS_EX_LAYERED | WS_EX_TOOLWINDOW

and the XAML overlay content has:

IsHitTestVisible = false

so the user should remain able to interact normally with the underlying application.

The current guidance implementation, however, is not sufficiently interactive.

For external actions it currently polls UI Automation approximately every 750 ms and determines completion primarily by checking whether a target disappeared or whether another expected UI element appeared.

For a number of external controls, completion is currently manual. The user has to return to our application and choose “I did this” before the next step becomes available.

That defeats much of the purpose of the application. The intended user may need the helper to actually talk them through the process rather than repeatedly reading instructions, performing an action, returning to the helper, and manually confirming completion.

The application already has an offline speech/voice system. The issue is not generating speech itself. We need to correctly connect external user interaction, guidance state progression, highlighting, and speech.

What is the recommended Windows architecture for implementing this type of event-driven guided interaction?

Specifically:

Can Microsoft UI Automation reliably notify our application when the user activates a UI element that we previously located and highlighted?

For buttons and links supporting InvokePattern, should we subscribe to UIA_Invoke_InvokedEventId / InvokePattern.InvokedEvent on the resolved element?

How should we detect user actions on controls that use other UI Automation patterns, such as SelectionItem, Toggle, ExpandCollapse, or text entry?

For text-entry steps, what is the recommended way to determine that the user has entered the expected text without our application taking control of the external program?

Should we combine UI Automation events with state/property-change events and use polling only as a fallback?

How should UI Automation event handlers be managed when the target element disappears and a new window/control appears as a result of the user's action?

Are there known issues with using a transparent topmost WinUI 3 overlay in combination with UI Automation event monitoring?

Is there a recommended state-machine/event pattern for safely doing:

locate target → highlight target → speak instruction → wait for user action → detect completion → acknowledge verbally → advance → locate/highlight next target → speak next instruction

The important requirement is that our application remains guidance-only. It should observe enough Windows UI state to know when the user completed a step, but the actual mouse click, keyboard input, selection, or other action must always be performed by the user.

We would appreciate guidance on the correct Microsoft UI Automation/WinUI 3 approach for building this reliably on Windows 11.

Windows development | WinUI
0 comments
Answer accepted by question author
Jay Pham (WICLOUD CORPORATION) 4,435 Reputation points Microsoft External Staff Moderator
2026-08-13T00:17:38.9533333+00:00

Hi @Perry Jones

I see the gap in the current design. UI Automation events can make the guidance responsive, but an event alone should not be treated as proof that the user completed the step.

I recommend an event-driven state machine with post-condition validation:

  1. Locate the target and define what completion means for that step.
  2. Subscribe to the target's relevant UIA event.
  3. Highlight the target and speak the instruction.
  4. When an event arrives, re-read the target or resulting UI state.
  5. Advance only when the expected post-condition is true.
  6. Unsubscribe, cancel the old step, and register handlers for the next target.

For a button or link that supports InvokePattern, subscribe to UIA_Invoke_InvokedEventId / InvokePattern.InvokedEvent, but do not call Invoke() yourself. For selection, toggle, and expand/collapse steps, subscribe to the corresponding selection event or pattern-property change and then verify IsSelected, ToggleState, or ExpandCollapseState. For text entry, observe Value or text-change events, debounce briefly, then read the value and compare it with the expected text. Do not read password fields.

Microsoft documents both the benefit and limitation of this approach: UIA subscriptions reduce continuous polling, but providers do not always raise every possible event. Microsoft also notes that some events can be raised without an actual state change. Therefore, keep low-frequency, bounded polling as a fallback and always validate the expected result before advancing:

Run UIA discovery and event registration on a dedicated MTA thread and remove handlers on that same thread. Use a step generation ID or cancellation token because an event can still arrive after unsubscription: Understanding Threading Issues.

The transparent overlay is compatible with this design. Microsoft documents that WS_EX_TRANSPARENT on a layered window passes mouse events to the windows underneath it: Window Features - Layered Windows. Also exclude your own overlay process from UIA target searches and ensure it does not take focus.

This keeps the application guidance-only. The user still performs every input action; the helper observes the resulting UI state and advances only after validating completion.

If you found my response helpful or informative, I would greatly appreciate it if you could provide feedback by interacting with the system or leaving a comment below.

Thank you.

Was this answer helpful?

1 person found this answer helpful.

1 additional answer

Sort by: Newest
Most helpful Newest Oldest
  1. Perry Jones 45 Reputation points
    2026-08-13T02:05:01.0966667+00:00

    “We implemented the event-driven UIA state machine and now have a real Windows runner proof. The app launches and reaches the first guidance step, but the Windows Start button highlight is not observed. What is the most reliable way in WinUI 3/UI Automation to resolve and highlight the Windows 11 Start button from another process, especially when the taskbar/Start surface may be hosted by Explorer or StartMenuExperienceHost?”

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.