KPIs for Windows AI Foundry (e.g. Tokens/sec)

Hello John Herwig,

Thank you for pointing this out and for sharing the exact command example that’s very helpful. You are absolutely correct: when running models locally through Windows AI Foundry, KPIs such as tokens/sec and latency are provided using the --verbose flag.

For example:

foundry.exe model run qwen2.5-1.5b-instruct-generic-gpu --verbose --prompt "what is the capital of india?"

With --verbose, the output includes detailed performance statistics such as:

Throughput (tokens/sec) – both generated tokens per second and total tokens per second

Latency – time to first token (TTFT) and overall completion latency

Other performance logs – depending on the hardware backend (CPU, GPU, or NPU)

This is the recommended way to benchmark performance across different hardware implementations (e.g., Qualcomm NPU vs. Intel CPU vs. GPU).

I appreciate you flagging this, as it helps ensure that others looking for the same information know that KPIs are indeed available locally.

Could you please take a moment to retake the survey on the above response? Your feedback is greatly appreciated.

Thank you!

using a call like this with --verbose provides kpis: foundry.exe model run qwen2.5-1.5b-instruct-generic-gpu --verbose --prompt "what is the capital of india?"

Hello John Herwig,

Thank you for raising this important question. You've highlighted a critical need for developers: the ability to measure performance KPIs like tokens/second directly from the Windows AI Foundry when comparing different hardware implementations. This is essential for optimizing applications and making informed decisions about deploying AI models on devices with varying capabilities, such as those with Qualcomm NPUs versus Intel CPUs.

Based on the current information, your observation is correct. The local development experience with what you're referring to as Windows AI Foundry does not currently provide built-in, out-of-the-box KPIs like tokens/second in its output.

However, the broader Azure AI Foundry platform (the cloud-based service) does extensively track and publish these types of performance metrics for models in its catalog. This indicates that Microsoft recognizes the importance of these KPIs.

Performance KPIs in Azure AI Foundry

In the Azure AI Foundry portal, models are benchmarked against several performance metrics. These are the key indicators used, which you would want to replicate in your local testing :

• Throughput GTPS (Generated Tokens Per Second): This is the number of output tokens generated per second, measuring the model's generation speed. This is a primary indicator of performance.

Throughput TTPS (Total Tokens Per Second): This measures the total tokens processed per second, including both the input prompt and the generated output.

Latency TTFT (Time to First Token): This is the time it takes for the very first token of the response to be returned after the request is sent. This is crucial for user-perceived responsiveness.

Latency Mean: The average time taken to process a full request.

Throughput Index: A summary metric representing the mean generated tokens per second. Higher values are better.

How to Measure KPIs for Your Local Implementations

Since this functionality isn't built-in for local runs, you will need to implement a simple benchmarking wrapper in your code to calculate these KPIs yourself. Here is a general approach you can take:

Start a High-Precision Timer: Before you make the call to the model for inference, record a start time.

Measure Time to First Token (TTFT): If you are using a streaming API, record the time when the first chunk of data (the first token) is received. The difference between this time and the start time is your TTFT.

Count Generated Tokens: As you receive the output from the model, keep a running count of the number of tokens generated.

Record the End Time: Mark the time when the final token has been received and the generation is complete.

Calculate Tokens per Second:

Calculate the total elapsed time for generation (End Time - Start Time).

  Divide the total number of generated tokens by the total elapsed time in seconds.
  
     **Tokens/sec = (Total Generated Tokens) / (Total Elapsed Time in Seconds)**
     

By running the same prompt and generation task across your different hardware configurations (CPU, NPU) and measuring with this method, you can create your own comparison benchmarks.

Future Plans

You asked about future plans to add these KPIs. While this is a highly valuable feature request, there is no public information available at this time regarding Microsoft's roadmap for integrating these specific performance metrics directly into the local Windows AI Foundry output.

The best course of action is to provide this feedback directly to Microsoft. The Q&A platform where you posted your question is a good start, and you can also use the feedback channels within the tools or on the official GitHub repositories for the relevant SDKs.

Best Regards,

Jerald Felix