AgentEvaluationExtensions.EvaluateAsync Method
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Overloads
EvaluateAsync(AIAgent, IEnumerable<AgentResponse>, IEnumerable<String>, IAgentEvaluator, String, IEnumerable<String>, IEnumerable<IEnumerable<ExpectedToolCall>>, CancellationToken)
Evaluates pre-existing agent responses without re-running the agent.
public static System.Threading.Tasks.Task<Microsoft.Agents.AI.AgentEvaluationResults> EvaluateAsync(this Microsoft.Agents.AI.AIAgent agent, System.Collections.Generic.IEnumerable<Microsoft.Agents.AI.AgentResponse> responses, System.Collections.Generic.IEnumerable<string> queries, Microsoft.Agents.AI.IAgentEvaluator evaluator, string evalName = "AgentFrameworkEval", System.Collections.Generic.IEnumerable<string>? expectedOutput = default, System.Collections.Generic.IEnumerable<System.Collections.Generic.IEnumerable<Microsoft.Agents.AI.ExpectedToolCall>>? expectedToolCalls = default, System.Threading.CancellationToken cancellationToken = default);static member EvaluateAsync : Microsoft.Agents.AI.AIAgent * seq<Microsoft.Agents.AI.AgentResponse> * seq<string> * Microsoft.Agents.AI.IAgentEvaluator * string * seq<string> * seq<seq<Microsoft.Agents.AI.ExpectedToolCall>> * System.Threading.CancellationToken -> System.Threading.Tasks.Task<Microsoft.Agents.AI.AgentEvaluationResults><Extension()>
Public Function EvaluateAsync (agent As AIAgent, responses As IEnumerable(Of AgentResponse), queries As IEnumerable(Of String), evaluator As IAgentEvaluator, Optional evalName As String = "AgentFrameworkEval", Optional expectedOutput As IEnumerable(Of String) = Nothing, Optional expectedToolCalls As IEnumerable(Of IEnumerable(Of ExpectedToolCall)) = Nothing, Optional cancellationToken As CancellationToken = Nothing) As Task(Of AgentEvaluationResults)Parameters
- agent
- AIAgent
The agent (used for tool definitions).
- responses
- IEnumerable<AgentResponse>
Pre-existing agent responses.
- queries
- IEnumerable<String>
The queries that produced each response (must match count).
- evaluator
- IAgentEvaluator
The evaluator to score responses.
- evalName
- String
Display name for this evaluation run.
- expectedOutput
- IEnumerable<String>
Optional ground-truth expected outputs, one per query.
- expectedToolCalls
- IEnumerable<IEnumerable<ExpectedToolCall>>
Optional expected tool calls, one list per query.
- cancellationToken
- CancellationToken
Cancellation token.
Returns
Evaluation results.
Applies to
EvaluateAsync(AIAgent, IEnumerable<AgentResponse>, IEnumerable<String>, IEvaluator, ChatConfiguration, String, IEnumerable<String>, IEnumerable<IEnumerable<ExpectedToolCall>>, CancellationToken)
Evaluates pre-existing agent responses using an MEAI evaluator directly.
public static System.Threading.Tasks.Task<Microsoft.Agents.AI.AgentEvaluationResults> EvaluateAsync(this Microsoft.Agents.AI.AIAgent agent, System.Collections.Generic.IEnumerable<Microsoft.Agents.AI.AgentResponse> responses, System.Collections.Generic.IEnumerable<string> queries, Microsoft.Extensions.AI.Evaluation.IEvaluator evaluator, Microsoft.Extensions.AI.Evaluation.ChatConfiguration chatConfiguration, string evalName = "AgentFrameworkEval", System.Collections.Generic.IEnumerable<string>? expectedOutput = default, System.Collections.Generic.IEnumerable<System.Collections.Generic.IEnumerable<Microsoft.Agents.AI.ExpectedToolCall>>? expectedToolCalls = default, System.Threading.CancellationToken cancellationToken = default);static member EvaluateAsync : Microsoft.Agents.AI.AIAgent * seq<Microsoft.Agents.AI.AgentResponse> * seq<string> * Microsoft.Extensions.AI.Evaluation.IEvaluator * Microsoft.Extensions.AI.Evaluation.ChatConfiguration * string * seq<string> * seq<seq<Microsoft.Agents.AI.ExpectedToolCall>> * System.Threading.CancellationToken -> System.Threading.Tasks.Task<Microsoft.Agents.AI.AgentEvaluationResults><Extension()>
Public Function EvaluateAsync (agent As AIAgent, responses As IEnumerable(Of AgentResponse), queries As IEnumerable(Of String), evaluator As IEvaluator, chatConfiguration As ChatConfiguration, Optional evalName As String = "AgentFrameworkEval", Optional expectedOutput As IEnumerable(Of String) = Nothing, Optional expectedToolCalls As IEnumerable(Of IEnumerable(Of ExpectedToolCall)) = Nothing, Optional cancellationToken As CancellationToken = Nothing) As Task(Of AgentEvaluationResults)Parameters
- agent
- AIAgent
The agent (used for tool definitions).
- responses
- IEnumerable<AgentResponse>
Pre-existing agent responses.
- queries
- IEnumerable<String>
The queries that produced each response (must match count).
- evaluator
- IEvaluator
The MEAI evaluator.
- chatConfiguration
- ChatConfiguration
Chat configuration for the MEAI evaluator.
- evalName
- String
Display name for this evaluation run.
- expectedOutput
- IEnumerable<String>
Optional ground-truth expected outputs, one per query.
- expectedToolCalls
- IEnumerable<IEnumerable<ExpectedToolCall>>
Optional expected tool calls, one list per query.
- cancellationToken
- CancellationToken
Cancellation token.
Returns
Evaluation results.
Applies to
EvaluateAsync(AIAgent, IEnumerable<String>, IAgentEvaluator, String, IEnumerable<String>, IEnumerable<IEnumerable<ExpectedToolCall>>, IConversationSplitter, Int32, CancellationToken)
Evaluates an agent by running it against test queries and scoring the responses.
public static System.Threading.Tasks.Task<Microsoft.Agents.AI.AgentEvaluationResults> EvaluateAsync(this Microsoft.Agents.AI.AIAgent agent, System.Collections.Generic.IEnumerable<string> queries, Microsoft.Agents.AI.IAgentEvaluator evaluator, string evalName = "AgentFrameworkEval", System.Collections.Generic.IEnumerable<string>? expectedOutput = default, System.Collections.Generic.IEnumerable<System.Collections.Generic.IEnumerable<Microsoft.Agents.AI.ExpectedToolCall>>? expectedToolCalls = default, Microsoft.Agents.AI.IConversationSplitter? splitter = default, int numRepetitions = 1, System.Threading.CancellationToken cancellationToken = default);static member EvaluateAsync : Microsoft.Agents.AI.AIAgent * seq<string> * Microsoft.Agents.AI.IAgentEvaluator * string * seq<string> * seq<seq<Microsoft.Agents.AI.ExpectedToolCall>> * Microsoft.Agents.AI.IConversationSplitter * int * System.Threading.CancellationToken -> System.Threading.Tasks.Task<Microsoft.Agents.AI.AgentEvaluationResults><Extension()>
Public Function EvaluateAsync (agent As AIAgent, queries As IEnumerable(Of String), evaluator As IAgentEvaluator, Optional evalName As String = "AgentFrameworkEval", Optional expectedOutput As IEnumerable(Of String) = Nothing, Optional expectedToolCalls As IEnumerable(Of IEnumerable(Of ExpectedToolCall)) = Nothing, Optional splitter As IConversationSplitter = Nothing, Optional numRepetitions As Integer = 1, Optional cancellationToken As CancellationToken = Nothing) As Task(Of AgentEvaluationResults)Parameters
- agent
- AIAgent
The agent to evaluate.
- queries
- IEnumerable<String>
Test queries to send to the agent.
- evaluator
- IAgentEvaluator
The evaluator to score responses.
- evalName
- String
Display name for this evaluation run.
- expectedOutput
- IEnumerable<String>
Optional ground-truth expected outputs, one per query. When provided,
must be the same length as queries. Each value is
stamped on the corresponding ExpectedOutput.
- expectedToolCalls
- IEnumerable<IEnumerable<ExpectedToolCall>>
Optional expected tool calls, one list per query. When provided,
must be the same length as queries. Each list is
stamped on the corresponding ExpectedToolCalls.
- splitter
- IConversationSplitter
Optional conversation splitter to apply to all items. Use LastTurn, Full, or a custom IConversationSplitter implementation.
- numRepetitions
- Int32
Number of times to run each query (default 1). When greater than 1, each query is invoked independently N times to measure consistency. Results contain all N × queries.Count items.
- cancellationToken
- CancellationToken
Cancellation token.
Returns
Evaluation results.
Applies to
EvaluateAsync(AIAgent, IEnumerable<String>, IEnumerable<IAgentEvaluator>, String, IEnumerable<String>, IEnumerable<IEnumerable<ExpectedToolCall>>, IConversationSplitter, Int32, CancellationToken)
Evaluates an agent by running it against test queries with multiple evaluators.
public static System.Threading.Tasks.Task<System.Collections.Generic.IReadOnlyList<Microsoft.Agents.AI.AgentEvaluationResults>> EvaluateAsync(this Microsoft.Agents.AI.AIAgent agent, System.Collections.Generic.IEnumerable<string> queries, System.Collections.Generic.IEnumerable<Microsoft.Agents.AI.IAgentEvaluator> evaluators, string evalName = "AgentFrameworkEval", System.Collections.Generic.IEnumerable<string>? expectedOutput = default, System.Collections.Generic.IEnumerable<System.Collections.Generic.IEnumerable<Microsoft.Agents.AI.ExpectedToolCall>>? expectedToolCalls = default, Microsoft.Agents.AI.IConversationSplitter? splitter = default, int numRepetitions = 1, System.Threading.CancellationToken cancellationToken = default);static member EvaluateAsync : Microsoft.Agents.AI.AIAgent * seq<string> * seq<Microsoft.Agents.AI.IAgentEvaluator> * string * seq<string> * seq<seq<Microsoft.Agents.AI.ExpectedToolCall>> * Microsoft.Agents.AI.IConversationSplitter * int * System.Threading.CancellationToken -> System.Threading.Tasks.Task<System.Collections.Generic.IReadOnlyList<Microsoft.Agents.AI.AgentEvaluationResults>><Extension()>
Public Function EvaluateAsync (agent As AIAgent, queries As IEnumerable(Of String), evaluators As IEnumerable(Of IAgentEvaluator), Optional evalName As String = "AgentFrameworkEval", Optional expectedOutput As IEnumerable(Of String) = Nothing, Optional expectedToolCalls As IEnumerable(Of IEnumerable(Of ExpectedToolCall)) = Nothing, Optional splitter As IConversationSplitter = Nothing, Optional numRepetitions As Integer = 1, Optional cancellationToken As CancellationToken = Nothing) As Task(Of IReadOnlyList(Of AgentEvaluationResults))Parameters
- agent
- AIAgent
The agent to evaluate.
- queries
- IEnumerable<String>
Test queries to send to the agent.
- evaluators
- IEnumerable<IAgentEvaluator>
The evaluators to score responses.
- evalName
- String
Display name for this evaluation run.
- expectedOutput
- IEnumerable<String>
Optional ground-truth expected outputs, one per query.
- expectedToolCalls
- IEnumerable<IEnumerable<ExpectedToolCall>>
Optional expected tool calls, one list per query.
- splitter
- IConversationSplitter
Optional conversation splitter to apply to all items. Use LastTurn, Full, or a custom IConversationSplitter implementation.
- numRepetitions
- Int32
Number of times to run each query (default 1). When greater than 1, each query is invoked independently N times to measure consistency.
- cancellationToken
- CancellationToken
Cancellation token.
Returns
One result per evaluator.
Applies to
EvaluateAsync(AIAgent, IEnumerable<String>, IEvaluator, ChatConfiguration, String, IEnumerable<String>, IEnumerable<IEnumerable<ExpectedToolCall>>, IConversationSplitter, Int32, CancellationToken)
Evaluates an agent using an MEAI evaluator directly.
public static System.Threading.Tasks.Task<Microsoft.Agents.AI.AgentEvaluationResults> EvaluateAsync(this Microsoft.Agents.AI.AIAgent agent, System.Collections.Generic.IEnumerable<string> queries, Microsoft.Extensions.AI.Evaluation.IEvaluator evaluator, Microsoft.Extensions.AI.Evaluation.ChatConfiguration chatConfiguration, string evalName = "AgentFrameworkEval", System.Collections.Generic.IEnumerable<string>? expectedOutput = default, System.Collections.Generic.IEnumerable<System.Collections.Generic.IEnumerable<Microsoft.Agents.AI.ExpectedToolCall>>? expectedToolCalls = default, Microsoft.Agents.AI.IConversationSplitter? splitter = default, int numRepetitions = 1, System.Threading.CancellationToken cancellationToken = default);static member EvaluateAsync : Microsoft.Agents.AI.AIAgent * seq<string> * Microsoft.Extensions.AI.Evaluation.IEvaluator * Microsoft.Extensions.AI.Evaluation.ChatConfiguration * string * seq<string> * seq<seq<Microsoft.Agents.AI.ExpectedToolCall>> * Microsoft.Agents.AI.IConversationSplitter * int * System.Threading.CancellationToken -> System.Threading.Tasks.Task<Microsoft.Agents.AI.AgentEvaluationResults><Extension()>
Public Function EvaluateAsync (agent As AIAgent, queries As IEnumerable(Of String), evaluator As IEvaluator, chatConfiguration As ChatConfiguration, Optional evalName As String = "AgentFrameworkEval", Optional expectedOutput As IEnumerable(Of String) = Nothing, Optional expectedToolCalls As IEnumerable(Of IEnumerable(Of ExpectedToolCall)) = Nothing, Optional splitter As IConversationSplitter = Nothing, Optional numRepetitions As Integer = 1, Optional cancellationToken As CancellationToken = Nothing) As Task(Of AgentEvaluationResults)Parameters
- agent
- AIAgent
The agent to evaluate.
- queries
- IEnumerable<String>
Test queries to send to the agent.
- evaluator
- IEvaluator
The MEAI evaluator (e.g., RelevanceEvaluator, CompositeEvaluator).
- chatConfiguration
- ChatConfiguration
Chat configuration for the MEAI evaluator (includes the judge model).
- evalName
- String
Display name for this evaluation run.
- expectedOutput
- IEnumerable<String>
Optional ground-truth expected outputs, one per query.
- expectedToolCalls
- IEnumerable<IEnumerable<ExpectedToolCall>>
Optional expected tool calls, one list per query.
- splitter
- IConversationSplitter
Optional conversation splitter to apply to all items. Use LastTurn, Full, or a custom IConversationSplitter implementation.
- numRepetitions
- Int32
Number of times to run each query (default 1). When greater than 1, each query is invoked independently N times to measure consistency.
- cancellationToken
- CancellationToken
Cancellation token.
Returns
Evaluation results.