Unable to add tool evaluators to agents or workflow in Microsoft Foundry

Question

Unable to add tool evaluators to agents or workflow in Microsoft Foundry

Anonymous

Hi everyone,

I am encountering two distinct blockers while trying to set up evaluations for an agent in Azure AI Foundry (utilizing an agent connected to a custom MCP server and an Azure AI Search index).

Issue 1: "Failed to save continuous evaluation settings" for Tool Evaluators

When navigating to my agent under Continuous Evaluations (Production/Continuous Monitoring), I can successfully add and save behavioral evaluators (like Coherence, Customer Satisfaction, and Intent Resolution).

However, whenever I attempt to add any of the following tool-based metrics:

Tool Call Accuracy Evaluator
Tool Input Accuracy Evaluator
Tool Output Utilization Evaluator

The portal UI instantly throws the error: Failed to save continuous evaluation settings. Please try again.

Context: My agent relies on an Azure AI Search index and a remote custom MCP server endpoint. Why does the continuous evaluation configuration fail to save specifically when these tool evaluators are selected? Is there a hidden schema or dataset dependency required to register them in production monitoring?

Issue 2: "No runs" in Foundry Portal & Missing Logs in App Insights / LAW

For the baseline evaluators that did successfully save (Coherence, Customer Satisfaction), the Foundry Portal dashboard shows "No runs" across all tabs.

To troubleshoot, I checked the underlying workspace-based Application Insights instance and the linked Log Analytics Workspace (LAW), but no evaluation traces or metrics are appearing there either.

Questions:
1. Do continuous evaluations require a minimum baseline threshold of production traffic or chat interactions before the evaluation pipeline executes and populates the "Runs" tab?
  1. Is there a specific background service, latency period, or diagnostic logging setting required to force the continuous evaluation engine to start piping metrics into App Insights/LAW?
    1. I am concurrently working on uploading a generated custom dataset for a batch evaluation. Is continuous evaluation completely separate from dataset-driven evaluation runs, and if so, how do I kickstart the tracking for it?

Any guidance or troubleshooting steps on resolving the tool evaluator UI crash and locating the missing evaluation telemetry would be greatly appreciated!

Hi everyone,

I am encountering two distinct blockers while trying to set up evaluations for an agent in Azure AI Foundry (utilizing an agent connected to a custom MCP server and an Azure AI Search index).

Issue 1: "Failed to save continuous evaluation settings" for Tool Evaluators

When navigating to my agent under Continuous Evaluations (Production/Continuous Monitoring), I can successfully add and save behavioral evaluators (like Coherence, Customer Satisfaction, and Intent Resolution).

However, whenever I attempt to add any of the following tool-based metrics:

Tool Call Accuracy Evaluator
Tool Input Accuracy Evaluator
Tool Output Utilization Evaluator

The portal UI instantly throws the error: Failed to save continuous evaluation settings. Please try again.

Context: My agent relies on an Azure AI Search index and a remote custom MCP server endpoint. Why does the continuous evaluation configuration fail to save specifically when these tool evaluators are selected? Is there a hidden schema or dataset dependency required to register them in production monitoring?

Issue 2: "No runs" in Foundry Portal & Missing Logs in App Insights / LAW

For the baseline evaluators that did successfully save (Coherence, Customer Satisfaction), the Foundry Portal dashboard shows "No runs" across all tabs.

To troubleshoot, I checked the underlying workspace-based Application Insights instance and the linked Log Analytics Workspace (LAW), but no evaluation traces or metrics are appearing there either.

Questions:
1. Do continuous evaluations require a minimum baseline threshold of production traffic or chat interactions before the evaluation pipeline executes and populates the "Runs" tab?
  1. Is there a specific background service, latency period, or diagnostic logging setting required to force the continuous evaluation engine to start piping metrics into App Insights/LAW?
    1. I am concurrently working on uploading a generated custom dataset for a batch evaluation. Is continuous evaluation completely separate from dataset-driven evaluation runs, and if so, how do I kickstart the tracking for it?

Any guidance or troubleshooting steps on resolving the tool evaluator UI crash and locating the missing evaluation telemetry would be greatly appreciated!

Anshika Varshney 15,625 Reputation points Microsoft External Staff Moderator

2026-07-20T17:01:12.9433333+00:00

Hello @Anonymous

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thankyou!
Anshika Varshney 15,625 Reputation points Microsoft External Staff Moderator

2026-07-21T16:56:29.35+00:00

Hello @Anonymous

If these steps resolve your issue, please mark this answer as Accepted so others can easily find the solution.

If you still face problems, share additional details and we’ll assist further.

Thankyou!

Answer accepted by question author

Anshika Varshney 15,625 Microsoft External Staff Moderator

Hello @Anonymous

Thanks for reaching out.

For Issue 1, the tool-based evaluators (Tool Call Accuracy, Tool Input Accuracy, and Tool Output Utilization) depend on tool invocation telemetry being available and correctly captured. Since your agent uses an Azure AI Search index and a custom remote MCP server, the "Failed to save continuous evaluation settings" error may indicate that the required tool-trace data isn't being detected or that the current agent configuration doesn't meet the prerequisites expected by these evaluators. I recommend checking the browser developer console and network traces for any detailed error messages, and verifying that tool calls are being logged successfully.

For Issue 2, continuous evaluation is separate from dataset-driven batch evaluations. Continuous evaluations rely on actual runtime interactions and telemetry from your deployed agent. If the portal shows**"No runs"**, verify that:

Production traffic is reaching the agent.
Azure AI Foundry tracing/monitoring is enabled.
Application Insights and the linked Log Analytics Workspace are correctly connected and receiving telemetry.
Sufficient interactions have occurred since enabling continuous evaluation.

To help narrow this down, could you share:

The Azure AI Foundry region.
Whether tracing/observability is enabled for the project.
A screenshot of the error shown when saving the tool evaluators.
Any errors captured in the browser's Developer Tools (Network/Console tab).

This information will help determine whether the issue is a configuration limitation, telemetry requirement, or a portal-side problem.

Thankyou!

Anonymous

Hi, my project is in East US but my application insights is in central (this is because of my company infrastructure and can't really be changed). Tracing is enabled for the project and it is succesfully connected to the application insights resource. This is the screenshot of the error:

User's image

3)

This is the error I get when I try to save the setting:

{
  "error": {
    "code": "UserError",
    "severity": null,
    "message": "Evaluation validation failed: missing required data mappings for evaluators 'builtin.tool_input_accuracy'.",
    "messageFormat": "Evaluation validation failed: {errors}",
    "messageParameters": {
      "errors": "missing required data mappings for evaluators 'builtin.tool_input_accuracy'."
    },
    "referenceCode": null,
    "detailsUri": "https://aka.ms/FoundryEvalsFormatIssues",
    "target": "testing_criteria",
    "details": [
      {
        "code": "MissingRequiredDataMapping",
        "severity": null,
        "message": "Data mapping for required field 'tool_definitions' is missing for evaluator 'builtin.tool_input_accuracy'.",
        "messageFormat": null,
        "messageParameters": {},
        "referenceCode": null,
        "detailsUri": null,
        "target": "evaluation.testing_criteria[0].data_mapping",
        "details": [],
        "innerError": null,
        "debugInfo": null,
        "additionalInfo": null
      }
    ],
    "innerError": {
      "code": "EvalValidationFailed",
      "innerError": null
    },
    "debugInfo": null,
    "additionalInfo": null
  },
  "correlation": {
    "operation": "c7ac869f408b4963acf0f676a2edf962",
    "request": "190ddf0caee682c7"
  },
  "environment": "eastus",
  "location": "eastus",
  "time": "2026-07-14T18:53:53.8224943+00:00",
  "componentName": "raisvc",
  "statusCode":

4)And this is the error that I get when I try to run an evaluation with a generated dataset:

{
    "error": {
        "code": "403",
        "message": "A Virtual Network is configured for this resource. Please use the correct endpoint for making requests. Check https://aka.ms/cogsvc-vnet for more details."
    }
}

My foundry project does exist within a private virtual network, but I don't see why the evaluation would be unable to access it. I see traces in my application insights resource but in all of them my agent is unable to see the account, even though I am passing in the necessary auth token.

Anonymous

Hi, my project is in East US but my application insights is in central (this is because of my company infrastructure and can't really be changed). Tracing is enabled for the project and it is succesfully connected to the application insights resource. This is the screenshot of the error:

User's image 3) This is the error I get when I try to save the setting:

{
  "error": {
    "code": "UserError",
    "severity": null,
    "message": "Evaluation validation failed: missing required data mappings for evaluators 'builtin.tool_input_accuracy'.",
    "messageFormat": "Evaluation validation failed: {errors}",
    "messageParameters": {
      "errors": "missing required data mappings for evaluators 'builtin.tool_input_accuracy'."
    },
    "referenceCode": null,
    "detailsUri": "https://aka.ms/FoundryEvalsFormatIssues",
    "target": "testing_criteria",
    "details": [
      {
        "code": "MissingRequiredDataMapping",
        "severity": null,
        "message": "Data mapping for required field 'tool_definitions' is missing for evaluator 'builtin.tool_input_accuracy'.",
        "messageFormat": null,
        "messageParameters": {},
        "referenceCode": null,
        "detailsUri": null,
        "target": "evaluation.testing_criteria[0].data_mapping",
        "details": [],
        "innerError": null,
        "debugInfo": null,
        "additionalInfo": null
      }
    ],
    "innerError": {
      "code": "EvalValidationFailed",
      "innerError": null
    },
    "debugInfo": null,
    "additionalInfo": null
  },
  "correlation": {
    "operation": "c7ac869f408b4963acf0f676a2edf962",
    "request": "190ddf0caee682c7"
  },
  "environment": "eastus",
  "location": "eastus",
  "time": "2026-07-14T18:53:53.8224943+00:00",
  "componentName": "raisvc",
  "statusCode":

4)And this is the error that I get when I try to run an evaluation with a generated dataset:

{
    "error": {
        "code": "403",
        "message": "A Virtual Network is configured for this resource. Please use the correct endpoint for making requests. Check https://aka.ms/cogsvc-vnet for more details."
    }
}

My foundry project does exist within a private virtual network, but I don't see why the evaluation would be unable to access it. I see traces in my application insights resource but in all of them my agent is unable to see the account, even though I am passing in the necessary auth token.

Anshika Varshney 15,625 Reputation points Microsoft External Staff Moderator

2026-07-19T22:26:34.3133333+00:00
Hello @Anonymous

Thank you for sharing the detailed error information.

The error is now more specific and indicates that the Tool Input Accuracy evaluator cannot be configured because a required data mapping is missing:

"MissingRequiredDataMapping"

"Data mapping for required field 'tool_definitions' is missing for evaluator 'builtin.tool_input_accuracy'"

This suggests that the evaluator is expecting access to the tool schema/definitions used by the agent, but those definitions are not being supplied in the evaluation configuration. Tool-based evaluators such as Tool Call Accuracy, Tool Input Accuracy, and Tool Output Utilization typically require additional tool metadata beyond the conversation traces, including the tool definitions that describe available tools, inputs, and expected behavior.

Could you please confirm:

How the evaluation is being configured (Portal, SDK, or REST API)?

Whether your generated dataset contains a mapping for tool_definitions?

If using a custom MCP server, can you verify that tool schemas are exposed and available to the evaluation process?

For reference, if configuring evaluations through JSON/SDK, the evaluator generally expects a data mapping similar to:

{ "data_mapping": { "query": "${data.query}", "response": "${data.response}", "tool_calls": "${data.tool_calls}", "tool_definitions": "${data.tool_definitions}" } }

The exact schema may vary depending on the evaluator type, but the validation error indicates that the required tool_definitions field is currently not being provided.

Regarding the second issue, the 403 response:

"A Virtual Network is configured for this resource. Please use the correct endpoint for making requests."

appears separate from the evaluator validation failure. It suggests that the evaluation service is unable to reach one or more resources through the expected network path when running the evaluation job. Since the Foundry project is deployed inside a private VNet, it would be helpful to confirm whether the evaluation service, Azure AI Search resource, and any custom MCP endpoints are reachable through the configured private networking path.

Could you share how the evaluation dataset is structured (with any sensitive information removed)? Seeing the dataset schema would help determine the correct tool_definitions mapping required by the evaluator.

Thankyou!

1 additional answer

Your answer

Anshika Varshney 15,625 Reputation points Microsoft External Staff Moderator

2026-07-20T17:01:12.9433333+00:00

Hello @Anonymous

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thankyou!
Anshika Varshney 15,625 Reputation points Microsoft External Staff Moderator

2026-07-21T16:56:29.35+00:00

Hello @Anonymous

If these steps resolve your issue, please mark this answer as Accepted so others can easily find the solution.

If you still face problems, share additional details and we’ll assist further.

Thankyou!

Answer 1

Hi, I would treat the save failure and the No runs result as two separate checks. Tool Call/Input/Output evaluators need traces that contain the expected tool-call fields; they cannot evaluate ordinary chat rows with no tool span. First confirm Application Insights tracing is enabled on the same Foundry project and that a fresh agent run shows tool spans and inputs/outputs in observability. Then create a small batch evaluation from an explicit dataset to validate the evaluator independently of continuous evaluation. Continuous evaluation is asynchronous and only processes new eligible traffic after configuration—it does not backfill old runs. If tool traces exist but the evaluator still will not save, capture the browser request/correlation ID and open a Foundry support case, because that points to project/preview backend state rather than the agent prompt.