Agent Evaluation: every test case routed to a connected Fabric data agent errors ("Something went wrong") while the same questions work in the Test pane
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 28.000000000000004%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Gagandeep Singh
0
Reputation points
Setup: Copilot Studio agent with generative orchestration. Tools: a published Microsoft Fabric data agent added as a connected agent (User authentication, shared_fabricdataagent connector), the Fabric IQ MCP (Preview) tool, and one knowledge file. UK region.
Issue: In Agent Evaluation (single-response test set), every test case whose question the orchestrator routes to the connected Fabric data agent fails. In the classic experience the exported CSV contained a silently empty actualResponse; in the new agent experience (September 2026) each such case shows an explicit per-case Error - "Something went wrong while evaluating this test case" - with an empty agent response, and both test methods record Error.
Latest reproduction (September 2026): 24-case test set, 24/24 processed in about 6 minutes, 19 errored. The 19 are exactly the cases that route to the connected Fabric data agent. The other 5 behave correctly in the same run, including a case answered with live data via the Fabric IQ MCP tool - so tool connections work under the same identity. The evaluation ran under the signed-in maker's profile with all connections connected.
Control: the identical questions, asked in the Test pane by the same user minutes earlier, are answered correctly by the connected agent (typical latency 20-60 s). Reproduced across three different connected Fabric data agents and on multiple days (10 August, three runs, classic experience; 9 September, new experience).
Expected: connected-agent responses recorded and graded, or a specific error explaining why the invocation failed.
Is this a known limitation of Agent Evaluation with connected Fabric data agents, and is there a workaround (for example a supported way for the evaluation harness to invoke connected agents, or a timeout setting)?
Microsoft Copilot | Other
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 21%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAP%3C/text%3E%3C/svg%3E)
Ashlesha Pawar (Tata Consultancy Services Limited) 395 Reputation points Microsoft External Staff2026-09-10T04:42:42.6266667+00:00 Hello Gagandeep Singh
This is a documented current limitation, rather than an authentication or timeout configuration issue.
Microsoft Learn states explicitly that Agent Evaluation currently doesn’t support Fabric data agents:
This explains why the same questions succeed in the Test pane but fail when the evaluation harness routes them to the connected Fabric data agent. The evaluation user profile authenticates supported knowledge sources and tools, but it doesn’t add support for Fabric data-agent invocation.
The successful Fabric IQ MCP case doesn’t contradict this limitation. Fabric IQ MCP is invoked as a tool, whereas the Fabric data agent is invoked as a connected agent through an additional orchestration layer.
There is currently:
- No documented timeout setting for individual evaluation cases.
- No supported option that enables Fabric data agents in Agent Evaluation.
- No REST API or Power Automate workaround; those interfaces trigger the same evaluation service and inherit its limitations.
For now, test Fabric data-agent scenarios manually through the Test pane and activity map. Where functionally appropriate, using the Fabric IQ MCP tool is an evaluatable alternative.
Sign in to comment
Sign in to answer