Published Agent Behaves Differently from Agent Test Environment

Differences between the test experience and published channels are expected in some cases, and several limitations and configuration issues can cause the behavior described.

Key points from current guidance:

1. Differences between test and published experiences

• In Microsoft 365 Copilot’s Agent Builder, the Try it (test) experience is designed for rapid iteration and reflects the current authoring state of the agent as instructions and knowledge are adjusted. It behaves like a published agent but is still an authoring-time sandbox.
• Some features available in the full Microsoft 365 Copilot or Teams experience are not available in the Try it pane (for example, sharing prompts, feedback, @mentions), and conversely, some runtime limitations apply only after publishing (for example, streaming and citations for agents published to Microsoft 365 are not supported).
• For agents published from Azure AI Foundry to Microsoft 365 or Teams, there are explicit limitations at runtime, including:
  • No file uploads or image generation in Microsoft 365 for published agents.
  • No streaming responses or citations for published agents.

2. Publishing and runtime issues that can cause mismatched behavior

• For agents published to Microsoft 365 or Teams, publishing can fail or partially apply if:
  • The agent identity metadata is invalid or missing (for example, agent.identity is null or not unique).
  • The Azure Bot Service resource cannot be created or updated due to missing permissions (Microsoft.BotService/botServices/write) or an unregistered provider.
  • Organization-scope agents are awaiting admin approval in the Microsoft 365 admin center.
  • The agent identity lacks RBAC permissions to the Azure resources it calls.
• When these issues occur, the runtime may:
  • Fail to respond in some channels.
  • Not reflect the latest configuration, because the updated version was not successfully published or the bot resource was not updated.

3. Knowledge and SharePoint-specific behavior

• For Copilot Studio generative answers over SharePoint:
  • Only the top three SharePoint search results are used to generate a response. If no results are returned, the node returns a fallback such as “I’m not sure how to help with that. Can you try rephrasing?”
  • Only supported file formats and modern SharePoint pages are used; unsupported content or non-indexed content will not be surfaced.
  • If SharePoint search results are missing (for example, due to indexing delay or search configuration), the published agent will not return knowledge answers even though the test environment appears to work when pointed at different or smaller test content.
• For knowledge sources in Copilot Studio more broadly:
  • Knowledge must be explicitly enabled in generative answers nodes.
  • The maximum number of knowledge sources per node must not be exceeded.
  • Content moderation settings can block or filter responses.
  • For some scenarios, agents may rely on Dataverse or SharePoint knowledge that is still being updated or indexed, which can cause a difference between what is seen in test versus published.

4. Known behaviors and limitations between test and published channels

• For agents published to Microsoft 365 Copilot and Teams:
  • File uploads and image generation are not available in Microsoft 365 for published agents, even if similar capabilities appear in other environments.
  • Streaming responses and citations are not supported for published agents.
• For Copilot Studio agents using knowledge sources:
  • Published agents use the configured knowledge sources to ground responses. If knowledge is only configured at the agent level (for example, in the Employee Self-Service template), topic-level knowledge nodes are not present by default.
  • A known issue exists where a published agent might use the latest unpublished updates of a Dataverse knowledge source, which can cause unexpected or inconsistent behavior between test and published runs.

5. Verifying that the latest published version is running

• Confirm that the agent has been successfully published and that no publishing errors occurred:
  • Ensure the agent has a unique identity and that agent.identity is not null.
  • Confirm that the Azure Bot Service resource was created or updated successfully and that Microsoft.BotService is registered.
  • Check for 403 AuthorizationFailed errors on Microsoft.BotService/botServices/write; if present, assign the Azure Bot Service Contributor role on the resource group and republish.
• For organization-scope agents:
  • Verify that admin approval has been granted in the Microsoft 365 admin center and that app policies allow the agent.
• If the agent appears in Microsoft 365 Copilot but behaves differently:
  • Confirm that the correct scope (individual vs organization) and correct environment are being used.
  • Republish after any significant configuration change and test again in the target channel.

6. Troubleshooting steps and diagnostics

A practical troubleshooting path based on current guidance:

1. Validate knowledge configuration
   • In Copilot Studio, verify that:
     • Knowledge sources are enabled in the relevant generative answers nodes.
     • The node does not exceed the supported number of knowledge sources.
     • Content moderation is set appropriately and not blocking expected responses.
   • For SharePoint-based knowledge:
     • Confirm that the generative answers node points to a SharePoint site or library with relevant content.
     • Ensure documents are in supported formats and are modern pages where applicable.
     • If content was recently uploaded, allow time for indexing and verify that SharePoint search returns the expected items.
2. Check for known SharePoint and generative answers issues
   • If the agent returns fallback messages like “I’m not sure how to help with that. Can you try rephrasing?”, treat this as a sign that SharePoint search returned no usable results.
   • Review SharePoint search configuration and troubleshoot missing search results if necessary.
3. Confirm publishing and channel configuration
   • Republish the agent and watch for any errors related to:
     • Agent identity (agent.identity must be unique and non-null).
     • Azure Bot Service creation or update.
     • RBAC permissions for the agent’s identity on any Azure resources it accesses.
   • For Teams or Microsoft 365 Copilot:
     • Ensure that the agent appears in the target channel and that admin approval is complete.
4. Test across apps and channels
   • Add the agent to multiple Microsoft 365 apps (Word, Excel, Teams, Outlook) and verify behavior in each.
   • Differences in how responses or suggested actions render in each app can surface channel-specific issues early.
5. Validate outputs and behavior
   • Spot-check published responses against the underlying knowledge sources and topic logic:
     • Confirm that summaries and citations align with the source content.
     • Ensure that for factual queries, the agent prefers the configured knowledge sources.
   • If the agent produces answers that were “never configured or trained,” verify whether general AI knowledge is being used instead of knowledge sources and adjust configuration (for example, using options like Only use specified sources where applicable in Agent Builder).
6. Use test and peer validation
   • Continue to use the built-in test chat (Try it) frequently during iteration.
   • Have peers test the published agent in Copilot and Teams to surface unexpected phrasing or flows that may not match the designed topics and quick replies.

If, after these steps, the published agent still diverges significantly from the test behavior, capturing specific examples (prompts, channel, time, and observed vs expected responses) and raising a support case is recommended so that product support can investigate runtime synchronization or channel-specific issues.

References:

• Best practices for building declarative agents
• Test your agent's knowledge sources
• Add knowledge sources to your declarative agent in Microsoft 365 Copilot
• Generative answers pointing to SharePoint sources don't return results in Copilot Studio
• Customize the Employee Self-Service agent
• Configure and deploy agents from the Agent Library
• Build agents by using Agent Builder in Microsoft 365 Copilot
• Publish agents to Microsoft 365 Copilot and Microsoft Teams
• Copilot Studio Agent will not read entire data set in Excel or markdown - Microsoft Q&A
• Copilot Studio Agent doesn't recognize the docs in the Knowledge Base. - Microsoft Q&A