Troubleshoot direct OTel observability

Use this guide to verify telemetry ingestion and diagnose problems with agent telemetry sent to Agent 365 directly over OTLP. It's scoped to the direct OTel path - if you're using the Agent 365 SDK or the Microsoft OpenTelemetry Distro, see those guides instead. For wire-level limits, error codes, and silent drop conditions, see Limits and drop conditions.

Verify ingestion

A 200 OK isn't proof of ingestion. Some drop conditions return 200 even though the response's results show that the spans were rejected. Always verify your first runs:

  1. Check HTTP status. 200 → proceed. 4xx → see Common pitfalls.
  2. Inspect results. For each span, confirm that the applicable destination has a status of sent. A status of rejected or not_routed includes a reason.
  3. Parse partialSuccess. This field reports some per-span filtering failures, but a value of 0 doesn't guarantee that the span was routed.
  4. Wait ~5 minutes, then run the Defender advanced-hunting query in the following section.
  5. No row? Use the decision tree under No data in Defender.

Defender advanced-hunting query

The canonical lookup (joining on the agent identity you sent):

let agentIdToFind = "YOUR-AGENT-APP-ID-HERE";
CloudAppEvents
| where Timestamp > ago(1d)
| where ActionType in ("InvokeAgent", "InferenceCall", "ExecuteToolBySDK", "ExecuteToolByGateway", "ExecuteToolByMCPServer")
| extend resData = parse_json(tostring(RawEventData))
| extend AgentId = resData.AgentId
| extend TargetAgentId = resData.TargetAgentId
| extend AlternateId = resData.PlatformTargetAgentId
| where AgentId == agentIdToFind or TargetAgentId == agentIdToFind or AlternateId == agentIdToFind
| project Timestamp, ActionType, resData
| order by Timestamp desc

For the full list of surfaces (Defender agent-activity views, Microsoft 365 admin center, Microsoft Purview) and what each one needs, see Where Agent 365 observability data appears.

No data in Defender

  • partialSuccess.rejectedSpans == totalSpans → all your spans had a bad gen_ai.operation.name. Fix: use one of invoke_agent, execute_tool, chat, output_messages (it's chat, not inference).
  • results shows rejected with reason tenant_not_licensed → the tenant isn't currently eligible. Fix: confirm at least one user in the tenant has a Microsoft 365 E7 or Microsoft Agent 365 license assigned, then check the tenant's eligibility.
  • Spans appear but the run tree is broken or some children are orphaned → missing parentSpanId, different traceId, or gen_ai.conversation.id not set on every span. Fix: review Span hierarchy and run grouping.

Common pitfalls

Symptom Most likely cause Fix
401 Unauthorized Wrong aud on token. Use 9b975845-388f-4429-889e-eab1ef63949c (or api://9b975845-...).
403 Forbidden, missing role or scope Token doesn't carry Agent365.Observability.OtelWrite. Onboard your Microsoft Entra app to the role (S2S) or scope (delegated) per Scopes and consent. For S2S, the token must be acquired with <resource>/.default.
403 Forbidden, agent identity mismatch {agentId} in URL ≠ appid or azp of token, or a span carries a gen_ai.agent.id that doesn't match the authenticated agent. The route agentId must be the appId of the calling app. For blueprint-derived identities, that's the agent identity appId, not the blueprint appId. Ensure every span's gen_ai.agent.id matches.
200 OK but partialSuccess.rejectedSpans == totalSpans All spans had a bad gen_ai.operation.name. Use one of invoke_agent, execute_tool, chat, output_messages. It is chat, not inference.
200 OK with results showing rejected and reason tenant_not_licensed The tenant isn't currently eligible for observability. Confirm at least one user in the tenant has a Microsoft 365 E7 or Microsoft Agent 365 license assigned (the SKU being present isn't enough), then use the optional tenant eligibility check.
Spans appear in CloudAppEvents but the run is missing from Defender agent-activity views and from the Microsoft 365 admin center The run has no invoke_agent span. Both surfaces key off invoke_agent. Emit exactly one invoke_agent span at the root of every run; make chat, execute_tool, and output_messages children of it via parentSpanId.
Run tree is broken or tool spans appear orphaned Missing parentSpanId or different traceId on child spans. See Span hierarchy and run grouping. Every non-root span sets parentSpanId and shares the run's traceId.
Tool spans show empty ChannelName or ConversationId in queries Channel or conversation not set on the tool span, and the parent invoke_agent wasn't in the same OTLP request. Set microsoft.channel.name and gen_ai.conversation.id on every span.
413 Payload Too Large Request body > 1 MB. Split the spans across multiple requests.
429 Too Many Requests Rate limit hit. Honor Retry-After: 1 and back off with jitter.
Agent appears unidentified in dashboards gen_ai.agent.id is empty or not a GUID. Use the agent's Entra appId. If the agent has no Entra registration, see Picking values.

Tenant eligibility preflight

Onboarded third-party S2S integrations can use the optional tenant eligibility endpoint before enabling an integration or sending telemetry. If your integration uses this endpoint, the following guidance can help you interpret its responses.

Symptom Most likely cause Fix
Eligibility returns 200 OK with enabled: false The tenant doesn't currently meet the eligibility requirements. Don't send telemetry. Verify that the tenant meets the prerequisites, and check again after its status changes.
Eligibility returns 403 Forbidden The token lacks Agent365.Observability.OtelWrite, or the token tid doesn't match {tenantId} in the URL. Correct the app-role assignment, tenant consent, or tenant mismatch.
Eligibility returns 429 Too Many Requests The caller exceeded the eligibility request limit. Honor the response's Retry-After value and retry with backoff and jitter.
Eligibility returns a bodyless 503 Service Unavailable Eligibility couldn't be determined. Honor Retry-After: 30 and retry. Don't interpret the missing body as enabled: false.

Next steps

  • Last updated on