Integrate agent observability using direct OTel

Learn how to integrate agent observability with Agent 365 by sending telemetry directly over OpenTelemetry (OTLP/HTTP+JSON). This approach helps agents that can't use the Agent 365 SDK or Microsoft OpenTelemetry Distro send telemetry efficiently and securely. Before starting, read Agent 365 observability concepts to understand the model, authentication flows, and where your data appears.

Prerequisites

Ensure the following configurations are in place before any telemetry flows.

Authentication recipes

All four recipes use the standard Microsoft Entra token endpoint:

The following recipes show raw HTTP for clarity. In production, use Microsoft.Identity.Web or another MSAL library, which handles token refresh and caching.

Which recipe do I need?

S2S, standard Microsoft Entra app

Send a POST request to the tenant's token endpoint with grant_type=client_credentials. Authenticate the app by using a client secret, a certificate (signed JWT assertion), or a managed identity or federated credential.

POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

client_id={your-app-id}
&scope=9b975845-388f-4429-889e-eab1ef63949c%2F.default
&client_secret={secret}
&grant_type=client_credentials

The returned token has appid/azp = {your-app-id}, roles containing Agent365.Observability.OtelWrite, and aud = 9b975845-.... Use it on the /observabilityService/.../traces route.

For certificate-based authentication, replace client_secret={secret} with client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer&client_assertion={signed-jwt}.

S2S, blueprint-derived agent identity

Agent identities don't have credentials of their own. The agent identity blueprint holds the credentials (managed identity FIC, certificate, or client secret) and mints tokens on behalf of its child agent identities through a two-step exchange. For more information, see autonomous app OAuth flow.

1. The blueprint authenticates and gets a federated identity exchange token T1:

   • {blueprint-credential} is the blueprint's MSI token, certificate-signed JWT, or secret exchange-token assertion - per blueprint configuration.

   POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
   Content-Type: application/x-www-form-urlencoded
   
   client_id={blueprint-app-id}
   &scope=api%3A%2F%2FAzureADTokenExchange%2F.default
   &fmi_path={agent-identity-app-id}
   &client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
   &client_assertion={blueprint-credential}
   &grant_type=client_credentials
   

2. The agent identity exchanges T1 for the Agent 365 Observability resource token:

   POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
   Content-Type: application/x-www-form-urlencoded
   
   client_id={agent-identity-app-id}
   &scope=9b975845-388f-4429-889e-eab1ef63949c%2F.default
   &client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
   &client_assertion={T1}
   &grant_type=client_credentials
   

   • The returned token has appid/azp = {agent-identity-app-id}, roles containing Agent365.Observability.OtelWrite, and aud = 9b975845-....
   • Use this token on the /observabilityService/.../traces route.
   • The URL {agentId} is the agent identity appId, not the blueprint appId.

OBO, standard Microsoft Entra app

Receive the user's incoming token Tc from your upstream caller (Bearer or PFAT), then exchange it:

POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

client_id={your-app-id}
&scope=9b975845-388f-4429-889e-eab1ef63949c%2FAgent365.Observability.OtelWrite
&client_secret={secret}
&grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer
&assertion={Tc}
&requested_token_use=on_behalf_of

For certificate auth, replace client_secret={secret} with the same client_assertion_type + client_assertion pair as in S2S.

The returned token has appid/azp = {your-app-id}, scp containing Agent365.Observability.OtelWrite, and aud = 9b975845-.... Use it on the /observability/.../traces route. A refresh token is returned alongside; cache and reuse it instead of rerunning the exchange on every call.

OBO, blueprint-derived agent identity (including AI teammate)

There are three main steps to the on-behalf-of flow. For more information, see Agent OAuth flows: On behalf of flow.

1. Receive the user token Tc. For an AI teammate, this token represents the agent's own user account; otherwise, it represents the human caller.

2. The blueprint authenticates and gets T1, the same as the S2S blueprint-derived agent identity flow.

3. The agent identity exchanges T1 and Tc for a delegated resource token:

   POST https://login.microsoftonline.com/{your-tenant-id}/oauth2/v2.0/token
   Content-Type: application/x-www-form-urlencoded
   
   client_id={agent-identity-app-id}
   &scope=9b975845-388f-4429-889e-eab1ef63949c%2FAgent365.Observability.OtelWrite
   &client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
   &client_assertion={T1}
   &grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer
   &assertion={Tc}
   &requested_token_use=on_behalf_of
   

The returned token has appid/azp = {agent-identity-app-id}, scp containing Agent365.Observability.OtelWrite, and represents the agent's user. Use it on the /observability/.../traces route. The URL {agentId} is the agent identity appId, not the blueprint appId. A refresh token is returned alongside; cache and reuse it.

Required claims on the returned token

S2S route (/observabilityService/...) - app-only token:

Delegated route (/observability/...) - user-delegated token (Bearer or PFAT):

The delegated route accepts both Bearer and MSAuth1.0 PFAT tokens. Direct callers should use Bearer. If you don't know which one you have, use Bearer.

Endpoints

Two routes; pick by how your service authenticates, not by what the user is doing:

POST https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1   # S2S
POST https://agent365.svc.cloud.microsoft/observability/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1          # OBO

Headers:

Authorization: Bearer <token>      # or MSAuth1.0 ... for delegated PFAT
Content-Type: application/json

URL parameters

• {tenantId} - the customer tenant GUID. The server treats this value as authoritative. If your spans set microsoft.tenant.id and it doesn't match, the server rejects the request.
• {agentId} - the calling application's appId (also the OAuth client_id). For blueprint-derived identities, this value is the agent identity appId, not the blueprint appId. It must match the appid or azp claim of your token.
• api-version=1 - required.

Check tenant eligibility

Onboarded third-party integrations that use the S2S authentication model can check whether a customer tenant is eligible for Agent 365 observability before enabling an integration or sending telemetry. This check can help integrations avoid sending telemetry for tenants that aren't currently eligible.

Use the same app-only token described in S2S authentication. The token must contain the Agent365.Observability.OtelWrite app role, and its tid claim must match {tenantId} in the request URL.

GET https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/eligibility?api-version=1
Authorization: Bearer <access-token>

A successful response contains the eligibility result:

{
  "enabled": true
}

Handle the response as follows:

Request body encoding

The body uses the standard OTLP/HTTP+JSON format: an ExportTraceServiceRequest with resourceSpans → scopeSpans → spans. Keep in mind the following details:

• Send traceId (16 bytes) and spanId (8 bytes) as lowercase hex strings.
• startTimeUnixNano and endTimeUnixNano are strings that hold Unix epoch nanoseconds.
• kind is the integer OTLP enum value (for example, 1 for INTERNAL). status.code is the integer enum (for example, 1 for OK, 2 for ERROR).
• Send all attribute values as stringValue.

Response shape

A 200 OK response means that Agent 365 processed the request. It doesn't guarantee that every span was routed to a destination. Inspect both partialSuccess and results.

The results array reports the outcome for each span at each applicable destination:

• sent - the span was routed to the destination.
• rejected - the span wasn't routed. The reason field explains why.
• not_routed - the destination wasn't selected for the span. The reason field explains why.

For example, a successfully routed span can return:

{
  "partialSuccess": {
    "rejectedSpans": 0,
    "errorMessage": ""
  },
  "results": [
    {
      "spanId": "0123456789abcdef",
      "sinks": {
        "flashpoint": {
          "status": "sent"
        },
        "sentinel": {
          "status": "sent"
        },
        "esp": {
          "status": "sent"
        }
      }
    }
  ]
}

If the tenant isn't eligible, the request can still return 200 OK. In that case, results shows that the spans were rejected:

{
  "partialSuccess": {
    "rejectedSpans": 0,
    "errorMessage": ""
  },
  "results": [
    {
      "spanId": "0123456789abcdef",
      "sinks": {
        "flashpoint": {
          "status": "rejected",
          "reason": "tenant_not_licensed"
        },
        "sentinel": {
          "status": "rejected",
          "reason": "tenant_not_licensed"
        },
        "esp": {
          "status": "rejected",
          "reason": "tenant_not_licensed"
        }
      }
    }
  ]
}

For whole-request routing decisions, partialSuccess.rejectedSpans can remain 0 even when results shows that every span was rejected. Don't use partialSuccess or the HTTP status alone as proof of ingestion. Field names are camelCase on the wire. See Limits and drop conditions for other reasons telemetry might not appear.

Smallest possible request

The simplest end-to-end test sends a single invoke_agent span. This span is the smallest body that lands in Microsoft Defender.

Step 1. Get a Bearer token. For S2S, use client credentials with scope 9b975845-388f-4429-889e-eab1ef63949c/.default (see Authentication recipes for the full recipe).

Step 2. POST a single span:

TOKEN="$(./get-token.sh)"
TENANT_ID="<customer-tenant-guid>"
AGENT_ID="<your-agent-app-id>"

curl -i -X POST \
  "https://agent365.svc.cloud.microsoft/observabilityService/tenants/${TENANT_ID}/otlp/agents/${AGENT_ID}/traces?api-version=1" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @- <<EOF
{
  "resourceSpans": [{
    "scopeSpans": [{
      "scope": { "name": "my-instrumentation", "version": "1.0.0" },
      "spans": [{
        "traceId": "0102030405060708090a0b0c0d0e0f10",
        "spanId":  "1111111111111111",
        "parentSpanId": "",
        "name": "invoke_agent",
        "kind": 1,
        "startTimeUnixNano": "1736175600000000000",
        "endTimeUnixNano":   "1736175601500000000",
        "status": { "code": 1 },
        "attributes": [
          { "key": "gen_ai.operation.name", "value": { "stringValue": "invoke_agent" } },
          { "key": "gen_ai.agent.id",       "value": { "stringValue": "${AGENT_ID}" } },
          { "key": "gen_ai.agent.name",     "value": { "stringValue": "MyAgent" } },
          { "key": "microsoft.a365.agent.blueprint.id", "value": { "stringValue": "${AGENT_ID}" } },
          { "key": "gen_ai.conversation.id","value": { "stringValue": "conv-001" } },
          { "key": "microsoft.channel.name","value": { "stringValue": "web" } },
          { "key": "user.id",               "value": { "stringValue": "<entra-user-objectid>" } },
          { "key": "client.address",        "value": { "stringValue": "10.1.2.80" } },
          { "key": "server.address",        "value": { "stringValue": "myagent.example.com" } },
          { "key": "server.port",           "value": { "stringValue": "443" } },
          { "key": "gen_ai.input.messages", "value": { "stringValue": "[{\"role\":\"user\",\"content\":\"hi\"}]" } },
          { "key": "gen_ai.output.messages","value": { "stringValue": "[{\"role\":\"assistant\",\"content\":\"hello\"}]" } }
        ]
      }]
    }]
  }]
}
EOF

Step 3. Expect 200 OK, then inspect results as described in Response shape. Confirm that the applicable destinations have a status of sent.

Step 4. Confirm the data actually landed. A 200 OK isn't proof of ingestion; see Verifying ingestion for the verification flow. To POST a saved body file instead, replace --data @- <<EOF ... EOF with --data @./otlp-request.json.

Agent run example

A user on Microsoft Teams asks "What's the weather in Seattle?". Your agent calls a GetWeather function, asks an LLM to format the answer, and replies. That single run is four spans:

graph TD
    A["<b>invoke_agent</b> · spanId=A · parentSpanId=∅<br/><i>root - the run itself</i>"]
    B["<b>chat</b> · spanId=B · parentSpanId=A<br/><i>LLM picks the tool / formats reply</i>"]
    C["<b>execute_tool</b> · spanId=C · parentSpanId=A<br/><i>the GetWeather call</i>"]
    D["<b>output_messages</b> · spanId=D · parentSpanId=A<br/><i>final reply emitted to the user</i>"]
    A --> B
    A --> C
    A --> D

Run-wide attributes set on every span:

Span A: invoke_agent (root)

{
  "traceId": "0102030405060708090a0b0c0d0e0f10",
  "spanId": "1111111111111111",
  "parentSpanId": "",
  "name": "invoke_agent",
  "kind": 1,
  "startTimeUnixNano": "1736175600000000000",
  "endTimeUnixNano":   "1736175601500000000",
  "status": { "code": 1 },
  "attributes": [
    { "key": "gen_ai.operation.name",   "value": { "stringValue": "invoke_agent" } },
    { "key": "gen_ai.execution.type",   "value": { "stringValue": "HumanToAgent" } },
    { "key": "gen_ai.input.messages",   "value": { "stringValue": "[{\"role\":\"user\",\"content\":\"What's the weather in Seattle?\"}]" } },
    { "key": "gen_ai.output.messages",  "value": { "stringValue": "[{\"role\":\"assistant\",\"content\":\"It's 65F and partly cloudy in Seattle.\"}]" } },
    { "key": "user.email",              "value": { "stringValue": "alice@contoso.com" } }
    /* plus all the run-wide attributes listed above */
  ]
}

Span B: chat (LLM call)

{
  "traceId": "0102030405060708090a0b0c0d0e0f10",
  "spanId": "2222222222222222",
  "parentSpanId": "1111111111111111",
  "name": "chat",
  "kind": 1,
  "startTimeUnixNano": "1736175600200000000",
  "endTimeUnixNano":   "1736175600900000000",
  "status": { "code": 1 },
  "attributes": [
    { "key": "gen_ai.operation.name",      "value": { "stringValue": "chat" } },
    { "key": "gen_ai.request.model",       "value": { "stringValue": "gpt-4o" } },
    { "key": "gen_ai.provider.name",       "value": { "stringValue": "openai" } },
    { "key": "gen_ai.usage.input_tokens",  "value": { "stringValue": "42" } },
    { "key": "gen_ai.usage.output_tokens", "value": { "stringValue": "23" } }
    /* plus all the run-wide attributes */
  ]
}

Span C: execute_tool

{
  "traceId": "0102030405060708090a0b0c0d0e0f10",
  "spanId": "3333333333333333",
  "parentSpanId": "1111111111111111",
  "name": "execute_tool",
  "kind": 1,
  "startTimeUnixNano": "1736175600950000000",
  "endTimeUnixNano":   "1736175601200000000",
  "status": { "code": 1 },
  "attributes": [
    { "key": "gen_ai.operation.name",      "value": { "stringValue": "execute_tool" } },
    { "key": "gen_ai.tool.name",           "value": { "stringValue": "GetWeather" } },
    { "key": "gen_ai.tool.type",           "value": { "stringValue": "function" } },
    { "key": "gen_ai.tool.call.id",        "value": { "stringValue": "call-001" } },
    { "key": "gen_ai.tool.call.arguments", "value": { "stringValue": "{\"location\":\"Seattle\"}" } },
    { "key": "gen_ai.tool.call.result",    "value": { "stringValue": "{\"tempF\":65,\"condition\":\"partly cloudy\"}" } }
    /* plus all the run-wide attributes */
  ]
}

Span D: output_messages

{
  "traceId": "0102030405060708090a0b0c0d0e0f10",
  "spanId": "4444444444444444",
  "parentSpanId": "1111111111111111",
  "name": "output_messages",
  "kind": 1,
  "startTimeUnixNano": "1736175601400000000",
  "endTimeUnixNano":   "1736175601500000000",
  "status": { "code": 1 },
  "attributes": [
    { "key": "gen_ai.operation.name",  "value": { "stringValue": "output_messages" } },
    { "key": "gen_ai.output.messages", "value": { "stringValue": "[{\"role\":\"assistant\",\"content\":\"It's 65F and partly cloudy in Seattle.\"}]" } }
    /* plus all the run-wide attributes */
  ]
}

Send telemetry

Use an OTel SDK

Most partners send traces through an OTel SDK rather than hand-rolled HTTP. The SDK handles batching, retry, and the OTLP/HTTP+JSON encoding for you. Set the exporter endpoint and inject the Authorization header.

The exporter endpoint is the route URL itself, including the query string:

https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1

(Use /observability/... instead of /observabilityService/... for the delegated route.)

Python

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1",
    headers={"Authorization": f"Bearer {token}"},
)

Package: opentelemetry-exporter-otlp-proto-http.

Node.js / TypeScript

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

const exporter = new OTLPTraceExporter({
  url: "https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1",
  headers: { Authorization: `Bearer ${token}` },
});

Package: @opentelemetry/exporter-trace-otlp-http.

.NET

using OpenTelemetry.Exporter;

services.AddOpenTelemetry().WithTracing(b => b
    .AddOtlpExporter(o =>
    {
        o.Endpoint = new Uri("https://agent365.svc.cloud.microsoft/observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces?api-version=1");
        o.Headers = $"Authorization=Bearer {token}";
        o.Protocol = OtlpExportProtocol.HttpJson;
    }));

Package: OpenTelemetry.Exporter.OpenTelemetryProtocol.

Manual HTTP

If you can't or don't want to use an OTel SDK, build the OTLP/HTTP+JSON request yourself and POST it. The OpenTelemetry OTLP/HTTP+JSON spec defines the body shape:

{
  "resourceSpans": [{
    "resource":  { "attributes": [ ... ] },          // optional
    "scopeSpans": [{
      "scope":  { "name": "<your-instrumentation>", "version": "1.0.0" },
      "spans":  [ <span>, <span>, ... ]
    }]
  }]
}

Each <span> is an object whose required fields are traceId, spanId, name, kind, startTimeUnixNano, endTimeUnixNano, attributes, and (for non-root spans) parentSpanId. For encoding rules (string-encoded times, hex traceId / spanId, integer kind / status.code, all attribute values as stringValue), see Endpoints and Request body encoding.

The set of attributes to set on each span is defined in Message contracts. For the full attribute list, see Attribute reference. Refer to the Agent run example for an end-to-end working sample with the Bearer token in the header and the body inline.

You can send all the spans of a run in a single POST body (preferred - one request, one trace) or across multiple POSTs. The server reconstructs the run from traceId + parentSpanId + gen_ai.conversation.id, so each span carries enough to be correlated either way.

Message contracts

This section defines what spans you can emit and which attributes go on each. For the full attribute-by-attribute spec, see the Attribute reference.

Operation types

Every span you send must carry gen_ai.operation.name set to one of these four values (case-insensitive). The server drops any span with a missing or unrecognized value and counts it in partialSuccess.rejectedSpans.

Span hierarchy and run grouping

Agent 365 reconstructs a run from the standard OTLP span graph (traceId, spanId, parentSpanId) plus the run-wide attributes from the Attribute reference.

Six rules:

1. Always set parentSpanId on every non-root span. Without it, you can't reconstruct the run's tree structure.
2. Reuse the same traceId across every span in a run.
3. Set gen_ai.conversation.id on every span with the same value. This value is the primary join key for "all spans in this run". It's not propagated automatically.
4. Set microsoft.channel.name on every span with the same value. Tool spans missing the channel or conversation can inherit them from their parent invoke_agent only if the parent is in the same OTLP request, so set them on every span yourself.
5. Set microsoft.session.id on every span when you have a logical session.
6. For agent-to-agent calls where the child agent is in a separate request, reuse the same gen_ai.conversation.id and use the microsoft.a365.caller.agent.* attributes (see Attribute reference) to capture the caller-agent context.

The four-span tree in the Agent run example is the canonical shape.

Common run shapes

Checklist for production onboarding

Run through this checklist before going to production.

Next steps

• Attribute reference - Per-attribute spec and value-picking guidance.
• Troubleshooting - Verifying ingestion, common pitfalls, and error responses.