Foundry AI Observability Starter Kit with Already Existing Workflow

Hello Abanes, Cameron A,

Greetings! Thanks for raising this question in Q&A forum.

Your understanding is exactly right. The AI Observability Starter Kit is intentionally structured as a reference implementation or a "known-good baseline" rather than a plug-in. The starter kit ships the wiring including an instrumented agent, evaluators bound to the right trace fields, a red-team taxonomy, an importable Grafana dashboard, and post-deploy checks in one command, so you start from a known-good baseline instead of assembling one. To use it with your own agent, you do not swap out the kit's agents for yours in the same repo. Instead, you replicate the two key things the kit demonstrates into your own workflow: the environment variable flags and the Application Insights connection.

Here is exactly what you need to do to bring observability to your existing portal-based, YAML-configured agent.

Step 1: Connect an Application Insights resource to your Foundry project.

This is the backend where all your traces will flow. Open your Foundry project. In the left navigation, select Tracing. To connect an existing Application Insights resource, select the resource and then select Connect. To create a new resource, select Create new and complete the wizard. A confirmation message appears when the connection succeeds. Once connected, your project is ready to receive traces.

Step 2: Add two environment variables to your existing agent's YAML configuration.

This is the critical step that makes your agent emit OpenTelemetry spans automatically, with no code changes required. ENABLE_INSTRUMENTATION=true activates OpenTelemetry child spans (one span per LLM call and per tool execution) for every chat model call and tool invocation. Without it, only the parent invoke_agent span is emitted and the Agents pane stays empty. ENABLE_SENSITIVE_DATA=true captures full prompt and response text on the spans, which evaluators need to score response quality.

In your existing agent YAML file, add these two lines under environment_variables:

kind: hosted
name: your-existing-agent-name
protocols:
  - protocol: responses
    version: 1.0.0
resources:
  cpu: '0.25'
  memory: '0.5Gi'
environment_variables:
  - name: AZURE_AI_MODEL_DEPLOYMENT_NAME
    value: ${MODEL_DEPLOYMENT_NAME}
  - name: ENABLE_INSTRUMENTATION
    value: "true"
  - name: ENABLE_SENSITIVE_DATA
    value: "true"

That is all that is required. Your existing agent will now emit the same OpenTelemetry spans that the starter kit's demo agent emits, without any other changes.

Step 3: View your traces in the Foundry portal.

Once your agent is redeployed with those environment variables, in your Foundry project, go to the Traces tab in your agents or workflows. You can search, filter, or sort ingested traces from the last 90 days. Select a trace to step through each span, identify issues, and observe how your application responds.

You can also access the built-in Agents pane. The Agents (Preview) pane in Azure Application Insights is the fastest way to see agent health. It populates automatically from the OpenTelemetry spans your agent emits, with no dashboard import or configuration required. Access it from Application Insights > left menu > Agents (Preview).

Step 4: Optionally import the Grafana dashboards from the starter kit.

If you want the same rich Grafana dashboards that the starter kit provides, you do not need to run the full starter kit pipeline. The kit ships two importable dashboard JSON files in artifacts/grafana/. The primary dashboard (agent-observability-dashboard.json) provides a full operational overview. A companion dashboard (agent-observability-custom-dashboard.json) adds five focused panels for deeper investigation including per-tool p95 latency, error rate, and session counts. You can download these JSON files from the starter kit GitHub repo at github.com/jvargh/ai-observability-starter-kit and import them into your own Grafana instance pointed at your Application Insights data source, completely independently of the rest of the kit.

Regarding the VS Code Foundry Toolkit:

The toolkit supports AI frameworks such as Foundry Agents Service through OpenTelemetry. You can see traces instantly in VS Code without needing cloud access. So your existing VS Code-based workflow is already compatible with the observability layer. You do not need to switch to a code-first agent approach. Your YAML-configured agent is a hosted Foundry agent and the ENABLE_INSTRUMENTATION flag is the standard way to activate tracing for it.

If this answer helps you kindly accept the answer which will help others who have similar questions.

Best Regards,

Jerald Felix.