Quickstart: Optimize a hosted agent (preview)

• azd CLI (Azure Developer CLI).

• Azure CLI for authentication.

• The microsoft.foundry extension for azd (0.1.40-preview or later of the azure.ai.agents dependency):

  azd ext install microsoft.foundry
  

  If already installed, upgrade:

  azd ext upgrade microsoft.foundry
  

• Azure CLI for authentication.

• Python 3.10 or later.

• The Python packages used in this path:

  pip install "azure-ai-projects>=2.3.0" azure-identity python-dotenv
  

• An existing Foundry project that already contains the hosted agent, registered dataset, and evaluator you want to use for optimization.

Step 1: Create the project

Initialize a new project from the optimization sample template:

mkdir my-agent && cd my-agent
azd ai agent init -m https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/bring-your-own/responses/optimization-customer-support/azure.yaml .

The interactive flow prompts for your Azure subscription, region, and model deployment settings. It adopts azure.yaml for hosted-agent configuration and generates .agent_configs/baseline/, the evaluation dataset, and infrastructure files.

Step 2: Provision and deploy

Authenticate and provision the Azure resources:

az login
azd auth login
azd provision

Provisioning takes approximately two minutes and creates a Foundry account, project, Azure Container Registry, and model deployments.

Deploy the agent:

azd deploy

Test the deployment:

azd ai agent invoke "What is 2+2?"

Step 3: Generate evaluation suite and optimize

Generate an evaluation dataset and evaluators for your agent:

azd ai agent eval generate

This step creates eval.yaml, a test dataset, and scoring evaluators based on your agent's instructions. The optimizer uses these files to measure improvement.

Run the optimizer:

azd ai agent optimize --max-candidates 2

The CLI prompts you to select an optimization model. To skip the prompt, pass it directly:

azd ai agent optimize --max-candidates 2 --optimize-model gpt-5

The CLI detects your agent from azure.yaml and uses the generated eval.yaml automatically. With two candidates, optimization typically completes in about 8 minutes. Real-time progress is shown:

Optimizing agent "customer-support-py"...
  Config: eval.yaml
  Baseline saved to .agent_configs/baseline/metadata.yaml
  Job ID: opt_162bd0f09....
  Status: pending
  Portal: <OPTIMIZATION-JOB-URL>

Use the portal URL to monitor your job in the Foundry portal.

The eval model scores each response (any chat-completion model works). The optimization model (--optimize-model) generates improved candidates and must be from the supported list (gpt-5 family or DeepSeek). You can also set optimization_model under options: in eval.yaml to avoid passing the flag each time.

Step 4: Deploy the winner

The star (*) in the output indicates the best candidate. Apply the optimized config locally, then deploy:

azd ai agent optimize apply --candidate <candidate-id>
azd deploy

The apply command downloads the optimized configuration into .agent_configs/<candidate_id>/ and updates your azure.yaml to use the new instructions. The deploy command pushes the optimized agent live using code deploy.

Invoke your agent to verify the improvement:

azd ai agent invoke "What is your return policy?"

You can also run evaluation to confirm the score improvement:

azd ai agent eval run

Python SDK path

Use the following steps if you want to run the optimizer from Python instead of the Azure Developer CLI workflow described earlier.

This path assumes you already have the following resources in an existing Foundry project:

• A hosted agent to optimize.
• A registered training dataset.
• A registered evaluator.

Unlike the Azure Developer CLI flow described earlier, the Python SDK path doesn't scaffold a project or generate eval.yaml, a dataset, or evaluators for you. If you want the sample to create those assets automatically, use azd ai agent eval generate first.

1. Create a .env file

Create a working folder, and then add a .env file with these values:

FOUNDRY_PROJECT_ENDPOINT=<your-project-endpoint>
FOUNDRY_AGENT_NAME=<your-hosted-agent-name>
DATASET_NAME=<your-registered-dataset-name>
EVALUATOR_NAME=<your-registered-evaluator-name>
DATASET_VERSION=1
POLL_INTERVAL_SECONDS=10
EVAL_MODEL=<your-eval-model-deployment-name>
OPTIMIZATION_MODEL=<your-optimization-model-deployment-name>

Run the script from this same working folder so load_dotenv() can load the .env file automatically. If you prefer to run it from another directory, set the same values in your shell environment first.

Use the exact project endpoint from your Foundry project's Overview page. The Python script sends its first request immediately. If FOUNDRY_PROJECT_ENDPOINT is only a placeholder or points to the wrong project, the run fails with ResourceNotFound: The project does not exist.

Set EVAL_MODEL and OPTIMIZATION_MODEL to deployment names that already exist in your Foundry project, not just model family names. For example, if your project deployment is named gpt-4.1-mini or DeepSeek-V3.2, use that exact deployment name in .env.

2. Run the optimization job

Create a file named optimize_hosted_agent.py in the same folder as .env:

import os
import time

from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
  JobStatus,
  OptimizationAgentIdentifier,
  OptimizationEvaluatorRef,
  OptimizationJob,
  OptimizationJobInputs,
  OptimizationOptions,
  OptimizationReferenceDatasetInput,
)
from azure.identity import DefaultAzureCredential
from dotenv import load_dotenv

load_dotenv()

endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
agent_name = os.environ["FOUNDRY_AGENT_NAME"]
dataset_name = os.environ["DATASET_NAME"]
evaluator_name = os.environ["EVALUATOR_NAME"]
dataset_version = os.environ.get("DATASET_VERSION", "1")
poll_interval = int(os.environ.get("POLL_INTERVAL_SECONDS", "10"))
eval_model = os.environ.get("EVAL_MODEL", "gpt-4o")
optimization_model = os.environ.get("OPTIMIZATION_MODEL", "gpt-5")

terminal_statuses = {JobStatus.SUCCEEDED, JobStatus.FAILED, JobStatus.CANCELLED}

with (
  DefaultAzureCredential() as credential,
  AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
  job = project_client.beta.agents.create_optimization_job(
    job=OptimizationJob(
      inputs=OptimizationJobInputs(
        agent=OptimizationAgentIdentifier(agent_name=agent_name),
        train_dataset=OptimizationReferenceDatasetInput(
          name=dataset_name,
          version=dataset_version,
        ),
        evaluators=[OptimizationEvaluatorRef(name=evaluator_name)],
        options=OptimizationOptions(
          max_candidates=2,
          eval_model=eval_model,
          optimization_model=optimization_model,
        ),
      )
    )
  )

  print(f"Created optimization job: {job.id}")
  print(f"Initial status: {job.status}")

  while job.status not in terminal_statuses:
    time.sleep(poll_interval)
    job = project_client.beta.agents.get_optimization_job(job_id=job.id)
    print(f"Status: {job.status}")

  if job.status == JobStatus.FAILED:
    message = job.error.message if job.error else "<no error message>"
    raise RuntimeError(f"Optimization job failed: {message}")

  if job.result:
    print(f"Baseline candidate: {job.result.baseline}")
    print(f"Best candidate: {job.result.best}")

    for candidate in job.result.candidates or []:
      print(
        f"{candidate.name}: candidate_id={candidate.candidate_id}, "
        f"avg_score={candidate.avg_score:.4f}, "
        f"avg_tokens={candidate.avg_tokens:.0f}"
      )

Run the script:

python optimize_hosted_agent.py

When the job succeeds, the script prints the winning candidate and its candidate_id.

Unlike azd ai agent optimize, the Python SDK flow doesn't create a local .agent_configs/baseline/metadata.yaml file. The optimization job metadata stays in the returned job object and in the Foundry service response, including the baseline candidate, best candidate, and scored candidate list.

3. Apply the winning candidate

If you're also working from the local azd project used in the CLI flow above, apply the winning candidate by using the candidate_id returned by the Python script:

azd ai agent optimize apply --candidate <candidate-id>
azd deploy

If you only need to inspect the result, use the candidate scores and evaluation identifiers printed by the script to review the winning configuration in Foundry before promoting it.