Optimize agent instructions, skills, tools, and models (preview)

The agent optimizer improves four aspects of your hosted agent: instructions, skills, tools, and model selection. It automatically detects which of these targets to optimize from your agent's baseline configuration.

This article shows how to run an optimization, configure and monitor the run, and deploy the results. For what each target does and when it activates, see Optimization targets. To set up the baseline inputs, see Make your agent optimizer-ready. For a quick reference on what the optimizer changes, see What each target changes.

Prerequisites

• A Foundry project with a deployed hosted agent
• The azure.ai.agents CLI extension installed (see Quickstart: Optimize a hosted agent)
• A model deployed for evaluation (for example, gpt-4.1-mini) and an optimization model from the supported list (for example, gpt-5.1)
• Your agent is optimizer-ready (calls load_config())

Run an optimization

Start an optimization run with a single command:

azd ai agent optimize

The optimizer evaluates your baseline, generates candidates, evaluates them, and ranks the results. For the full evaluate-and-improve cycle, see How the agent optimizer works. Which targets run depends on your baseline configuration—instruction tuning, skill improvement, and tool optimization activate automatically when the matching baseline files are present. See Optimization targets.

To control the run with a config file, pass an eval.yaml that references your dataset, evaluators, and options:

azd ai agent optimize --config eval.yaml

For the full eval.yaml schema, see Configure the optimization run.

Target a specific agent

By default, the CLI optimizes the agent detected from your current azd environment and your project's local agent.yaml. The optimizer resolves the agent name in this priority order:

Use the --agent flag when you have multiple agents in your project or want to override the default. The agent name must match a deployed hosted agent in your Foundry project.

Configure the optimization run

Configure optimization runs through an eval.yaml file that ties together your dataset, evaluators, and run options. The command azd ai agent eval generate writes this file for you, or you can create it by hand. The optimizer auto-detects eval.yaml in your project root, or you can pass it explicitly with --config eval.yaml.

# eval.yaml
name: my-optimization              # Optional label for the run
agent:
  name: my-agent                   # Deployed hosted agent name
  kind: hosted
  version: "1"                     # Agent version (optional)
  model: gpt-4.1-mini              # Baseline model deployment
  config: .agent_configs/baseline/metadata.yaml
dataset:
  local_uri: ./eval.jsonl          # A local JSONL file...
  # name: my-foundry-dataset       # ...OR a registered Foundry dataset
  # version: "1"
# validation_dataset:              # Optional held-out dataset
#   name: my-validation-dataset
#   version: "1"
evaluators:
  - builtin.task_adherence         # A built-in evaluator...
  # - name: my-custom-evaluator    # ...or a custom evaluator
  #   version: "1"
  #   local_uri: ./my_evaluator.json
options:
  eval_model: gpt-4.1-mini         # Scores responses
  optimization_model: gpt-5.1      # Generates candidates
  max_candidates: 4
  optimization_config:
    model_search_space:            # Optional: compare model deployments
      - gpt-4.1

Author the dataset and evaluators separately; see Create an evaluation dataset and evaluators. The following sections describe the run options.

Choose the eval and optimization models

The optimizer uses two models: an eval model that scores agent responses against criteria, and an optimization model that generates candidate configurations. Set them in eval.yaml or use CLI flags.

options:
  eval_model: gpt-4.1-mini
  optimization_model: gpt-5.1

azd ai agent optimize --eval-model gpt-4.1-mini --optimize-model gpt-5.1

Any chat-completion model deployed in your project works as the eval model. The optimization model must be from the supported list. For roles and supported models, see Models.

Set the number of candidates

The max_candidates option sets the expected number of candidate configurations for the run. The optimizer typically returns after it reaches that count, unless the run stops early because of an error or another stopping condition.

Higher values explore more variations but take longer. The optimizer learns from earlier candidates, so later candidates tend to score higher.

Evaluate multiple models

To compare model deployments in a single run, list them under optimization_config.model_search_space. The optimizer evaluates your agent with each model against the same dataset and ranks the results by score and token cost.

# eval.yaml
options:
  optimization_config:
    model_search_space:
      - gpt-4.1
      - gpt-4.1-mini
      - gpt-4o

Each model listed under model_search_space must be deployed in your Foundry project.

Model selection runs alongside the targets that activate automatically from your baseline. A single run can produce candidates that combine improved instructions, skills, and tool descriptions with different model options - you don't configure the combination yourself.

Monitor a running job

An optimization run is asynchronous. Use these commands when a job is long-running or you want to check its progress:

# Check status and stream progress
azd ai agent optimize status <operation-id> --watch

# List recent optimization jobs
azd ai agent optimize list

# Cancel a running job
azd ai agent optimize cancel <operation-id>

Capture the operation ID, portal URL, scores, and candidate IDs from the run output. You can also monitor the job in the Foundry portal using the URL shown when the run starts.

Interpret results

After optimization completes, review the results table. An asterisk (*) marks the best candidate. For the results table columns, scoring details, score-improvement thresholds, and the portal view, see Understand optimization results.

Deploy the winner

The recommended workflow is to apply the optimized config locally, then deploy:

# Apply the winning candidate locally
azd ai agent optimize apply --candidate <candidate-id>

# Deploy with the optimized config
azd deploy

This downloads the optimized configuration into .agent_configs/<candidate_id>/ in your project. On next deploy, your agent uses the improved instructions and tool descriptions.

Alternatively, you can deploy directly via the API (useful for quick A/B testing):

azd ai agent optimize deploy --candidate <candidate-id>

If all candidates score lower than the baseline, don't deploy any candidate. The baseline configuration remains active.

What each target changes

The optimizer automatically activates the targets that apply to your baseline. This section is a reference for what a run changes. Use the following table to anticipate what optimization does for your agent:

Your code stays the same across all targets because load_config() returns the optimized values automatically. Only the configuration the model sees changes.

Instructions

The optimizer rewrites the system prompt. Common improvements include:

• Adding explicit constraints that the original prompt implied but didn't state
• Restructuring instructions for clarity
• Adding output format specifications
• Strengthening safety and scope boundaries

For example, a minimal baseline prompt like You are a helpful assistant. might become:

You are a helpful coding assistant. Follow these guidelines:
1. Always include working code examples
2. Explain your reasoning step by step
3. If a question is outside your expertise, say so clearly
4. Use markdown formatting for code blocks
5. Handle edge cases in code examples

Skills

The optimizer refines each skill's description, body, and activation criteria while keeping the skill's purpose intact. The agent loads improved skills through load_config(), which appends them to the instruction set. Skills use the open Agent Skills format. For how your agent loads skills, see Make your agent optimizer-ready.

Tools

The optimizer refines your tools.json definitions. Common improvements include:

• Clearer function descriptions that help the model know when to call a tool
• More specific parameter descriptions that reduce inaccurate arguments
• Added constraints (enums, required fields) that prevent invalid inputs

Your tool implementation code stays the same. Only the definitions the model sees change.

Models

The optimizer ranks each candidate model by composite score and token cost, so you can choose the best quality-to-cost trade-off. To configure the candidates, see Evaluate multiple models.

Troubleshooting

Related content

• Agent optimizer overview
• Make your agent optimizer-ready
• Create an evaluation dataset and evaluators
• Quickstart: Optimize a hosted agent