Building and customizing solutions using Microsoft 365 Copilot APIs and tools
Analyze images with Copilot Studio Agents coming from Email attachment on Copilot Studio Workflow
As the title suggests, I'm testing a workflow where I receive an email containing one or more images. The images are unstructured and may contain tables, screenshots of PDFs, dashboards, Power BI visuals, error messages, or other content.
My goal is to pass both the email body and the attachments to a Copilot Studio agent and have the agent analyze the image content in the context of the email.
So far, I've tested with both GPT-5 Chat and Claude Sonnet 5 models. The email trigger correctly retrieves the attachment, including the full contentBytes field (Base64-encoded PNG). I independently verified that the Base64 is valid by decoding it externally and confirming that the image renders correctly.
However, when I pass the attachment object to the agent, the models appear to hallucinate the image contents rather than actually reading the image. The responses contain different invented text on each run, even though the underlying image is unchanged.
I also tried:
- Passing the full attachment object, including
contentBytes - Passing the email body and attachment together
- Using different prompts focused on OCR/transcription
- Explicitly instructing the model not to summarize or infer and instead return only visible text
- Switching from GPT-5 Chat to Claude Sonnet 5
Copilot suggested adding an Extract action, but that action appears to be designed for structured extraction where specific fields must be defined ahead of time. In my scenario, I don't have a fixed schema because users can send arbitrary screenshots and images.
Am I missing a step to enable image understanding for email attachments in a workflow? Is there a recommended way to convert attachment contentBytes into an image input that the agent can actually analyze, or should I be using a different action (OCR, Document Intelligence, AI Builder, etc.) before the agent?