Vision fine-tuning training-file preprocessing fails with HTTP 500 "Infrastructure Issues"

An HTTP 500 with an "Infrastructure Issues" tag right around that ~60-minute mark is a classic sign of an underlying gateway timeout or a background out-of-memory (OOM) crash during the image preprocessing stage. Since your small files and validation files pass without any issues, your file formatting is completely correct—the backend pipeline is just hitting a resource or execution wall when it tries to unpack, decode, and resize a massive batch of high-resolution images all at once.

To bypass this infrastructure bottleneck, you can try a couple of quick adjustments:

Downsize/Compress Your Source Images: If your training images are shot in extremely high resolution or raw formats, try bulk-resizing them down locally (e.g., to a maximum of 512x512 or 1024x1024 pixels) and compressing the file before uploading. The model will downsample them anyway, and this drastically cuts down the backend's unpacking and CPU processing time.

Streamline the Training Dataset: Check if you can trim down the total volume of images in the large training file slightly to keep the preprocessing window comfortably under that strict 60-minute container timeout threshold.

If you've already optimized the file sizes and it still chokes, a backend engineer will definitely need to pull your Job ID (ft-job-e0c116095e0a4da5a9a8ec5ccd88c4fd) to manually scale up the processing container's memory allocation for your tenant!

Hello @Giannis ,

Welcome to Microsoft Q&A .Thank you for reaching out to us.

The observed behavior suggests that the failure is occurring during large-scale preprocessing.While smaller datasets and validation files process successfully, backend telemetry is required to determine whether the condition is related to preprocessing resources, image retrieval, moderation processing, storage access, or another infrastructure-side event.

According to the current Vision Fine-Tuning documentation, the supported limits are:

• Maximum 50,000 examples containing images per training file.
• Maximum 64 images per example.
• Maximum 10 MB per image.
• Supported formats: JPEG, PNG, WEBP.
• Supported image modes: RGB and RGBA.

At the same time, there is currently no publicly documented limit for:

• Preprocessing timeout duration.
• Preprocessing memory allocation.
• Aggregate image volume across a dataset.
• Total decoded pixels.
• Maximum preprocessing runtime

As of now , there is no publicly documented mechanism to increase preprocessing memory allocation or preprocessing timeout values for a specific resource or tenant. The available quota documentation does not expose these controls.

Regarding a supported path for larger Vision Fine-Tuning datasets , Large datasets are supported provided they remain within the documented dataset requirements. However, there is currently no publicly documented large-dataset-specific preprocessing tier, extended preprocessing budget, dedicated region or configurable preprocessing mode for Vision Fine-Tuning workloads

Please check if the following workarounds help-

1. Validating Dataset Accessibility
   1. Please confirm all SAS URLs remain valid throughout the preprocessing duration.
   2. Verify all image URLs consistently return HTTP 200 responses.
   3. Review for redirects, IP restrictions, storage throttling policies, or near-expiry SAS tokens.
   4. Confirm images remain within the documented size requirements.
2. Performing Dataset Scaling Tests
   1. Split the training dataset into smaller subsets.
   2. Execute Fine-Tuning jobs with progressively larger portions of the dataset.
   3. Determine whether failures begin at a consistent dataset-size threshold.
   4. If a specific subset repeatedly fails, review the images and records contained within that subset
3. Reducing Processing Overhead
   1. Continue using resized images where possible.
   2. For diagnostic testing, consider using "detail": "low" to reduce image-processing requirements during training
4. Verifying the Upload Method
   1. Please confirm whether the dataset was uploaded using the Files API or the Uploads API.
   2. For larger datasets, validating whether the Uploads API was used may be worthwhile, as it is designed to support significantly larger uploads than the Files AP

The following references might be helpful , please check them out

• Vision fine-tuning - Microsoft Foundry | Microsoft Learn
• Azure OpenAI in Microsoft Foundry Models Quotas and Limits - Microsoft Foundry | Microsoft Learn
• Customize a model with fine-tuning - Microsoft Foundry | Microsoft Learn  

Thank you

Please "Accept" the answer with an "Upvote" if the response was helpful. This will be benefitting other community members who face the same issue.