Azure Document Intelligence Combining fields together

Question

Azure Document Intelligence Combining fields together

Becky Burgess 40

We have been facing a significant decrease in performance (today mostly) after training the model once again with newer PO documents. Today alone has been a major struggle. It's unclear what exactly is causing the issues, but basically the model will combine a field it was trained to read and another field it was NOT trained to read and combining it together. In our use case, it's reading customer purchase orders. It is combining customer codes with other items from the PO that are nearby on the location of the PO. This has been happening today on multiple purchase order documents.

Additionally, it will read a field, such as unit price unit of measure (i.e. LB for pounds) on a PO but it's reading a number that's not even a unit of measure.

We have trained a single model on multiple document formats and I'm not sure if the high number of variations is causing it to get confused or not. Please advise if there are any server issues or what could decrease the performance for document intelligence.

0 comments

2 answers

Your answer

Answer 1

Hi @Becky Burgess

Since this started after retraining the model with newer PO documents, start by reviewing the training dataset and document variations rather than assuming a service issue.

Customer Code being combined with nearby PO text, or Unit of Measure picking up an adjacent numeric value, can happen when the model has trouble distinguishing spatially close fields or when the training examples have inconsistent layouts/labels.

I recommend using a balanced training dataset that represents the expected document variations. For custom template models, separate structural/layout variations into different models and then compose them.

If you're using a custom neural model, Microsoft allows more variation in a single model, but their current guidance is still useful: with a manageable number of variations (roughly 15 or fewer), include several representative samples of each variation. With larger numbers of substantially different layouts, consider multiple models.

Also review the newly added training documents carefully, especially the affected fields. Make sure CustomerCode, PurchaseOrder, UnitPrice, and UnitOfMeasure are consistently labeled and that the labeled regions don't unintentionally include adjacent text.

As a useful test, analyze the same affected PO using the old model → newly retrained model

If the old model still extracts the document correctly but the newly trained model combines the fields, that would strongly point toward a training/model regression rather than an Azure service problem.

Keep a small validation set of POs out of the training dataset and compare field-level results between model versions. This makes it much easier to identify which layouts or fields regressed after retraining.

If your PO formats are substantially different, an architecture such as

Incoming PO > Document classifier > PO format/type > Appropriate custom extraction model

may work better than trying to teach one extraction model every possible PO layout. Document Intelligence v4.0 composed models now support an explicit classifier for routing documents to the appropriate extraction model.

Regarding a possible Azure service issue, don't rule it out, but the timing alone isn't enough to establish that. If the same model ID and same document suddenly started returning different extraction results without retraining or configuration changes, that would be more indicative of something worth escalating as a service-side regression.

Could you confirm whether you're using a Custom Neural or Custom Template model, the API version, approximately how many PO layout variations are in the training set, and whether the previous model still processes the same affected POs correctly?

References:

Microsoft - Custom Document Models

Microsoft - Labeling and balanced datasets

Microsoft - Composed custom models

Help make this community better for everyone: If this answer helped or resolved your issue, please accept it or upvote it. If not, share more details in a comment so we can continue the discussion and find the right solution. Thank you.

Becky Burgess 40 Reputation points

2026-09-16T14:40:15.73+00:00

We are using a Custom Extraction Neural model and we do add 5 PO samples if the model doesn't pass 100% on model analysis test. We were guided in the beginning from Microsoft to use a single model for our primary use case which includes over 500 customers and 1,000 plus PO documents. It's currently reading all our orders that come in on the daily (200-300 orders a day). It's usually 80% success rate with 20% orders not being read accurately, we don't have the capacity for post processing logic so we just deal with it. However, yesterday was unusually worse than normal which is why I reached out.

Answer 2

What is being seen matches known Azure Document Intelligence behavior and limitations rather than a documented server-side feature change.

For custom neural models, the most relevant causes are:

Too many layout variations in one model Custom neural models can generalize across variations, but the recommended practice is to use a single model for one document type and include at least five labeled samples for each variation. If the PO formats vary heavily and some variations are underrepresented, extraction quality can drop.
Labeling issues that encourage field merging For custom neural training, value tokens for one field must be either:
- in a consecutive sequence in natural reading order, without interleaving with other fields, or
- in a region that doesn’t cover any other fields.
If a labeled region is broad, overlaps nearby content, or captures text that is visually close on the same reading line, the model can learn to combine adjacent content.
Training data not representative enough Training values should be diverse and representative. If the newer PO samples introduced new layouts or nearby distracting values, the retrained model can perform worse on those cases.
Known OCR/model accuracy limitations Document Intelligence has active known issues where OCR can misread characters and where layout-stage limitations affect downstream extraction. Custom training does not resolve all OCR-stage or layout-stage problems.

For the specific symptoms:

Customer code merged with nearby text: this is consistent with field-merging behavior reported for custom extraction models when nearby tokens are grouped together.
Unit of measure read as a number instead of LB: this is consistent with OCR ambiguity issues such as 1 vs l and similar character confusion. Post-processing validation is the documented workaround.

Recommended actions:

Review the latest training set variation coverage Group the PO samples by format/variation and confirm there are at least five labeled samples for each variation included in the training set.
Tighten labels for affected fields Recheck labels for customer code, unit of measure, and other nearby fields. Ensure each labeled value is contiguous and the labeled region does not cover neighboring fields.
Use field names that match the document language and meaning For example, use precise names such as customer_code or unit_of_measure. Relevant field naming improves key-value extraction accuracy.
Add representative failure samples Include the exact PO layouts where merging or wrong UOM extraction occurs, and label them carefully.
Add post-processing validation Since OCR-stage issues are not fully fixed by retraining, validate extracted values after analysis:
- For unit of measure, allow only expected values such as LB, EA, KG, and reject numeric-only results.
- For customer codes, use regex or character-set rules to detect concatenated or invalid values.
- Use per-word confidence scores to flag low-confidence extractions for correction or review.
Check for service-side latency separately from accuracy Document Intelligence is a multitenant asynchronous service, so latency can vary even for similar documents. That explains runtime slowness, but it does not explain field-merging behavior by itself.
If the issue started immediately after retraining Compare the previous and new training datasets. The most likely cause is the newer samples or labeling quality, not a documented platform change.
If training jobs are failing or behaving abnormally There is a known issue where training can fail with HTTP 500 / InternalServerError if managed identity is not configured correctly. Also, if a training job is stuck in Not started, backend cleanup can take about seven days, or an Azure portal support request can be submitted for earlier removal.

There is context showing occasional service incidents and regional performance investigations, so a temporary service issue is possible for latency or availability. But for the extraction errors described, the supported guidance points more strongly to model variation coverage, label boundaries, and OCR/layout limitations.