Azure AI Doc Intelligence data extraction in salesforce

Question

Azure AI Doc Intelligence data extraction in salesforce

IT Cognity 0

Hello Microsoft Team,

We are using Azure AI Document Intelligence in a Salesforce-based digital onboarding flow for document classification and data extraction.

Our current flow is:

Document upload → Azure Document Intelligence custom classification model → Routing to the corresponding extraction model based on the predicted document type → Field extraction and business validation

The classifier is used to identify different types of identification documents, such as older ID cards, newer ID cards, passports, residence permits, security force IDs, etc.

To handle cases where users upload documents that are not valid identification documents, we have also added a separate class/category named other.

For the other class, we have used training samples that represent non-ID documents we expect users may upload by mistake, such as utility bills, property-related documents, and similar business documents. For these expected non-ID documents, the classifier generally works correctly and classifies them as other.

However, our issue is with completely irrelevant or unexpected files/images. For example, random images, screenshots, arbitrary documents, or files that are clearly not identification documents are sometimes classified as one of the ID document classes instead of other.

In some of these cases, the classification confidence returned by Azure is also high. We have observed cases where the confidence score for an irrelevant file classified as an ID type is higher than the confidence score for a valid ID document classified correctly. Because of this, using only a confidence threshold is risky for our use case, since it could either fail to reject irrelevant files or incorrectly reject valid ID documents.

We would like Microsoft’s official guidance on the expected behavior and recommended design for this scenario.

Specifically:

Would Microsoft recommend a different architecture for this use case?
If the other class is expected to work for this scenario, what type of training data should be included in it?
Is there a recommended maximum or practical limit for how broad the other class should be before it becomes ineffective or negatively affects classification between valid ID document classes?
Since the confidence score is not consistently lower for irrelevant files, what is Microsoft’s recommended approach for rejecting documents that do not belong to any valid ID category?
More generally, what is the recommended way to handle out-of-scope / out-of-distribution documents in Azure Document Intelligence custom classification, especially when the possible invalid inputs are very broad and cannot be fully represented in training data?

Our goal is to understand whether the current behavior is expected product behavior, whether a broad other class can reliably solve this problem, and what Microsoft recommends as the best-practice approach for rejecting irrelevant files without incorrectly rejecting valid ID documents.

IT Cognity 0 Reputation points

2026-07-07T13:50:23.2066667+00:00

Thank you for the information.

I would like to ask specifically about how custom classifier confidence is calculated.

In our tests, we have observed cases where an irrelevant or out-of-scope file is classified as a valid ID document type with high confidence. We have also seen cases where a correctly classified valid ID document receives lower confidence than an incorrectly classified irrelevant file.

Could you please explain how the classifier confidence score is calculated and how it should be interpreted?

Also, why can a wrong classification, especially for an out-of-scope document, receive higher confidence than a correct classification of a valid document?
IT Cognity 0 Reputation points

2026-07-07T13:53:35.23+00:00

Thank you for the information.

I would like to ask specifically about how custom classifier confidence is calculated.

In our tests, we have observed cases where an irrelevant or out-of-scope file is classified as a valid ID document type with high confidence. We have also seen cases where a correctly classified valid ID document receives lower confidence than an incorrectly classified irrelevant file.

Could you please explain how the classifier confidence score is calculated and how it should be interpreted?

Also, why can a wrong classification, especially for an out-of-scope document, receive higher confidence than a correct classification of a valid document?
Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-07-07T15:08:47.3533333+00:00

Hi IT Cognity ,

Let me check this with the backend team. I will get back to you shortly.
Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-07-11T09:52:48.84+00:00
Hi @IT Cognity ,

Thanks for the follow-up. Microsoft doesn't publish the classifier's exact internal formula, so I'll focus on what the score means and why you're seeing this ordering.

How to interpret it: The confidence is an "estimated probability between 0 and 1 that the prediction is correct." The key point is that the classifier is closed-set: by design it "attempts to assign each document to one of the classes" and always picks one. So the score tells you which of your trained classes the input most resembles, relative to the others — not whether the input is a genuine ID. It's meant to set an accept-vs-review threshold, not to validate legitimacy.

Why a wrong out-of-scope file can outscore a correct ID:

There's no "none of these" option — an irrelevant file's score must be distributed across your existing classes, so if it aligns strongly with one, it scores high with nowhere else to go.

Confidence reflects separation between your classes, not correctness or whether the input is in-distribution. A random file decisively unlike all classes but one can beat a genuine ID sitting near a boundary.

Valid IDs legitimately score lower when they have variations, glare, skew, or poor scans — the docs note low document-type confidence is "indicative of template or structural variations."

So this is expected behavior of relative, closed-set scoring — and it's why a confidence threshold alone can't reliably reject irrelevant files.

Recommendation: Use confidence as a coarse accept-vs-review filter only; pair it with an other class plus a threshold for anticipated non-IDs; and put the real rejection decision after classification by validating that the routed extraction model returns well-formed ID fields (document number, MRZ checksum, dates). A random image classified as "passport" with high confidence will still fail that validation — that's your reliable signal.

References

https://learn.microsoft.com/azure/ai-services/document-intelligence/concept/accuracy-confidence?view=doc-intel-4.0.0

https://learn.microsoft.com/azure/ai-services/document-intelligence/train/custom-classifier?view=doc-intel-4.0.0

Kindly let us know if the above helps or you need further assistance on this issue. If the answer is helpful, kindly upvote it. If you have extra questions about this answer, please click "Comment".
Christos Panagiotidis 3,551 Reputation points

2026-07-19T14:48:50.49+00:00

You’re seeing expected closed-set behavior: Microsoft says the classifier attempts to assign every document to one of its trained classes. Therefore, an irrelevant file can receive high confidence; document-type confidence indicates resemblance to training data, not a guaranteed “in-scope” probability. An “other” class is appropriate, but train it with representative negatives—especially frequent mistakes and documents visually similar to IDs—not an arbitrary collection meant to cover every file. Microsoft specifies 100 samples per class, but no reliability guarantee for an unlimited negative universe.

In production, handle rejection in your application: evaluate a held-out set of valid IDs, hard negatives, and unseen documents; derive thresholds from measured errors; route “other,” low-confidence, or failed required-field and business-rule cases to human review; and automate only when classification and extraction checks pass. Monitor reviewed failures, add recurring variations or new classes, and update the classifier incrementally. Avoid rejecting valid IDs solely from one score.
Christos Lizos - Cognity 0 Reputation points

2026-07-23T08:58:56.4166667+00:00

Thank you for the clarification.

Based on your explanation, we understand that this is expected closed-set classifier behavior and that confidence alone cannot reliably reject irrelevant or out-of-scope files.

However, due to our business requirements, routing these cases to human review is not a feasible option for this flow. The process needs to remain automated.

If we use a classification confidence threshold alone, is there a risk that many correctly classified valid ID documents will also be rejected, especially in cases with lower image quality, glare, skew, or structural variations?

Generally, what would Microsoft recommend as the best automated approach to reduce cases where completely irrelevant files are classified as valid ID document types?

Thank you,

Cognity Team
Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-07-24T16:19:21.4233333+00:00
IT Cognity ,Christos Lizos - Cognity Thank you for the additional details.

Yes, there is a trade-off when using a classification confidence threshold as the primary rejection mechanism. As you noted, valid ID documents can receive lower confidence scores due to factors such as image quality, glare, skew, partial captures, template variations, or document versions that are underrepresented in the training set. Because of this, increasing the threshold to reject more irrelevant files typically also increases the false-rejection rate for legitimate documents.

For that reason, Microsoft generally would not recommend relying solely on the classifier confidence score to determine whether a document is valid for processing. The confidence score is one signal, but it was not designed to be a definitive out-of-scope detector.

For a fully automated workflow, a more robust approach is usually to combine multiple validation stages:

Document classification

Use the custom classifier to predict the document type.

Optionally include an other class for known non-ID document categories.

Document-type-specific extraction

Route the document to the corresponding extraction model.

Business and structural validation

Validate whether the expected fields for that document type are present and internally consistent.

Examples may include document number formats, date validation, MRZ validation for passports, mandatory field presence, expiry-date checks, checksum validation, etc., depending on the document type.

Automated acceptance/rejection decision

Accept only when both classification and downstream validation succeed.

Reject documents that fail validation even if the classifier confidence is high.

In practice, this final validation layer is often a stronger indicator of document validity than classification confidence alone. An out-of-scope image may be classified as a passport with high confidence, but it will frequently fail the expected extraction and validation checks for an actual passport document.

Regarding the other class, it can help with anticipated invalid inputs, but it is generally not possible to represent the entire space of out-of-scope documents through training data alone. As the variety of documents included in the other category becomes broader, it can become increasingly difficult for the model to learn a clear representation of that class. Therefore, Microsoft would typically view the other class as one component of the solution rather than the sole mechanism for rejecting arbitrary documents.

Based on the behavior you described, what you are seeing is consistent with the expected behavior of a closed-set classifier. For scenarios where the set of invalid inputs is effectively unbounded, the recommended approach is usually a combination of classification, extraction, and downstream document-specific validation rather than confidence-thresholding alone.

1 answer

Your answer

IT Cognity 0 Reputation points

2026-07-07T13:50:23.2066667+00:00

Thank you for the information.

I would like to ask specifically about how custom classifier confidence is calculated.

In our tests, we have observed cases where an irrelevant or out-of-scope file is classified as a valid ID document type with high confidence. We have also seen cases where a correctly classified valid ID document receives lower confidence than an incorrectly classified irrelevant file.

Could you please explain how the classifier confidence score is calculated and how it should be interpreted?

Also, why can a wrong classification, especially for an out-of-scope document, receive higher confidence than a correct classification of a valid document?
IT Cognity 0 Reputation points

2026-07-07T13:53:35.23+00:00

Thank you for the information.

I would like to ask specifically about how custom classifier confidence is calculated.

In our tests, we have observed cases where an irrelevant or out-of-scope file is classified as a valid ID document type with high confidence. We have also seen cases where a correctly classified valid ID document receives lower confidence than an incorrectly classified irrelevant file.

Could you please explain how the classifier confidence score is calculated and how it should be interpreted?

Also, why can a wrong classification, especially for an out-of-scope document, receive higher confidence than a correct classification of a valid document?
Thanmayi Godithi 11,825 Reputation points Microsoft External Staff Moderator

2026-07-07T15:08:47.3533333+00:00

Hi IT Cognity ,

Let me check this with the backend team. I will get back to you shortly.
Christos Panagiotidis 3,551 Reputation points

2026-07-19T14:48:50.49+00:00

You’re seeing expected closed-set behavior: Microsoft says the classifier attempts to assign every document to one of its trained classes. Therefore, an irrelevant file can receive high confidence; document-type confidence indicates resemblance to training data, not a guaranteed “in-scope” probability. An “other” class is appropriate, but train it with representative negatives—especially frequent mistakes and documents visually similar to IDs—not an arbitrary collection meant to cover every file. Microsoft specifies 100 samples per class, but no reliability guarantee for an unlimited negative universe.

In production, handle rejection in your application: evaluate a held-out set of valid IDs, hard negatives, and unseen documents; derive thresholds from measured errors; route “other,” low-confidence, or failed required-field and business-rule cases to human review; and automate only when classification and extraction checks pass. Monitor reviewed failures, add recurring variations or new classes, and update the classifier incrementally. Avoid rejecting valid IDs solely from one score.
Christos Lizos - Cognity 0 Reputation points

2026-07-23T08:58:56.4166667+00:00

Thank you for the clarification.

Based on your explanation, we understand that this is expected closed-set classifier behavior and that confidence alone cannot reliably reject irrelevant or out-of-scope files.

However, due to our business requirements, routing these cases to human review is not a feasible option for this flow. The process needs to remain automated.

If we use a classification confidence threshold alone, is there a risk that many correctly classified valid ID documents will also be rejected, especially in cases with lower image quality, glare, skew, or structural variations?

Generally, what would Microsoft recommend as the best automated approach to reduce cases where completely irrelevant files are classified as valid ID document types?

Thank you,

Cognity Team

Answer 1

Hi IT Cognity ,

Thank you for reaching out on Microsoft Q&A forum.

Azure AI Document Intelligence is the extraction engine — it returns structured JSON (fields + confidence) via an asynchronous REST API/SDK. There's no built-in Salesforce connector, so the pattern is: call Document Intelligence, then have an integration layer write the results into Salesforce.

The Document Intelligence call (prebuilt or your custom model):

POST https://<resource>.cognitiveservices.azure.com/documentintelligence/documentModels/<modelId>:analyze?api-version=2024-11-30
Ocp-Apim-Subscription-Key: <key>
{ "urlSource": "<blob-SAS-url>" }        # or base64Source / bytes
→ 202 Accepted; read the Operation-Location header
GET <Operation-Location>  → poll until status "succeeded"
  → analyzeResult.documents[].fields (value + confidence)

It's async by design — whatever calls it must poll Operation-Location; there's no synchronous fields-now call.

Recommended Azure-native orchestration:

Logic Apps — the built-in Document Intelligence action handles the polling for you, then the Salesforce connector creates/updates the record. Lowest-code, cleanest.

Azure Functions — call DI, poll, shape the JSON, then POST to the Salesforce REST API with an integration user's OAuth token (store secrets in Key Vault, use managed identity for the DI call).

If the write-to-Salesforce must stay on the Salesforce side, expose an Azure Function/APIM endpoint that runs DI submit+poll internally and returns finished fields for their Apex/MuleSoft to consume.

A few Azure-side tips: send documents as bytes or a Blob SAS URL (DI must be able to reach them); use the per-field confidence scores to route low-confidence extractions to human review before writing; and prefer Entra + managed identity over keys for production.

Kindly let us know if the above helps or you need further assistance on this issue. If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".