Any configuration to set the azure ai search indexer (pull) to upload mode instead of default merge mode

Question

Any configuration to set the azure ai search indexer (pull) to upload mode instead of default merge mode

Venkata Rohith Chennakesavulu 20

Hi Team,

I am pulling the JSON files from the blob storage, problem is that whenever there are few fields removed from the few of JSON files the indexer (blob indexer -pull) is not removing those values from index, i guess this is the default behavior of indexer and it is in merge mode i guess. Now, i want to change it to behave like a "upload" so it will override each and every field eventhough it exists in data or not.

Is there any way to do it from indexer pov? I know we can handle it using Azure python SDK upload function, but before going into coding i want to make sure is there anything else we can do at azure portal itself

0 comments

Answer accepted by question author

Venkatesan S 10,830 Microsoft External Staff Moderator

Hi @Venkata Rohith Chennakesavulu

Thanks for reaching out in Microsoft Q&A forum,

With the Azure AI Search Blob Indexer, when an existing document is re-indexed, fields supplied by the source JSON are updated. However, if a field that previously existed in the search index is simply removed from the JSON document, the indexer does not automatically remove that field from the existing search document.

For example, if the original JSON contains:

{   "id": "123",   "name": "John",   "department": "IT",   "location": "Texas" }

and the updated JSON removes the location property:

{
  "id": "123",
  "name": "John",
  "department": "IT"
}

the existing location value can remain in the search index.

There is currently no Blob Indexer configuration in the Azure portal to change this behavior to an upload operation, where the entire existing document is replaced whenever the blob is re-indexed.

If the intention is to clear a particular field while continuing to use the Blob Indexer, you can explicitly provide an appropriate empty value in the JSON. For example, for a string field:

{
  "id": "123",
  "name": "John",
  "department": "IT",
  "location": ""
}

This allows the existing string value to be cleared rather than leaving the previous value in the index.

User's image

If you require true full-document replacement (upload) semantics, then the Azure AI Search REST API or SDK would be required. With the Search Documents API, the upload action replaces the existing document with the document supplied in the request, whereas merge updates only the fields supplied in the request.

Therefore, from the Blob Indexer/portal perspective, there is currently no setting to switch the indexer to upload mode.

The practical options are:

Ensure the source JSON explicitly provides empty values for fields that need to be cleared, where supported; or
Use the Azure AI Search REST API/SDK to perform an upload operation with the complete document.

For reference, please see the official Microsoft documentation:

Kindly let us know if the above helps or you need further assistance on this issue.

Please do not forget to and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

0 comments

1 additional answer

Your answer

Answer 1

No. There is no indexer setting to switch blob indexer pull behavior from merge-like updates to full upload replacement.

What is supported:

For JSON blob indexing, if a source JSON field is null, the indexer retains the existing value in the index.
To explicitly clear a field through the indexer, the source JSON must send an empty string ("") instead.
If the field mappings don’t match clearly, explicit fieldMappings can be used, but that does not change update semantics.
If full document replacement is required, the supported document action with that behavior is upload, which is available when pushing data to the index through REST API or an SDK.

For the scenario described, the portal/indexer path does not provide a setting to force missing or removed JSON properties to overwrite existing indexed values as null. From the indexer point of view, the practical options are:

change the source JSON so fields that must be cleared are emitted as empty strings
use a push-based ingestion path and send documents with the upload action instead of relying on the pull indexer

Related detail: for push indexing, upload replaces the document, while merge updates an existing document and mergeOrUpload does either depending on whether the document already exists.