Does the SharePoint (Indexed) knowledge source / indexer have a working high-water-mark (change-tracking) mechanism?

Brian Levine 20 Reputation points
2026-08-18T17:08:30.5233333+00:00

For a SharePoint (Indexed) Foundry IQ knowledge source (IndexedSharePointKnowledgeSource, backed by the SharePoint-in-Microsoft-365 indexer/connector) — does a real change-tracking / high-water-mark mechanism exist that lets the indexer skip items that haven't changed since the last run, the way other data source types (e.g. Azure SQL's _ts/rowversion-based tracking) support?

Why I'm asking

Indexing runs against our SharePoint site take a long time (large corpus, heavy chunking/embedding), so we want to confirm that a run isn't needlessly re-processing/re-embedding content that hasn't actually changed since the previous run. In other words: once the indexer has fully caught up, should a subsequent run with zero actual SharePoint changes complete quickly and touch effectively 0 items — or does this connector re-evaluate/re-touch the full corpus (or large parts of it) on every run regardless of whether anything changed?

I'm aware of Microsoft's documented note that SharePoint metadata-only operations (renames, moves) can trigger incremental reindexing even without a real content change — that's a separate, expected behavior. What I haven't found documented is whether, setting metadata churn aside, there's a genuine per-item high-water-mark that skips truly untouched content, or whether this specific (preview) connector lacks that and reprocesses more broadly than expected.

What I'd like to know

  1. Does this indexer type track a last-modified marker per SharePoint item to skip unchanged content, or does it re-evaluate every item on every run?
  2. Has anyone verified this empirically — e.g., run twice in a row with no known SharePoint changes in between, and confirmed the second run processes ~0 items?
  3. Any known preview limitations specific to this connector's change-tracking implementation?

Any pointers to docs, source, or first-hand experience appreciated.

Azure AI Search
Azure AI Search

An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.

0 comments
Answer accepted by question author
Jose Benjamin Solis Nolasco 12,201 Reputation points Volunteer Moderator
2026-08-18T23:39:13.14+00:00

Welcome to Microsoft Q&A!

Hello @Brian Levine

Yes. The current Microsoft documentation indicates that the SharePoint in Microsoft 365 indexer does perform incremental indexing.

For the SharePoint (Indexed) knowledge source, Azure AI Search automatically creates an indexer pipeline. Microsoft states that the SharePoint indexer:

  • Picks up only new and changed files, list items, pages, and metadata.

Detects deleted SharePoint content automatically.

Maintains its indexing state so subsequent runs don't require a full reprocessing of the corpus. (learn.microsoft.com)

Azure AI Search describes this generally as an internal high-water mark. The marker isn't exposed to users, but the indexer uses it to determine where it should resume processing. If the underlying content hasn't changed, a normal Run Indexer operation should have no effect, and the execution history should show approximately 0 documents processed. (learn.microsoft.com)

So, for your specific scenario, after an initial successful ingestion:

Run 1: large initial corpus → documents are indexed/chunked/embedded.

Run 2 with no SharePoint changes: → the indexer should detect no new/changed content and should not reprocess the entire corpus.

You should therefore be able to validate this empirically by running the indexer twice without changing the SharePoint content and comparing the execution history and processed-document counts.

Your observation about metadata changes is relevant. A SharePoint item can be considered changed even when the actual document content hasn't changed for example, a rename or move can cause the item to be reconsidered. Therefore, "zero content changes" doesn't necessarily mean "zero changes detected."

Also, this SharePoint connector is currently preview, so I would avoid assuming that every change-detection edge case behaves exactly like a mature Azure SQL high-water-mark implementation. (learn.microsoft.com)

If you are seeing substantial reprocessing despite no SharePoint changes at all, I would capture the indexer execution history from two consecutive runs and compare:

Items processed

Items failed

Items skipped

Duration

Whether chunking/embedding was invoked again

That would distinguish normal SharePoint change detection from unexpected reprocessing.

References:

Index content from SharePoint in Microsoft 365

Run or reset indexers

Create an indexed SharePoint knowledge source

If this answer helped clarify the platform capabilities and save you troubleshooting time, please consider marking it as Accepted. This helps others in the community find similar solutions.

Was this answer helpful?

1 person found this answer helpful.
0 comments

1 additional answer

Sort by: Newest
Most helpful Newest Oldest
  1. Ajay Rathod 440 Reputation points Microsoft External Staff Moderator
    2026-08-19T06:35:12.36+00:00

    Hi @Brian Levine

    Greetings for the day, Thanks for reaching out to Q&A

    1. How SharePoint Change Tracking Works

    Change tracking is supported here, but it is not a full re-crawl, nor is it based on a configurable high-water-mark column.

    The SharePoint connector documentation states that it:

    “Indexes incrementally, picking up just the new and changed files, list items, pages, and metadata.”

    “Detects deleted content automatically,” with deletions picked up during the next indexer run and the corresponding search document removed from the index.

    The actual change-tracking mechanism is described in the ACL documentation, which is easy to overlook:

    “The indexer uses SharePoint change tokens to pick up role assignment additions and removals incrementally, in the same way it picks up content changes.”

    Therefore, the connector uses a service-managed cursor over the SharePoint change log, rather than comparing a configurable per-item timestamp column.

    That said, the platform does use the term high-water mark for its internal state. The reset documentation states that “Reset clears the high water mark,” and the Create an Indexer documentation has a Change detection and internal state section explaining that a reset clears the high-water mark to allow a full reindex.

    So, the HWM does exist internally; for the SharePoint connector, it is simply opaque and service-managed rather than user-configurable.

    One important distinction: metadata_spo_item_last_modified is exposed as a field in the index, but there is no documentation indicating that this field is used as the change-tracking key.

    2. How to Validate Incremental Processing

    I don't have a published benchmark for two consecutive no-change runs, and I haven't personally run this test. Rather than relying on anecdotal results, the behavior can be validated directly through the indexer status API:

    GET https://[service].search.windows.net/indexers/[generated-indexer]/status?api-version=2026-04-01

    The response contains the 50 most recent executions, with the newest listed first.

    Compare itemsProcessed across two consecutive executions. Also check initialTrackingState and finalTrackingState for each execution. The indexer troubleshooting documentation identifies these fields as the way to determine the indexer's change-tracking state; they represent the tracking state at the beginning and end of an execution.

    There is one caveat: the API reference states that both fields are null when no data-change-tracking policy is configured on the data source. Since SharePoint's change tracking is built into the connector rather than configured as a separate policy, these fields may also be null in this scenario.

    If the tracking-state fields are populated and advance between runs while itemsProcessed drops significantly on the second no-change run, that would provide strong confirmation that incremental change tracking is working as expected.

    The generated indexer name can be obtained from the createdResources list returned during knowledge-source creation or viewed under Indexers in the Azure portal.

    3. Where the Cost Risk Actually Sits

    The more significant cost concern is downstream of change tracking, specifically enrichment.

    The enrichment cache is normally used to prevent skills from running again against unchanged content. However, the SharePoint indexer (Preview) documentation explicitly advises against using incremental enrichment with this connector:

    “If you're using the SharePoint indexer (Preview), avoid incremental enrichment. Under certain circumstances, the cache becomes invalid. To reload it, perform an indexer reset and full rebuild.”

    This means that when an item is reprocessed for any reason, the associated skillset operations can run again. For workloads involving chunking and embeddings, that can result in additional processing and embedding costs.

    Combined with the metadata churn already observed, even relatively modest content or metadata changes could therefore translate into meaningful re-embedding costs.

    There is also an architectural limitation here: Foundry IQ generates the data source, skillset, indexer, and index automatically. The documentation advises against directly modifying these generated resources because doing so can introduce incompatibilities or break the indexing pipeline. Consequently, adding an enrichment cache directly to the generated indexer should not be treated as a supported workaround.

    4. Additional Limitations

    Two other points are worth keeping in mind:

    Incremental ACL updates require API version 2026-05-01-preview or later.

    Permission changes made at a parent scope—such as the site, library, list, or folder level—are not automatically picked up.

    5. If Change-Driven Cost Becomes Material

    If the investigation confirms that metadata/content churn is causing significant reprocessing and embedding costs, there are documented alternatives.

    One approach is to move change detection to a surface that you control, for example:

    SharePoint webhooks + Microsoft Graph, exporting content into Azure Blob Storage and then using the Blob indexer.

    Logic Apps, using the SharePoint and Azure AI Search connectors.

    These approaches give you greater control over change detection and also allow the enrichment cache to be used in a supported architecture.

    Doc:

    https://learn.microsoft.com/azure/search/search-how-to-index-sharepoint-online

    https://learn.microsoft.com/azure/search/search-indexer-sharepoint-access-control-lists https://learn.microsoft.com/azure/search/enrichment-cache-how-to-configure https://learn.microsoft.com/azure/search/enrichment-cache-how-to-manage https://learn.microsoft.com/azure/search/search-howto-run-reset-indexers https://learn.microsoft.com/azure/search/search-monitor-indexers

    https://learn.microsoft.com/azure/search/search-indexer-troubleshooting

    https://learn.microsoft.com/azure/search/agentic-knowledge-source-how-to-sharepoint-indexed

    Thanks

    Was this answer helpful?

    1 person found this answer helpful.
    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.