URGENT: Our sharepoint pipelines on ADF has been failing due to ACS authentication depreciation

Santosh Sudhindira 40 Reputation points
2026-07-13T04:04:54.4833333+00:00

Migration steps taken:

**•**	Received a new .pfx certificate; imported into Key Vault (base64-encoded secret), with the certificate password stored as a separate secret

**•**	Created a new **REST linked service** with authenticationType: AadServicePrincipal, servicePrincipalCredentialType: ServicePrincipalCert, referencing the cert + password secrets in Key Vault, aadResourceId: https://{tenant}.sharepoint.com

**•	Confirmed working**: this REST linked service successfully authenticates and retrieves data via _api/web/GetFileByServerRelativeUrl('...')/$value when the sink is a **JSON dataset** — verified with a working Copy Activity (auth succeeds, correct file bytes returned, only fails at JSON deserialization since the payload is genuinely binary XLSX content)

**•	Blocked**: when attempting to use this same REST linked service with a **Binary-format dataset** as the sink, Copy Activity fails with UserErrorFormatIsRequired / format-related errors — regardless of manually editing the dataset JSON to reference the REST linked service (the Studio dataset picker only allows Binary format to pair with HTTP-type linked services, not REST-type)

Also tested and ruled out:

**•**	HTTP linked service with authenticationType: ClientCertificate (mTLS) + Binary dataset → connects (TLS handshake succeeds) but SharePoint REST API returns 403 Forbidden on the actual file request, confirming SharePoint’s app-only API requires OAuth2 bearer token auth (appidacr=2), not mTLS

**•**	Verified certificate/app registration/permissions are correct via a standalone Python/MSAL script outside ADF — successfully acquires a token and retrieves the same file via requests.get() with the bearer token, confirming the certificate and Entra app config are valid

Question:

Is there a supported way in ADF to use a REST-type linked service with certificate-based Service Principal authentication as the source for a Binary-format Copy Activity sink? If not, is the officially recommended pattern for this scenario (cert-based app-only auth + binary file download from SharePoint) to:

(a) proxy through an intermediary (e.g., Azure Function) that performs the authenticated call and streams bytes back over plain HTTP, or

(b) use Microsoft Graph’s @microsoft.graph.downloadUrl pattern instead (though our tenant’s Graph /sites calls currently return generalException/401 even with valid tokens — separate issue, possibly a Sites.Selected per-site grant gap we’re still confirming with our Entra admin)?

Any guidance on the supported native ADF pattern here — without resorting to a custom proxy — would be appreciated.

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

Answer accepted by question author
Alex Burlachenko 25,285 Reputation points MVP Volunteer Moderator
2026-07-13T09:18:26.3+00:00

hi Santosh Sudhindira & thx for sharing urs issue here at Q&A portal,

ur testing already confirms the main limitation: the REST connector can authenticate to SharePoint with the certificate, but ADF does not support using a REST dataset as a raw Binary source. Binary format is supported with the HTTP connector, not the REST connector. https://learn.microsoft.com/azure/data-factory/format-binary & https://learn.microsoft.com/azure/data-factory/connector-rest So there isn’t a clean native Copy Activity combination that gives u both REST certificate-based OAuth and binary pass-through in one step. The HTTP ClientCertificate option is mTLS, not OAuth client-certificate authentication, so the SharePoint 403 is expected. It proves the certificate reached the TLS layer, but SharePoint still didn’t receive the required bearer token.

The most practical supported pattern is an Azure Function or Logic App that gets the SharePoint token with MSAL and streams the file to ADF or directly into Storage. Microsoft Graph is another valid path, but @microsoft.graph.downloadUrl still requires the app to have working SharePoint permissions first. A missing Sites.Selected site grant could explain the current 401/generalException. Unless Microsoft adds Binary support to the REST connector, the proxy/custom activity pattern is the safer answer here. Trying to force the REST linked service into a Binary dataset through JSON won’t make an unsupported connector-format combination work.

rgds,

Alex

&

If my answer was helpful pls mark it and additional thx if u follow me at Q&A portal

and at my blog https://ctrlaltdel.blog/

 

Was this answer helpful?

1 person found this answer helpful.
0 comments
Answer accepted by question author
Harold Picado 160 Reputation points
2026-07-13T07:26:48.2033333+00:00

Hello, I am an independent contributor and a Microsoft Certified Azure Solutions Architect.

Based on my research, there is currently no native way in Azure Data Factory (ADF) to use a REST linked service with certificate-based Service Principal authentication as the source for a Binary dataset. The commonly recommended approach is either to use an intermediary service (such as an Azure Function or Logic App) to perform the authenticated download, or to use Microsoft Graph (for example, the @microsoft.graph.downloadUrl pattern) once the required permissions (such as Sites.Selected with the appropriate site-level grants) have been configured.

Recommended approach:

  • Short term: Implement an Azure Function (or similar intermediary) to restore the pipelines as quickly as possible.
  • Long term: Work with your Entra ID and SharePoint administrators to configure the required Microsoft Graph permissions and migrate to the Graph download endpoint as the supported API for SharePoint file downloads.
  • Avoid: Attempting to force Binary datasets to work with REST linked services in ADF, as this scenario is not currently supported.

Regards.

Was this answer helpful?

1 person found this answer helpful.
Answer recommended by moderator
Santosh Sudhindira 40 Reputation points
2026-07-15T04:23:38.0166667+00:00

Posting the full resolution for anyone hitting the same wall migrating off SharePoint ACS.

Root causes (there were two, layered on top of each other):

**1.	Missing Microsoft Graph API permission.** Our Entra app had Sites.Read.All granted under the **SharePoint** API, but nothing under **Microsoft Graph**. These are registered separately even though both touch SharePoint data. This caused every graph.microsoft.com/v1.0/sites/... call to fail with a generic generalException/401, even with a fully valid, correctly-signed app-only token (verified independently via a standalone Python/MSAL script outside ADF). Fix: added Sites.Read.All under **Microsoft Graph → Application permissions**, granted admin consent.

**2.	ADF’s Binary dataset format cannot pair with a REST-type linked service.** We confirmed cert-based Service Principal auth works correctly via a REST linked service (AadServicePrincipal + ServicePrincipalCert), including successfully retrieving real file bytes — but Copy Activity’s data movement engine rejects Binary output from a RestService-type source at a platform level, even when manually forcing the dataset JSON to reference the REST linked service. This isn’t fixable through configuration.

Working solution: two-step pattern using Microsoft Graph’s pre-signed download URL

**1.	Web Activity** — calls Graph (/v1.0/sites/{hostname}:/sites/{site} then /v1.0/sites/{site-id}/drive/root:/{path}) using native Service Principal + Certificate auth (cert stored in Key Vault via az keyvault certificate import, referenced directly  no separate password field needed). Returns file metadata JSON including @microsoft.graph.downloadUrl, a temporary pre-signed URL requiring no auth.

**2.	Set Variable** - captures that URL: @activity('GetMetadata').output['@microsoft.graph.downloadUrl']

**3.	Copy Activity** - HTTP linked service with **Anonymous** auth, Binary dataset, source URL = the pre-signed link (with the domain prefix stripped via replace(), since Base URL + full absolute URL otherwise double-concatenate). Sink = ADLS Binary dataset, unchanged.

This sidesteps the Binary/REST pairing restriction entirely, since the actual byte transfer goes through a plain anonymous HTTP+Binary combination - the only auth-heavy step is the small JSON metadata call.

Key gotchas along the way, for anyone following this pattern:

**•**	Graph’s colon-segment path addressing (/sites/{host}:/sites/{site}:/drive/root:/{path}) doesn’t reliably support chaining two segments in one call -  resolve the site ID first, then address the file separately.

**•**	Graph’s drive root **is** the document library itself (Shared Documents) - don’t include that as a path segment, or you’ll get itemNotFound.

**•**	SharePoint’s backend enforces appidacr: 2 (certificate-based tokens only) on app-only calls proxied through it - client secrets get silently rejected on some SharePoint-backed endpoints

Was this answer helpful?

1 person found this answer helpful.
0 comments

2 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Srikanth Chavithina 265 Reputation points Microsoft External Staff Moderator
    2026-07-13T09:19:06.27+00:00

    Hi @Santosh Sudhindira ,

    Thank you for sharing the detailed validation steps.

    Your testing confirms that certificate-based authentication through the REST linked service is working correctly. The limitation is with the connector and dataset combination: the REST connector cannot be directly paired with a Binary dataset for this Copy Activity scenario.

    The HTTP connector supports Binary datasets, but its ClientCertificate authentication is not equivalent to the OAuth certificate-based service principal authentication required by SharePoint. Also, acquiring a certificate-based OAuth token through a Web Activity and passing it to an HTTP Copy Activity is not a straightforward native alternative, because the certificate flow requires a signed client assertion.

    Therefore, manually forcing a REST linked service to work with a Binary dataset is not a supported workaround. For this scenario, using an intermediary service such as an Azure Function to perform the certificate-authenticated request and handle the binary content is the practical option.

    Alternatively, Microsoft Graph can be evaluated after resolving the current authentication issue. If Sites.Selected is being used, please verify that the application has the required Graph application permission, admin consent, and an explicit permission grant for the target SharePoint site.

    I hope this clarifies your query. If you have any further questions or need additional assistance,
    please feel free to reach out.

    Was this answer helpful?

    0 comments
  2. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

    Comments have been turned off. Learn more

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.