Self-Hosted IR Copy activities hitting OutOfMemoryException and indefinite queuing since Synapse-to-Fabric migration doubled concurrent load

Mohd Anas 20 Reputation points
2026-09-15T15:41:25.6566667+00:00

Here's the paragraph draft for the Q&A portal:


We're mid-migration from Synapse to Microsoft Fabric and are running the same SFTP-based Copy Data pipelines on both platforms concurrently, pulling from a local E:\ drive through a single Self-Hosted Integration Runtime (HA disabled, 16 concurrent job limit). Previously, on Synapse alone, our major run executed roughly 350 activities against E:\ without issue. Since bringing Fabric online with the same pipeline set now also pointed at doubled source directories, we've started seeing two problems: first, child activities failing on the source side with ErrorCode=SystemErrorOutOfMemory, Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, Message=A task failed with out of memory., Source=Microsoft.DataTransfer.TransferTask, ultimately a System.OutOfMemoryException from Microsoft.DataTransfer.ClientLibrary; and second, some pipeline runs on Synapse entering a queued state and never progressing toward completion, even when the SHIR node's concurrent job count is below its configured limit. Since the workload and IR configuration didn't change — only the addition of Fabric running the same pipelines in parallel with roughly double the directory/file volume — we suspect the two platforms are now contending for the same SHIR node's memory and connection capacity.

We'd like guidance on: (1) whether Synapse and Fabric pipelines sharing one Self-Hosted IR node is a known contention point that requires separate IR instances per platform during migration, (2) what's actually driving the OutOfMemoryException on the source side — node memory, SFTP client buffering, or something scaling with concurrent activity count — and how to size or tune around it, and (3) what could cause activities to sit queued indefinitely despite available concurrent-job headroom on the node.

Azure Synapse Analytics
Azure Synapse Analytics

An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.

0 comments No comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.