An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
Self-Hosted IR Copy activities hitting OutOfMemoryException and indefinite queuing since Synapse-to-Fabric migration doubled concurrent load
Here's the paragraph draft for the Q&A portal:
We're mid-migration from Synapse to Microsoft Fabric and are running the same SFTP-based Copy Data pipelines on both platforms concurrently, pulling from a local E:\ drive through a single Self-Hosted Integration Runtime (HA disabled, 16 concurrent job limit). Previously, on Synapse alone, our major run executed roughly 350 activities against E:\ without issue. Since bringing Fabric online with the same pipeline set now also pointed at doubled source directories, we've started seeing two problems: first, child activities failing on the source side with ErrorCode=SystemErrorOutOfMemory, Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, Message=A task failed with out of memory., Source=Microsoft.DataTransfer.TransferTask, ultimately a System.OutOfMemoryException from Microsoft.DataTransfer.ClientLibrary; and second, some pipeline runs on Synapse entering a queued state and never progressing toward completion, even when the SHIR node's concurrent job count is below its configured limit. Since the workload and IR configuration didn't change — only the addition of Fabric running the same pipelines in parallel with roughly double the directory/file volume — we suspect the two platforms are now contending for the same SHIR node's memory and connection capacity.
We'd like guidance on: (1) whether Synapse and Fabric pipelines sharing one Self-Hosted IR node is a known contention point that requires separate IR instances per platform during migration, (2) what's actually driving the OutOfMemoryException on the source side — node memory, SFTP client buffering, or something scaling with concurrent activity count — and how to size or tune around it, and (3) what could cause activities to sit queued indefinitely despite available concurrent-job headroom on the node.