Azure Data Factory Data Flow CDC Source Not Refreshing When Using Dynamic Table Names

Question

Azure Data Factory Data Flow CDC Source Not Refreshing When Using Dynamic Table Names

Kishor kumar Reddy Akki 50

Environment

Azure Data Factory running in a Visual Studio subscription
SQL Server source using CDC-enabled tables
AdventureWorks sample database used for testing
Mapping Data Flow with:
- Source (SQL CDC)
  - Sink (Parquet or JSON)
  - Pipeline structure:

Lookup / Query activity to retrieve CDC-enabled tables
ForEach activity
Execute Data Flow activity

Approach:

The query successfully identifies CDC-enabled tables and passes the table names into the ForEach activity.

The ForEach activity successfully passes the current table name as a parameter into the Data Flow.

The Data Flow is configured to use this parameter to dynamically specify both the source and sink table/folder names.

For testing purposes I have used both:

Dataset-based sources
Inline sources

The behavior is identical in both scenarios.

Problem

Although the Data Flow receives a different table name parameter on each iteration, the source data does not appear to refresh correctly.

For example, I have CDC enabled on the following AdventureWorks tables:

Customer
SalesOrderHeader
SalesOrderDetail
CustomerAddress

During pipeline execution:

The correct table name is passed to the Data Flow on each iteration.
The sink location is dynamically named and may create folders corresponding to the current table.
However, regardless of which table is being processed, the output file always contains data from the first table processed (Customer).

This suggests that the source definition within the Data Flow is not being re-evaluated between iterations, even though the source parameter value changes.

Expected Behaviour

For each iteration of the ForEach loop:

The table name parameter should be passed into the Data Flow.
The Data Flow source should connect to the CDC stream for that table.
The sink should contain CDC data for the current table being processed.

I’ve attached two files to illustrate the behaviour. adf-input.txt is the Dataflow logs indicating the source table is SalesOrderDetail. Adf-output.txt demonstrates the schema which is that of AdventureworksLT Customer table.

Actual Behaviour

The table parameter changes on each iteration, but the data written to the sink always contains records from the first table processed (Customer).

The sink folder name may reflect the current table name, but the file contents do not.

Question

Is this expected behaviour or a known limitation of CDC sources in Mapping Data Flows when using dynamic source table names?

If supported, are there additional configuration requirements to ensure the CDC source is reinitialized for each Data Flow execution within a ForEach loop?Environment

Azure Data Factory running in a Visual Studio subscription
SQL Server source using CDC-enabled tables
AdventureWorks sample database used for testing
Mapping Data Flow with:
- Source (SQL CDC)
  - Sink (Parquet or JSON)
  - Pipeline structure:

Lookup / Query activity to retrieve CDC-enabled tables
ForEach activity
Execute Data Flow activity

Approach

The query successfully identifies CDC-enabled tables and passes the table names into the ForEach activity.

The ForEach activity successfully passes the current table name as a parameter into the Data Flow.

The Data Flow is configured to use this parameter to dynamically specify both the source and sink table/folder names.

For testing purposes I have used both:

Dataset-based sources
Inline sources

The behaviour is identical in both scenarios.

Problem

Although the Data Flow receives a different table name parameter on each iteration, the source data does not appear to refresh correctly.

For example, I have CDC enabled on the following AdventureWorks tables:

Customer
SalesOrderHeader
SalesOrderDetail
CustomerAddress

During pipeline execution:

The correct table name is passed to the Data Flow on each iteration.
The sink location is dynamically named and may create folders corresponding to the current table.
However, regardless of which table is being processed, the output file always contains data from the first table processed (Customer).

This suggests that the source definition within the Data Flow is not being re-evaluated between iterations, even though the source parameter value changes.

Expected Behaviour

For each iteration of the ForEach loop:

The table name parameter should be passed into the Data Flow.
The Data Flow source should connect to the CDC stream for that table.
The sink should contain CDC data for the current table being processed.

I’ve attached two files to illustrate the behavior.
adf-input.txt is the Dataflow logs indicating the source table is SalesOrderDetail. Adf-output.txt demonstrates the schema which is that of AdventureworksLT Customer table.

Actual Behavior

The table parameter changes on each iteration, but the data written to the sink always contains records from the first table processed (Customer).

The sink folder name may reflect the current table name, but the file contents do not.

Question

Whether dynamically changing the native CDC source table within the same Mapping Data Flow and Execute Data Flow activity is a supported design.
Whether the observed behavior is expected because the CDC checkpoint, source state, or source projection is reused across ForEach iterations.
Whether a separate checkpoint key, Execute Data Flow activity, Mapping Data Flow, or CDC configuration is required for each source table.
Whether any additional source, schema drift, projection, or sink configuration is required to support a metadata-driven framework at this scale.
Whether Microsoft can provide an official article or documented reference that confirms the supported configuration or applicable limitation.

SIVASANKAR YEDDULA 150 Reputation points Microsoft External Staff Moderator

2026-08-25T10:10:01.2966667+00:00

Hi @Kishor kumar Reddy Akki ,

Thank you for reaching out to Microsoft Q&A.

Based on the information provided, this appears to be a service-side platform issue that occurred during or immediately after the scale-up operation. The Resource Health status shows "Unavailable (Unplanned)" with an active Unknown Reason (Unplanned) event, indicating that Azure has detected an availability issue affecting the PostgreSQL Flexible Server.

Since the server is currently stuck in an Unavailable (Unplanned) state and management operations such as Stop, Start, and Restart are unresponsive, this is unlikely to be caused by a PostgreSQL configuration or application-level issue. The symptoms are more consistent with a failed infrastructure operation or a backend orchestration process that did not complete successfully during the scale-up activity.

I recommend the following actions:

Review the Activity Log for any failed scale-up, resize, or deployment operations that occurred immediately before the server became unavailable.

Verify whether there are any active Azure Service Health incidents affecting Azure Database for PostgreSQL Flexible Server in the target region.

Review the Resource Health blade and correlate the health events with the timestamp of the scale operation.

If the server remains unavailable and management operations continue to fail, open an Azure Support request and provide:

Subscription ID

Resource Group Name

Flexible Server Name

Region

Timestamp of the failed scale operation

Resource Health event details and screenshots

Activity Log Correlation IDs (if available)

Given that this is a production workload and the resource appears to be impacted by a backend platform issue, Azure Support should be engaged as soon as possible. The Azure Database for PostgreSQL service team can investigate the resource state, identify any stuck orchestration or deployment locks, and perform the required backend remediation.

As an additional mitigation step, if Point-in-Time Restore (PITR) is enabled and business continuity is critical, consider restoring the server to a new instance while the original server is being investigated. This may help reduce downtime depending on your recovery objectives.

I hope this helps. Please let us know if you have any further questions.

If this answer helps, please consider clicking Accept Answer so that other community members facing similar issues can benefit from it as well.
Kishor kumar Reddy Akki 50 Reputation points

2026-08-25T10:16:51.64+00:00
@SIVASANKAR YEDDULA ,

Thank you for your response.

However, the response appears to relate to a different issue involving Azure Database for PostgreSQL Flexible Server becoming unavailable after a scale-up operation.

Our question is specifically about Azure Data Factory Mapping Data Flow and SQL Server Change Data Capture. The customer is attempting to build a metadata-driven framework to process CDC data from approximately 200 SQL Server tables by dynamically passing the source table name through a ForEach activity.

The issue is as follows:

The correct table name is passed during each ForEach iteration.

The sink folder and table parameters reflect the current table.

However, the Mapping Data Flow source continues to resolve to the schema of the first table processed.

The diagnostic output shows Customer table columns even when the current iteration is intended to process SalesOrderDetail.

The same CDC checkpoint key is currently being used by the Execute Data Flow activity.

Could you please review the original ADF CDC question and confirm:

Whether native CDC in Mapping Data Flow supports dynamically changing the SQL Server source table within the same Data Flow activity.

Whether each source table requires a separate CDC checkpoint key.

Whether the CDC source definition and capture instance are reinitialized when the source table parameter changes.

Whether using a single generic Mapping Data Flow for multiple CDC-enabled tables is a supported design.

Whether this behavior represents a current product limitation or requires investigation by Azure Support.

Please note that there is no PostgreSQL Flexible Server, scale-up operation, Resource Health event, or server availability issue involved in this case.Thank you for your response.

However, the response appears to relate to a different issue involving Azure Database for PostgreSQL Flexible Server becoming unavailable after a scale-up operation.

Our question is specifically about Azure Data Factory Mapping Data Flow and SQL Server Change Data Capture. The customer is attempting to build a metadata-driven framework to process CDC data from approximately 200 SQL Server tables by dynamically passing the source table name through a ForEach activity.

The issue is as follows:

The correct table name is passed during each ForEach iteration.

The sink folder and table parameters reflect the current table.

However, the Mapping Data Flow source continues to resolve to the schema of the first table processed.

The diagnostic output shows Customer table columns even when the current iteration is intended to process SalesOrderDetail.

The same CDC checkpoint key is currently being used by the Execute Data Flow activity.

Could you please review the original ADF CDC question and confirm:

Whether native CDC in Mapping Data Flow supports dynamically changing the SQL Server source table within the same Data Flow activity.

Whether each source table requires a separate CDC checkpoint key.

Whether the CDC source definition and capture instance are reinitialized when the source table parameter changes.

Whether using a single generic Mapping Data Flow for multiple CDC-enabled tables is a supported design.

Whether this behavior represents a current product limitation or requires investigation by Azure Support.

Please note that there is no PostgreSQL Flexible Server, scale-up operation, Resource Health event, or server availability issue involved in this case.
SIVASANKAR YEDDULA 150 Reputation points Microsoft External Staff Moderator

2026-08-25T11:51:55.3333333+00:00
Hi @Kishor kumar Reddy Akki ,

Thank you for the clarification.

Based on the behavior described, this appears to be a limitation of how native CDC is implemented within Mapping Data Flow rather than an issue with parameter passing in the pipeline.

The fact that the correct table name is passed during each ForEach iteration, while the source schema continues to resolve to the first table processed, suggests that the CDC source metadata and checkpoint state are being reused across executions. As a result, changing the source table parameter alone may not force the CDC source definition to be reinitialized for a different capture instance.

For your specific questions:

Native CDC in Mapping Data Flow is not designed to dynamically switch between multiple CDC-enabled SQL Server tables within a single generic Data Flow while maintaining independent CDC state per table.

Each CDC-enabled source table should use its own unique checkpoint key. Reusing the same checkpoint key across multiple source tables can result in incorrect state tracking and unexpected CDC behavior.

Changing only the source table parameter does not guarantee that the CDC source metadata and capture instance will be reinitialized during execution. This can lead to the Data Flow continuing to reference the schema and CDC state from a previously processed table.

Using a single generic Mapping Data Flow to process hundreds of CDC-enabled tables with native CDC is not generally considered a recommended design pattern, particularly when different tables have different schemas and independent CDC checkpoints.

Based on the symptoms observed, this may represent a current product limitation of native CDC with Mapping Data Flow. If separate checkpoint keys per table do not resolve the issue, I recommend opening a support case with Azure Data Factory Support so the product team can verify whether the behavior is expected or whether a product investigation is required.

As a next step, I recommend testing with:

A unique CDC checkpoint key for each source table.

Separate CDC executions for a small subset of tables.

Validation of whether the schema correctly refreshes when each table maintains an independent CDC state.

If the issue persists after assigning unique checkpoint keys, please share the Data Flow configuration, CDC source settings, and checkpoint configuration details so that the behavior can be investigated further.
Refer https://learn.microsoft.com/en-us/azure/data-factory/concepts-change-data-capture

If this answer helps, please consider clicking Accept Answer so that other community members facing similar issues can benefit from it as well.
SIVASANKAR YEDDULA 150 Reputation points Microsoft External Staff Moderator

2026-08-25T17:07:01.6833333+00:00

Hi @Kishor kumar Reddy Akki ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
SIVASANKAR YEDDULA 150 Reputation points Microsoft External Staff Moderator

2026-08-27T11:54:46.58+00:00

Hi @Kishor kumar Reddy Akki ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.

1 answer

Your answer

SIVASANKAR YEDDULA 150 Reputation points Microsoft External Staff Moderator

2026-08-25T17:07:01.6833333+00:00

Hi @Kishor kumar Reddy Akki ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.
SIVASANKAR YEDDULA 150 Reputation points Microsoft External Staff Moderator

2026-08-27T11:54:46.58+00:00

Hi @Kishor kumar Reddy Akki ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.

Answer 1

Kishor kumar Reddy Akki 50

Thanks for your help with this request. we have run through the advice given and it has resolved the issue we were having.

SIVASANKAR YEDDULA 150 Reputation points Microsoft External Staff Moderator

2026-08-28T17:37:29.0166667+00:00

Hi @Kishor kumar Reddy Akki ,

I hope the information provided has been helpful. If you have any further questions or need additional assistance, please let us know in the comments, and we will be glad to help. If this answer helps, please consider clicking Accept Answer so that other community members with similar questions can benefit as well.