Azure Data Factory Data Flow Trigger Runs Successfully but Source File Is Not Updated (Debug Works Fine)

Question

Azure Data Factory Data Flow Trigger Runs Successfully but Source File Is Not Updated (Debug Works Fine)

Saksham Dubey 0

We are currently experiencing an issue with Azure ADF data flows. While the data flow functions as expected in debug mode and correctly updates the storage file in sink, it fails to update the storage file when triggered, even though the logs indicate that the run was successful and 113 have been processed in QA instance.

SIVASANKAR YEDDULA 165 Reputation points Microsoft External Staff Moderator

2026-08-14T11:16:41.2466667+00:00

Hi Saksham Dubey
Since the Data Flow works in Debug mode but the triggered run does not update the sink file, please compare the Debug and triggered executions.
Make sure the latest Data Flow changes are published before triggering.
Verify that the source and sink paths/file names are the same in both runs.
Check the parameters and dynamic expressions passed by the trigger.
Verify the Integration Runtime, because Debug uses the active Debug cluster, while the triggered run uses the Integration Runtime configured for the activity.
In Monitor → Data Flow Activity, check the source rows read and sink rows written.
If rows are written successfully, verify the actual sink location, file name, partitioning, and write settings.

Also verify the source/sink connection and permissions as recommended in Microsoft’s troubleshooting guidance.

Recommended conclusion: The most important step is to compare the Debug and triggered run details, especially the parameters, source path, sink path, Integration Runtime, and rows written. This should help identify why the triggered execution is behaving differently.

Reference: https://learn.microsoft.com/en-us/azure/data-factory/data-flow-troubleshoot-guide
Saksham Dubey 0 Reputation points

2026-08-14T11:20:27.14+00:00

Hi @SIVASANKAR YEDDULA , Thankyou for your response. I tested on published branch itself and verified all the given step as well still getting this discrepancy.
SIVASANKAR YEDDULA 165 Reputation points Microsoft External Staff Moderator

2026-08-14T12:17:13.8833333+00:00

HiSaksham Dubey

Since the published branch and all previously recommended checks have already been verified, please run the Data Flow using Debug → Use Activity Runtime.
This will use the same Integration Runtime configured for the pipeline instead of the separate Debug cluster.
After that, compare the Use Activity Runtime result with the triggered run, especially the source/sink path, parameters, rows read/written, and sink output.
If the issue occurs only during the triggered execution, this will help isolate whether the difference is related to the pipeline runtime or trigger execution.
Microsoft recommends using Trigger Now / Debug → Use Activity Runtime to validate the pipeline’s configured Integration Runtime.
SIVASANKAR YEDDULA 165 Reputation points Microsoft External Staff Moderator

2026-08-17T08:58:07.7666667+00:00

Hi Saksham Dubey
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

3 answers

Your answer

SIVASANKAR YEDDULA 165 Reputation points Microsoft External Staff Moderator

2026-08-14T11:16:41.2466667+00:00

Hi Saksham Dubey
Since the Data Flow works in Debug mode but the triggered run does not update the sink file, please compare the Debug and triggered executions.
Make sure the latest Data Flow changes are published before triggering.
Verify that the source and sink paths/file names are the same in both runs.
Check the parameters and dynamic expressions passed by the trigger.
Verify the Integration Runtime, because Debug uses the active Debug cluster, while the triggered run uses the Integration Runtime configured for the activity.
In Monitor → Data Flow Activity, check the source rows read and sink rows written.
If rows are written successfully, verify the actual sink location, file name, partitioning, and write settings.

Also verify the source/sink connection and permissions as recommended in Microsoft’s troubleshooting guidance.

Recommended conclusion: The most important step is to compare the Debug and triggered run details, especially the parameters, source path, sink path, Integration Runtime, and rows written. This should help identify why the triggered execution is behaving differently.

Reference: https://learn.microsoft.com/en-us/azure/data-factory/data-flow-troubleshoot-guide
Saksham Dubey 0 Reputation points

2026-08-14T11:20:27.14+00:00

Hi @SIVASANKAR YEDDULA , Thankyou for your response. I tested on published branch itself and verified all the given step as well still getting this discrepancy.
SIVASANKAR YEDDULA 165 Reputation points Microsoft External Staff Moderator

2026-08-14T12:17:13.8833333+00:00

HiSaksham Dubey

Since the published branch and all previously recommended checks have already been verified, please run the Data Flow using Debug → Use Activity Runtime.
This will use the same Integration Runtime configured for the pipeline instead of the separate Debug cluster.
After that, compare the Use Activity Runtime result with the triggered run, especially the source/sink path, parameters, rows read/written, and sink output.
If the issue occurs only during the triggered execution, this will help isolate whether the difference is related to the pipeline runtime or trigger execution.
Microsoft recommends using Trigger Now / Debug → Use Activity Runtime to validate the pipeline’s configured Integration Runtime.
SIVASANKAR YEDDULA 165 Reputation points Microsoft External Staff Moderator

2026-08-17T08:58:07.7666667+00:00

Hi Saksham Dubey
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.

Answer 1

This pattern usually points to a difference between data preview/debug behavior and pipeline-triggered execution.

Verify the sink by running the data flow from a pipeline debug or trigger In mapping data flows, Data Preview does not write to the sink. It only shows a snapshot of transformed data in Spark memory, and sink drivers are not tested there. To validate actual file output, run the data flow through an Execute Data Flow activity in a pipeline.
Check source and sink connectivity first In each source and sink transformation, open the linked service for each dataset and test the connection.
Use Data Preview only to confirm source access In debug mode, open Data Preview on the source transformations and confirm the source data is accessible. If the source data changed, use Refresh > Refetch from source.
Do an end-to-end pipeline debug Put the data flow in a pipeline activity and debug the pipeline end to end. This is the correct way to test whether the sink file is actually written.
Check whether source file operations are expected in debug mode If the flow depends on source file operations such as move or delete after completion, those operations run only when the data flow is started from a pipeline run using Execute Data Flow. They do not run in Data Flow debug mode.
If the run shows success but the sink file is not updated, check sink and storage configuration For Azure Data Lake Storage Gen2 sinks, a known cause is insufficient permissions for the runtime identity. If using Managed Identity or Service Principal, ensure the linked service has R/W/E permission, and at minimum grant Storage Blob Data Contributor on the storage account.
Check for file-handling jobs that interfere with sink output If another process moves or deletes files in the target folder, data flow output can be affected. Data flows can first write partition files into the target folder and then perform merge/rename operations, so intermediate file names might not match custom file-handling rules.
Review sink settings if using hierarchical namespace storage with certain options Some combinations can fail when using an Azure Blob Storage linked service against a hierarchical namespace enabled storage account. Known problematic settings include Clear the folder, nondefault File name option, and Key partitioning. A recommended fix is to create a new Azure Data Lake Storage Gen2 linked service for that storage account, or disable those settings and test again.
If the issue is intermittent, add retries Successful-looking runs with missing output can also be related to transient execution issues. Configure retries on the pipeline activity.
If using parallel execution, avoid sharing the same debug cluster Multiple parallel runs submitted to the same debug cluster can cause resource failures. For actual execution, use Trigger Now or Debug > Use Activity Runtime so the run uses the integration runtime defined in the pipeline activity.

A practical validation path for the QA instance is:

Test source and sink linked services.
In the data flow source, use Data Preview and Refetch from source.
Run the pipeline with Execute Data Flow.
Check the sink folder after the run.
If no updated file appears, verify the runtime identity permissions on ADLS Gen2.
If permissions are correct, review sink settings such as Clear the folder, file naming, and partitioning, and confirm no external job is moving or deleting intermediate files.

References:

Answer 2

Hi ,

Thanks for reaching out to Microsoft Q&A.

The fact that the Data Flow works correctly in Debug mode but does not appear to update the expected file when executed through a triggered pipeline run usually indicates a difference in runtime configuration rather than a Data Flow failure.

Since the activity reports Succeeded and shows 113 rows processed, the Data Flow is reading and processing data successfully. The next step is to confirm where the triggered execution is actually writing its output.

Compare Debug and Triggered Run Parameters
- Verify dataset parameters, Data Flow parameters, pipeline parameters, and any dynamic content used in source or sink paths.
- Ensure the triggered run resolves to the same source and sink locations that were used during debugging.
Validate the Actual Source and Sink Paths
- From the Data Flow activity output in Monitor, confirm:
- Source file/folder being read
- Sink file/folder being written
- A common cause is that the trigger resolves to a different path than expected.
Review Sink Configuration
- Check whether the sink is configured for overwrite, append, update, or upsert behavior.
- Verify that key columns and mappings are correctly configured if update/upsert logic is being used.
- Confirm that the output file name is not dynamically generated.
Check Partitioning Behavior
- Mapping Data Flows may write multiple output files depending on the sink partitioning settings.
- The original file may remain unchanged while new output files are created successfully in the target location.
Check for Concurrent Runs
- If multiple pipeline executions target the same output location, one run may overwrite or replace the results of another.
- Review pipeline history for overlapping executions.

Information that would helpt o investigate further:

Could you share:

Source dataset configuration (including parameters)
Sink configuration
Pipeline and trigger parameter settings
Data Flow activity Output from the triggered run
Expected source and sink paths
Whether the sink is configured as overwrite, append, update, or upsert

The key point is that "113 rows processed" confirms that records were processed by the Data Flow, but it does not by itself confirm that the expected storage object was updated. I would first verify the exact sink path and file name recorded in the triggered run output and compare them with the successful debug execution.

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Answer 3

Hi @Saksham Dubey ,

Thank you for reaching out to the Microsoft Q&A forum.

Based on the behavior described, the Data Flow logic itself appears to be working correctly because:

The Data Flow successfully updates the sink file in Debug mode.

The triggered run completes successfully.

The monitoring output indicates that 113 rows were processed.

However, the expected file in the sink location is not updated after the triggered execution.

This typically indicates a difference between the Debug execution context and the Trigger execution context rather than an issue with the transformation logic.

We recommend validating the following areas:

Sink Path and Dataset Configuration

Verify that the triggered run is writing to the same storage account, container, folder, and file path used during Debug execution.

Check whether dataset parameters are resolving to different values during the triggered run.

Pipeline and Data Flow Parameters

Compare parameter values passed during Debug and Trigger executions.

Ensure that environment-specific values are not redirecting output to a different location.

Sink Write Settings

Review the sink write behavior (Overwrite, Insert, Upsert, Merge, etc.).

Confirm that the trigger run is not creating new partition folders or writing to an unexpected location.

Integration Runtime and Permissions

Validate that the Integration Runtime used by the triggered pipeline has the same storage permissions and configuration as the Debug session.

Storage Validation

Check for newly created files, partition folders, or timestamped output files in the target storage account.

Verify whether the Data Flow is writing successfully but to a different output path.

Activity Output Review

Open the Data Flow activity output from the triggered run and review the sink statistics, output path details, and row counts.

Confirm that rows are actually reaching the sink and are not being filtered before the write operation.

Since the run completes successfully and processes records, the most likely causes are parameter differences, sink path resolution issues, overwrite behavior, or environment-specific configuration differences between Debug and Trigger executions.

Could you please share:

A screenshot of the Sink configuration.

The Data Flow activity output from the triggered run.

Details of any parameters being passed from the pipeline to the Data Flow.

This information will help determine why the triggered run reports success while the expected sink file is not being updated.

If this answer helps, please consider clicking Accept Answer so that other community members facing similar issues can benefit from it as well.

If you have any further questions, please let us know. We will be happy to assist