Azure Data Factory Data Flow Trigger Runs Successfully but Source File Is Not Updated (Debug Works Fine)

Hi @Saksham Dubey ,

Thank you for reaching out to the Microsoft Q&A forum.

Based on the behavior described, the Data Flow logic itself appears to be working correctly because:

The Data Flow successfully updates the sink file in Debug mode.

The triggered run completes successfully.

The monitoring output indicates that 113 rows were processed.

However, the expected file in the sink location is not updated after the triggered execution.

This typically indicates a difference between the Debug execution context and the Trigger execution context rather than an issue with the transformation logic.

We recommend validating the following areas:

Sink Path and Dataset Configuration

Verify that the triggered run is writing to the same storage account, container, folder, and file path used during Debug execution.

Check whether dataset parameters are resolving to different values during the triggered run.

Pipeline and Data Flow Parameters

Compare parameter values passed during Debug and Trigger executions.

Ensure that environment-specific values are not redirecting output to a different location.

Sink Write Settings

Review the sink write behavior (Overwrite, Insert, Upsert, Merge, etc.).

Confirm that the trigger run is not creating new partition folders or writing to an unexpected location.

Integration Runtime and Permissions

Validate that the Integration Runtime used by the triggered pipeline has the same storage permissions and configuration as the Debug session.

Storage Validation

Check for newly created files, partition folders, or timestamped output files in the target storage account.

Verify whether the Data Flow is writing successfully but to a different output path.

Activity Output Review

Open the Data Flow activity output from the triggered run and review the sink statistics, output path details, and row counts.

Confirm that rows are actually reaching the sink and are not being filtered before the write operation.

Since the run completes successfully and processes records, the most likely causes are parameter differences, sink path resolution issues, overwrite behavior, or environment-specific configuration differences between Debug and Trigger executions.

Could you please share:

A screenshot of the Sink configuration.

The Data Flow activity output from the triggered run.

Details of any parameters being passed from the pipeline to the Data Flow.

This information will help determine why the triggered run reports success while the expected sink file is not being updated.

If this answer helps, please consider clicking Accept Answer so that other community members facing similar issues can benefit from it as well.

If you have any further questions, please let us know. We will be happy to assist

Hi ,

Thanks for reaching out to Microsoft Q&A.

The fact that the Data Flow works correctly in Debug mode but does not appear to update the expected file when executed through a triggered pipeline run usually indicates a difference in runtime configuration rather than a Data Flow failure.

Since the activity reports Succeeded and shows 113 rows processed, the Data Flow is reading and processing data successfully. The next step is to confirm where the triggered execution is actually writing its output.

1. Compare Debug and Triggered Run Parameters
   • Verify dataset parameters, Data Flow parameters, pipeline parameters, and any dynamic content used in source or sink paths.
   • Ensure the triggered run resolves to the same source and sink locations that were used during debugging.
2. Validate the Actual Source and Sink Paths
   • From the Data Flow activity output in Monitor, confirm:
   • Source file/folder being read
   • Sink file/folder being written
   • A common cause is that the trigger resolves to a different path than expected.
3. Review Sink Configuration
   • Check whether the sink is configured for overwrite, append, update, or upsert behavior.
   • Verify that key columns and mappings are correctly configured if update/upsert logic is being used.
   • Confirm that the output file name is not dynamically generated.
4. Check Partitioning Behavior
   • Mapping Data Flows may write multiple output files depending on the sink partitioning settings.
   • The original file may remain unchanged while new output files are created successfully in the target location.
5. Check for Concurrent Runs
   • If multiple pipeline executions target the same output location, one run may overwrite or replace the results of another.
   • Review pipeline history for overlapping executions.

Information that would helpt o investigate further:

Could you share:

• Source dataset configuration (including parameters)
• Sink configuration
• Pipeline and trigger parameter settings
• Data Flow activity Output from the triggered run
• Expected source and sink paths
• Whether the sink is configured as overwrite, append, update, or upsert

The key point is that "113 rows processed" confirms that records were processed by the Data Flow, but it does not by itself confirm that the expected storage object was updated. I would first verify the exact sink path and file name recorded in the triggered run output and compare them with the successful debug execution.

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.