Edit

Upgrade Azure Data Factory mapping data flows pipelines to Fabric (Preview)

Important

The upgrade experience for Azure Data Factory Mapping Data Flows to Fabric is currently in public preview and is subject to change.

Upgrade Azure Data Factory pipelines containing Mapping Data Flows to Microsoft Fabric Data Factory by following the steps in this article. This article covers the migration path for pipelines that use Mapping Data Flows, which are converted to mapping data flow (MDF) transforms in Dataflow Gen2. To migrate pipelines that don't contain Mapping Data Flows, see Upgrade your Azure Data Factory pipelines to Fabric.

Prerequisites

Before you start the upgrade, make sure you have the following:

  • Access to an Azure Data Factory instance containing Mapping Data Flows.
  • Access to a Microsoft Fabric-enabled tenant.
  • Contributor or higher permissions to the Fabric workspace.
  • Existing Fabric connections for supported data sources and destinations.

Supported upgrade scenarios

The upgrade tool supports the following scenarios:

  • Azure Data Factory pipelines containing Mapping Data Flows.
  • Azure Synapse Analytics pipelines containing Mapping Data Flows.
  • Lift-and-shift migration of existing Mapping Data Flow workloads to Fabric.

Start the migration

You can start migrating Azure Data Factory pipelines containing Mapping Data Flows from either of the following entry points.

Entry point Best for Starting step
From Azure Data Factory Running an assessment of pipeline readiness before migration Option A: Start from Azure Data Factory
From a Fabric workspace Directly mounting and migrating an Azure Data Factory instance from Fabric Option B: Start from Fabric

Both options continue at Migrate pipelines.

Option A: Start from Azure Data Factory

Use this option when you want to:

  • Review migration readiness
  • Run migration assessment
  • Evaluate supported pipelines before migration

Start migration from Azure Data Factory

  1. Open the Azure Data Factory portal.

  2. Select Migrate to Fabric (Preview).

  3. Select Get started.

    Screenshot of the Azure Data Factory portal showing the Migrate to Fabric (Preview) option and Get started button.

Review migration assessment

  1. On the Fabric Migration Assessment (Preview) page, review the list of supported pipelines available for migration.

  2. Select Next.

    Screenshot of the Fabric Migration Assessment page listing pipelines ready for migration.

Mount Azure Data Factory to Fabric

  1. Select the Fabric workspace where you want to mount the Azure Data Factory instance.

  2. Select Mount.

  3. After the mount operation completes, select Continue in Fabric.

    Screenshot of the Fabric workspace selection screen used to mount an Azure Data Factory instance.

    Screenshot of the Continue in Fabric screen for opening the mounted Azure Data Factory instance in Microsoft Fabric.

Option B: Start from Fabric

Use this option when you already know which Azure Data Factory instance you want to migrate and want to start directly from Fabric.

Mount Azure Data Factory from Fabric

  1. Open your Fabric workspace.

  2. Select New item.

  3. Select Azure Data Factory.

    Screenshot of the Microsoft Fabric workspace New item experience showing Azure Data Factory selection.

  4. Select the Azure Data Factory instance to mount.

  5. Select OK.

    Screenshot of the Azure Data Factory instance selection dialog.

Migrate pipelines

After you mount Azure Data Factory in Fabric, continue with the migration workflow.

Select pipelines to migrate

  1. Open the mounted Azure Data Factory item in Fabric.

  2. Select Migrate to Fabric (Preview).

    Screenshot of the mounted Azure Data Factory item in Fabric showing the Migrate to Fabric (Preview) option.

  3. Select the pipeline containing the Mapping Data Flow.

  4. Select Review connections.

    Screenshot of the pipeline selection screen showing pipelines ready for migration.

Map linked services to Fabric connections

  1. On the Map connections page:

    • Review Azure Data Factory linked services.
    • Map them to Fabric connections.

  2. Select Confirm.

    Screenshot of the Map connections page for mapping Azure Data Factory linked services to Fabric connections.

Review migration results

After migration completes, the tool:

  • Migrates pipelines and Mapping Data Flows together.
  • Converts Mapping Data Flows to MDF transforms in Dataflow Gen2.
  • Places migrated artifacts in a new folder in the Fabric workspace.

Select View in workspace.

Screenshot of the migration results page showing migrated resources and their statuses.

Validate migrated mapping data flow transforms

After migration, open the migration folder and validate the upgraded MDF transforms before you run them.

Open the migrated Dataflow Gen2 item

  1. Open the migrated Dataflow Gen2 item.

  2. Review the migrated MDF transform logic.

  3. Validate the transformation graph using the MDF transform toolbar.

    Screenshot of the mapping data flow transform authoring experience showing the migrated transformation graph.

Save the migrated mapping data flow transform

  1. Open the Save & run menu.

  2. Select Save.

    Screenshot of the Save and run menu in the mapping data flow transform authoring experience.

Note

Only the Save action is currently supported for MDF transforms during public preview.

Run migrated pipelines

After validation, run the migrated Fabric pipeline.

Configure the pipeline run

  1. Open the migrated Fabric pipeline.

  2. Select the Dataflow activity.

  3. Review the selected MDF transform query.

  4. Configure Spark runtime settings if needed.

    Screenshot of the pipeline editor showing the Dataflow activity configuration for a mapping data flow transform.

Run the migrated pipeline

  1. Validate the pipeline.

  2. Run the pipeline manually or configure schedules and triggers.

    Screenshot of the pipeline run and scheduling options for the migrated pipeline.

Monitor migrated mapping data flow transform executions

You can monitor migrated pipeline and MDF transform executions through:

  • Pipeline output pane
  • Monitoring Hub
  • Activity Runs
  • Dataflow activity execution details

Screenshot of the pipeline output pane showing the Dataflow activity run status.

To review execution details:

  1. Open the pipeline run details.

  2. Select the Dataflow activity from Activity Runs.

  3. Review execution status and runtime details.

    Screenshot of the Monitoring Hub showing pipeline activity runs and their statuses.

    Screenshot of the Dataflow activity execution details showing processing metrics.

Limitations

The following limitations currently apply during public preview:

Area Limitation
Flowlets Not supported.
Data Flow Library Not supported.
User-defined functions (UDFs) Not supported.
Dataflow execution MDF transforms can only be executed through the Pipeline Dataflow activity. Refresh dataflow isn't supported.
Managed Virtual Network Managed Virtual Network support isn't available.
Spark runtime execution MDF transforms currently use Spark runtime infrastructure similar to Azure Data Factory and Azure Synapse Analytics Mapping Data Flows.

Related content

  • Last updated on