Data Reading issues from EVENTHub to Databricks using DLT.

Hi ,

Thanks for reaching out to Microsoft Q&A.

The main point is that Lakeflow Declarative Pipelines (formerly DLT) manage the compute for you, so you generally do not select or change the cluster access mode of the underlying serverless compute. Serverless pipeline compute is Databricks-managed and does not support manually configuring cluster settings such as access mode.

Additionally, if you're using:

spark.readStream
.format("eventhubs")
.options(**ehConf)
.load()

be aware that Lakeflow pipelines do not support the traditional Azure Event Hubs Spark connector, because it requires a third-party JVM library. Databricks recommends connecting to Azure Event Hubs through its Kafka-compatible endpoint and using the built-in Structured Streaming Kafka connector instead.

Therefore, I would recommend:

1. Verify whether your pipeline is running on Serverless or Classic/Pro compute.
2. If using Lakeflow Declarative Pipelines, use the Kafka connector with the Event Hubs Kafka endpoint rather than the eventhubs format.
3. Do not attempt to change the access mode of the automatically created serverless pipeline cluster.
4. If you have a specific organizational requirement for Single User access, review whether that requirement applies to the pipeline itself or to a separate interactive cluster. Event Hubs ingestion does not inherently require a Single User cluster

please share the exact error message and whether you're using Serverless Lakeflow Pipelines, Classic/Pro Pipelines, or a regular Databricks cluster.

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Hi @Damodara, K

Please follow the below steps will helpful to fix your issue.

Step1 :

Go to: Workflows → Delta Live Tables → Your Pipeline → Settings

Step 2:

Under Compute, change: Mode: Serverless → Custom

This unlocks the ability to choose your own cluster.

Step3:

Choose a policy like:

single_user_access

no_shared_compute

private_network_only

If you don’t see any → your admin must enable cluster policies.

Step4:

In pipeline settings → Advanced → Configuration:

spark.eventhubs.connectionString = "<your EH connection>"

spark.eventhubs.consumerGroup = "$Default"

spark.eventhubs.maxEventsPerTrigger = 10000

Now your streaming code will work.

Thanks.