Azure Stream Analytics — Nested JSON fields (record type) output as null in Parquet format despite being correctly populated in query test results

Question

Azure Stream Analytics — Nested JSON fields (record type) output as null in Parquet format despite being correctly populated in query test results

GP13 6

We are using Azure Stream Analytics to read Debezium CDC events from Azure Event Hubs and write to ADLS Gen2 in Parquet format.

Event structure from Debezium:

json

{
  "before": null,
  "after": {
    "id": 1,
    "field1": "value1",
    "field2": 100,
    "field3": null
  },
  "source": {
    "version": "2.4.2.Final",
    "db": "mydb",
    "table": "mytable"
  },
  "op": "c",
  "ts_ms": 1234567890000
}

Observed behaviour:

ASA Input preview shows after and source as record type with actual data
ASA Test query results show after and source correctly populated
Output Parquet files show before, after and source as null
Scalar fields (op, ts_ms) are correctly populated in Parquet output
Switching output format to JSON — works correctly, all fields populated

What we have tried:

TRY_CAST AS record — nulls in Parquet
TRY_CAST AS nvarchar(max) — nulls in Parquet
No casting — nulls in Parquet
Explicit column list in final SELECT instead of SELECT * — nulls in Parquet
JSON output — works correctly

Questions:

Is there a known way to correctly serialise nested JSON record fields to Parquet in ASA?
Is TRY_CAST AS record supported for Parquet output or is this a known limitation?
We have seen this work in a previous ASA job with simpler test data — has ASA behaviour changed recently regarding nested types in Parquet output?

Environment:

Azure Stream Analytics — Standard tier
Input: Azure Event Hubs, JSON serialisation
Output: ADLS Gen2, Parquet format, Append mode
Debezium MySQL connector 2.4.2We are using Azure Stream Analytics to read Debezium CDC events from Azure Event Hubs and write to ADLS Gen2 in Parquet format.

Ganesh Chelluri 195 Reputation points Microsoft External Staff Moderator

2026-07-17T23:42:08.6966667+00:00
Hi @GP13 ,

Just circling back on this one so you have everything in one place to move forward on the Debezium CDC to Parquet piece.

Yes, you can absolutely land the Debezium envelope (before / after / op / ts_ms) into Parquet from ASA it just needs to be projected as flat, strongly typed scalar columns in the query, since Parquet output in ASA is schema-on-write and doesn't materialize record-typed columns as nested structs. The resulting Parquet file still gives you the full CDC semantics you referenced:

before_* columns -> state before the change (naturally null for op = 'r' and 'c', populated for 'u' and 'd')

after_* columns -> state after the change (populated for 'r', 'c', 'u'; null for 'd')

op -> r / c / u / d

ts_ms -> source change timestamp in ms

Applying that to your payload (adjust column names/types to match mytable's actual schema):

SELECT

-- before.* TRY_CAST([before].[id] AS bigint) AS before_id, TRY_CAST([before].[field1] AS nvarchar(max)) AS before_field1, TRY_CAST([before].[field2] AS bigint) AS before_field2, TRY_CAST([before].[field3] AS nvarchar(max)) AS before_field3, -- after.* TRY_CAST([after].[id] AS bigint) AS after_id, TRY_CAST([after].[field1] AS nvarchar(max)) AS after_field1, TRY_CAST([after].[field2] AS bigint) AS after_field2, TRY_CAST([after].[field3] AS nvarchar(max)) AS after_field3, -- CDC metadata op AS op, TRY_CAST(ts_ms AS bigint) AS ts_ms, -- optional: source lineage flattened TRY_CAST([source].[db] AS nvarchar(max)) AS source_db, TRY_CAST([source].[table] AS nvarchar(max)) AS source_table, TRY_CAST([source].[version] AS nvarchar(max)) AS source_version

INTO [ParquetOutput]

FROM [eventhub-input]

That gives you a clean Parquet file per event where op / ts_ms are always populated and before_* / after_* behave exactly per the Debezium spec.

If preserving the nested envelope as-is inside a single Parquet file is a hard requirement (rather than flat columns), the supported pattern is to land JSON in ADLS Gen2 from ASA and convert to nested Parquet downstream in Databricks / Synapse Spark / Fabric, or use Event Hubs Capture directly for raw retention.

3 answers

Your answer

Answer 1

For Parquet outputs, nested JSON record fields must have a stable, strongly typed schema in the query projection. Avro and Parquet are schema-on-write formats and don’t support variable or loosely typed schemas within a single file. If the projected schema changes row by row, or if a field is treated as a generic record/array without a fixed structure, rows can be rejected or values can be written as null. The recommended approach is to:
- Explicitly project the nested fields as separate scalar columns (for example, after.id, after.field1, source.db, etc.) using dot notation, so that the Parquet schema is uniform and strongly typed.
- Avoid passing the entire record object (after, source, before) as a single record-typed column to Parquet.
Example pattern (adapted from the JSON parsing guidance):
```
   SELECT
       after.id          AS after_id,
       after.field1      AS after_field1,
       after.field2      AS after_field2,
       after.field3      AS after_field3,
       source.version    AS source_version,
       source.db         AS source_db,
       source.table      AS source_table,
       op,
       ts_ms
   INTO ParquetOutput
   FROM input
```
This aligns with the guidance that Avro/Parquet outputs should be treated as strongly typed and that queries should use explicit projections and conversions to ensure a uniform schema.
TRY_CAST to record is not a supported way to produce Parquet output. For Avro and Parquet, the documentation states that these formats should be considered strongly typed and schema-on-write, and that queries should be written with explicit conversions and projections for a uniform schema. There is no documented support for emitting generic record-typed columns directly into Parquet; instead, nested JSON objects should be parsed and flattened into scalar columns before writing to Parquet. In particular, the data type mapping table shows that record is represented as a JSON object for JSON in/out and as an Avro record type for Avro in/out, but it does not describe record as a first-class Parquet output type. For Parquet, the guidance is to avoid variable schemas and use explicit conversions.
Behavior changes that affect nested types in Parquet output aren’t documented in the provided material. What is documented is:
- Avro and Parquet outputs don’t support variable schemas within a single file; schema changes can cause file splitting or row rejection.
- Outputs using these formats should be treated as strongly typed, with explicit conversions and projections.
Given this, the reliable pattern is to:
- Parse nested JSON (record) fields using dot notation or helper functions where needed.
- Project only scalar (or otherwise well-defined) columns into the Parquet output.
- If the full nested structure is required, keep a parallel JSON output where record and array types are natively supported.

References:

Answer 2

GP13 6

Is there a way to output as parquet? we are looking to get the below structure of cdc events in parquet.

Field Description

before State of the record before the change (null for inserts and snapshot rows)

after State of the record after the change: the actual record data

op Type of change: r = snapshot, c = insert, u = update, d = delete

ts_ms Timestamp of the change in the source database in milliseconds

0 comments

Answer 3

Hi @GP13

Thanks for the detailed repro, this is actually expected behavior, not a bug or a recent change in ASA.

What's happening

In Azure Stream Analytics, the record (nested object) type is only supported as an output for JSON and Avro — it is not in the type-mapping matrix for Parquet. Parquet/Avro outputs are treated as strongly-typed / schema-on-write, so nested columns like before, after, and source can't be materialized and come out as null. Scalar fields (op, ts_ms) work fine because they're simple types. That's also why:

TRY_CAST(... AS record) doesn't help — the column stays record-typed.
TRY_CAST(... AS nvarchar(max)) on a nested object doesn't JSON-stringify it either.
JSON output works, because record maps directly to a JSON object.

Refs:

Outputs from Azure Stream Analytics: https://learn.microsoft.com/azure/stream-analytics/stream-analytics-define-outputs#avro-and-parquet-file-splitting-behavior
ASA Data Types – type mappings: https://learn.microsoft.com/stream-analytics-query/data-types-azure-stream-analytics#type-mappings-and-serialization-formats

Two ways to fix it

Option 1 (recommended) — Flatten before writing to Parquet.

Project each nested field as its own scalar column using dot notation:


SELECT

 -- before.*

 TRY_CAST([before].[id] AS bigint) AS before_id,

 TRY_CAST([before].[field1] AS nvarchar(max)) AS before_field1,

 TRY_CAST([before].[field2] AS bigint) AS before_field2,

 -- after.*

 TRY_CAST([after].[id] AS bigint) AS after_id,

 TRY_CAST([after].[field1] AS nvarchar(max)) AS after_field1,

 TRY_CAST([after].[field2] AS bigint) AS after_field2,

 -- source.*

 TRY_CAST([source].[version] AS nvarchar(max)) AS source_version,

 TRY_CAST([source].[db] AS nvarchar(max)) AS source_db,

 TRY_CAST([source].[table] AS nvarchar(max)) AS source_table,

 op,

 TRY_CAST(ts_ms AS bigint) AS ts_ms

INTO [ParquetOutput]

FROM [eventhub-input]

For inserts (op = 'c'), the before_* columns will naturally be null — that's correct Debezium semantics.

Option 2 — Keep the full nested envelope.

If you must preserve before / after / source as nested structs, ASA isn't the right serializer. Write JSON to ADLS Gen2 from ASA, then convert to Parquet downstream in Databricks / Synapse Spark / Fabric where nested Parquet structs are fully supported. Or, for pure capture, use Event Hubs Capture directly.

Quick answers to your 3 questions

Serialize nested records to Parquet in ASA? Not as nested structs. Flatten to scalars (Option 1) or convert downstream (Option 2).
Is TRY_CAST AS record supported for Parquet? No. record isn't in the Parquet output mapping, so it lands as null. Documented limitation.
Has ASA changed? No, record → Parquet was never in the supported matrix. The earlier job likely used JSON/Avro or a simpler payload that happened to flatten cleanly.

Please try Option 1 and let us know how the flattened columns look in the Parquet output. Happy to help further if you want to preserve the full nested structure.