A unified data governance solution that helps manage, protect, and discover data across your organization
Purview Unified Catalog Data Quality marks all String format rules as failed for NVARCHAR phone column
Hello,
I am experiencing an issue with Microsoft Purview Unified Catalog Data Quality when validating a PhoneNumber column (Thai PhoneNumber) from an Azure Synapse Dedicated SQL Pool table.
Environment:
- Source: Azure Synapse Dedicated SQL Pool
- Column: PhoneNumber
- Source data type: NVARCHAR(50)
- Purview profile data type: String
- Total rows: 308,200
Database validation results:
- SQL NULL rows: 0
- Rows with a length of 10 characters: 308,200
- Rows starting with 06, 08, or 09: 308,200
- Numeric-only rows: 308,200
A valid sample value is: 0610017000
The database returns:
- LEN(PhoneNumber) = 10
- DATALENGTH(PhoneNumber) = 20, which is expected for NVARCHAR
- First character = 0
- Prefix = 06
However, Microsoft Purview Data Quality marks all 308,246 rows as Failed and returns a score of 0 for every validation method tested.
Tests performed:
- String format match using Regular expression:
^0[689][0-9]{8}$ - Regular expression without anchors:
0[689][0-9]{8} - Regular expression without anchors:
0[0-9]{9}/.*[0-9]{9}.* - Regular expression without anchors:
.*[0-9]{10}.* - Regular expression without anchors: [A-Za-z0-9]{10,}
- Like pattern: 0%
- Like pattern matching any string: ________________________
- Custom expression: length(trim(PhoneNumber)) == 10
- Custom SQL: LENGTH(TRIM(PhoneNumber)) = 10
The Empty/blank fields rule returns 100%, confirming that Purview can read values from the column. However, all format and custom validation rules return 0%.
Actions already performed:
- Verified the source data directly in Synapse
- Confirmed that Purview profiles PhoneNumber as String
- Saved each updated rule and ran a new Data Quality scan
- Tested Regular expression, Like pattern, Custom, and Custom SQL rules
Questions:
- Is there a known issue with String format match or Custom rules for NVARCHAR columns from Azure Synapse Dedicated SQL Pool?
- Does the Purview Data Quality engine process NVARCHAR values differently from the source value?
- What is the recommended approach for validating phone number formats (PhoneNumber Format Validation) in Microsoft Purview Data Quality? In this scenario, how should we validate Thai mobile phone numbers (e.g. 06XXXXXXXX, 08XXXXXXXX, 09XXXXXXXX) stored in a column profiled as String and sourced from an NVARCHAR column in Azure Synapse Dedicated SQL Pool? Are there any Microsoft-recommended validation rules, supported patterns, best practices, or reference implementations for validating phone numbers stored as String/NVARCHAR values?
- Are there any known limitations or issues in Microsoft Purview Data Quality when processing numeric-looking values stored as String/NVARCHAR columns?
For example, can leading zeros, Unicode strings (NVARCHAR), type conversions, or internal profiling behavior affect String format matching, Regular Expression rules, Like pattern rules, or Custom validation rules and cause otherwise valid values to fail validation?
- If all String format rules return 0% while Empty/Blank validation succeeds, is there a recommended troubleshooting process to determine whether this is a configuration issue, a known product limitation, or a product defect?