What is the logic behind detecting the PII for country spefic ?

Question

What is the logic behind detecting the PII for country spefic ?

Mohsin Khan 65

I am implementing DLP using Azure AI Langugae PII detection service, with that there are several categories and country.

For Example with Philipness below ID (Unified Multi-Purpose ID) format is valid and will be detected as PII.

1234-5678901-2

Though each part is explained below

Structure:

First 4 digits: Agency prefix (e.g., SSSS, GSIS)

Next 7 digits: Unique serial number

Last digit: Check digit (computed via Luhn or custom checksum algorithm)

But if I use the format based approach for Portugual(Citizen Card Number) is not detecting utill i need to calculate the check sum based on there logic.

Example :

12345678 4 AB1 -Will not work

42634925 3 ZY8 will work.

how the service detect the PII does it apply the format/regex based or checksum calculation ?

Locked Question. You can vote on whether it's helpful, but you can't add comments or replies or follow the question.

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Answer accepted by question author

Saideep Anchuri 9,545 Moderator

Hi there Mohsin Khan

Some of these categories do follow specific checksum or validation formats.

Category	Uses Format/Checksum?

ABA routing number	Yes
IBAN (International Banking Account Number)	Yes
Argentina National Identity (DNI) Number	Yes
Austria identity card	No
Austria tax identification number	Yes
Austria VAT number	Yes
Australia bank account number	Yes
Australian business number	Yes
Australia Company Number	Yes
Australia driver's license	No
Australia medical account number	No
Australia passport number	No
Australia tax file number	Yes
Belgium national number	Yes
Belgium VAT number	Yes
Brazil legal entity number (CNPJ)	Yes
Brazil CPF number	Yes
Brazil National ID Card (RG)	No
Canada bank account number	Yes
Canada driver's license number	No
Canada health service number	No
Canada passport number	No
Canada Personal Health Identification Number (PHIN)	No
Canada social insurance number	Yes
Chile identity card number	Yes
China Resident Identity Card (PRC) number	Yes
EU debit card number	Yes
EU driver's license number	No
EU GPS coordinates	No
EU national identification number	Varies by country
EU passport number	No
EU Social Security Number (SSN) or equivalent ID	Varies by country
EU Tax Identification Number (TIN)	Varies by country
France driver's license number	No
France health insurance number	No
France national ID card (CNI)	Yes
France passport number	No
France Social Security Number (INSEE)	Yes
France tax identification number (Numéro SPI)	Yes
France VAT number	Yes
German Driver's License Number	No
Germany Identity Card Number	Yes
Germany passport number	No
Germany Tax Identification Number	Yes
Germany VAT Number	Yes
Hong Kong Identity Card (HKID) Number	Yes
Hungary Personal Identification Number	Yes
Hungary Tax Identification Number	Yes
Hungary VAT Number	Yes
India Permanent Account Number (PAN)	Yes
Indonesia Identity Card (KTP) Number	Yes
Ireland Personal Public Service (PPS) Number	Yes
Israel National ID	Yes
Israel Bank Account Number	Yes
Italy Driver's License ID	No
Italy Fiscal Code	Yes
Italy VAT Number	Yes
Japan Bank Account Number	Yes
Japan Driver's License Number	No
Japan "My Number" (personal)	Yes
Japan "My Number" (Corporate)	Yes
Japan Resident Registration Number	Yes
Japan Residence Card Number	Yes
Japan Social Insurance Number (SIN)	Yes
Japan Passport Number	No
Luxembourg National Identification Number (Natural persons)	Yes
Luxembourg National Identification Number (Non-natural persons)	Yes
Malta Identity Card Number	Yes
Malta Tax Identification Number	Yes
New Zealand Bank Account Number	Yes
New Zealand Driver's License Number	No
New Zealand Inland Revenue Number	Yes
New Zealand Ministry of Health Number	No
New Zealand Social Welfare Number	No
Philippines Unified Multi-Purpose ID Number	Yes
Portugal Citizen Card Number	Yes
Portugal Tax Identification Number	Yes
Singapore National Registration ID Card (NRIC) Number	Yes
South Africa Identification Number	Yes
South Korea Resident Registration Number	Yes
Spain DNI	Yes
Spain Social Security Number (SSN)	Yes
Spain Tax Identification Number	Yes
Swiss Social Security Number (AHV)	Yes
Taiwan National ID	Yes
Taiwan Resident Certificate (ARC/TARC)	Yes
Taiwan Passport Number	No

Thank You.

Saideep Anchuri 9,545 Reputation points Moderator

2025-05-09T03:36:37.31+00:00

Hi there Mohsin Khan

Could please accept my answer that which you have upvoted my comment so i have converted my comment to answer which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thank you.

1 additional answer

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Saideep Anchuri 9,545 Reputation points Moderator

2025-05-09T03:36:37.31+00:00

Hi there Mohsin Khan

Could please accept my answer that which you have upvoted my comment so i have converted my comment to answer which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thank you.

Answer 1

Azar 31,725 MVP Volunteer Moderator

Hi there Mohsin Khan

thanks for using QandA platform

Regex & format matching is the first layer this catches most PII patterns that follow known structures.
Checksum validation is applied only for countries and PII types where the structure alone isn't unique or reliable enough (like Portugal Citizen Card Numbers etc.).
If the checksum fails or is missing, the PII entity may be downgraded in confidence.

In your Portugal example, the PII won't be flagged unless it passes the Portuguese ID checksum validation. That’s why 42634925 3 ZY8 works and the other doesn’t.

So yes it’s not just regex, but also country-specific logic, including checksums or control digit verification where applicable

If this helps kindly accept the answer thanks .

Mohsin Khan 65 Reputation points

2025-05-08T03:40:57.94+00:00

Hey Azar, Thank you for the response.

I have validated that for country like south korea/Philipness it is followed the regex pattern match and few of other countries it is follow check sum validation. Just wondering country specific PII detection sometimes following regex and sometime checksum validation. This leads to problem during quality assurance and lose confidence with testing data.
Saideep Anchuri 9,545 Reputation points Moderator

2025-05-08T04:29:05.3133333+00:00

Hi there Mohsin Khan

Some countries may have identification numbers that include a checksum digit as part of their format, which requires a checksum validation to ensure the accuracy of the number. In contrast, other countries might rely solely on regex patterns to match the structure of their identification numbers without requiring a checksum.

This inconsistency can make testing unpredictable, especially when different validation methods apply to different regions. One way to improve confidence in testing data is to implement hybrid validation, where both regex and checksum checks are applied where possible. Additionally, maintaining a country-specific validation framework that adapts based on regional requirements could help standardize the process.

Kindly refer below link: sensitive-information-type-regular-expression-validators

Thank you.
Mohsin Khan 65 Reputation points

2025-05-08T05:50:04.0733333+00:00

Hey @saideep anchuri

Thank you for the response and sharing.

Initially I hold the same impression as you suggested when I was checking for Philipness country as it is accepting any number if follow the format (1234-5678901-2 valid and PII detected).

But on the official web site Unified Multi-Purpose ID reserve the structure as mentioned below.

Structure:

First 4 digits: Agency prefix (e.g., SSSS, GSIS)

Next 7 digits: Unique serial number

Last digit: Check digit (computed via Luhn or custom checksum algorithm)
Saideep Anchuri 9,545 Reputation points Moderator

2025-05-08T06:14:30.42+00:00
Hi there Mohsin Khan

The logic behind detecting Personally Identifiable Information (PII) for country-specific identification numbers, such as the Philippines Unified Multi-Purpose ID Number, is based on predefined formats and structures that are unique to each country's identification systems.

For the Philippines Unified Multi-Purpose ID Number, the structure consists of:

First 4 digits: Agency prefix (e.g., SSSS, GSIS)

Next 7 digits: Unique serial number

Last digit: Check digit (computed via Luhn or custom checksum algorithm)

This structured format allows the system to validate the number and ensure it adheres to the expected pattern, which is essential for accurately detecting and categorizing PII. The detection logic relies on recognizing these patterns and validating them against the defined criteria.

Kindly refer below link: government-and-country-region-specific-identification

Thank You.
Mohsin Khan 65 Reputation points

2025-05-08T07:22:16.9666667+00:00

Hey @saideep anchuri

Thank you for sharing,

Let me revise my question,

For an example : Philippines Unified Multi-Purpose ID Number, it is following the format pattern and detect as PII instead validating check sum and calculate though the official Id include checksum number.

Where as For country Portugal/NewZeland/Italy it is validating again checksum not with format validation.
Saideep Anchuri 9,545 Reputation points Moderator

2025-05-08T08:19:40.42+00:00

Hi there Mohsin Khan

The Philippines Unified Multi-Purpose ID Number (UMID) follows a format-based pattern for detection as Personally Identifiable Information (PII) rather than validating its checksum, even though the official ID includes a checksum. This means that systems primarily recognize the structure of the ID rather than verifying its mathematical integrity.

In contrast, countries like Portugal, New Zealand, and Italy enforce checksum validation rather than relying solely on format-based detection. A checksum is a mathematical verification method that ensures the integrity of an ID number by following predefined rules, helping to detect errors or invalid entries.

Kindly refer below link: purpose-identification-number

Thank You.
Mohsin Khan 65 Reputation points

2025-05-08T08:52:08.3566667+00:00

Hey @Saideep Anchuri,

Thank you for clarification on this. As this is the system behavior would be really helpful to share list of categories which validate based on format/checksum methods.

So, we will craft our system accordingly.

Regards,

Mohsin
Saideep Anchuri 9,545 Reputation points Moderator

2025-05-08T08:55:59.0066667+00:00
Hi there Mohsin Khan

Categories Validated by Format-Based Detection

These rely on predefined patterns, such as length, structure, and character composition:

National Identification Numbers (e.g., Philippines Unified Multi-Purpose ID)

Phone Numbers (e.g., country-specific formats)

Postal Codes (e.g., ZIP codes)

Credit Card Numbers (basic format validation without Luhn checksum)

Taxpayer Identification Numbers (some countries use format-based validation)

Categories Validated by Checksum Methods

These use mathematical algorithms to verify integrity:

Credit Card Numbers (Luhn algorithm)

Bank Account Numbers (IBAN checksum)

Social Security Numbers (some countries apply checksum validation)

Vehicle Identification Numbers (VINs) (checksum digit validation)

Barcodes & QR Codes (error detection via checksum)

Kindly refer below link: https://binaryte.com/blog/post/checksum-algorithm-a-guide-to-data-verification.md/

Thank you
Mohsin Khan 65 Reputation points

2025-05-08T09:53:48.4366667+00:00

Please share the list of categories which follow format/Checksum methods ? As these are not limited with above categories.
Saideep Anchuri 9,545 Reputation points Moderator

2025-05-08T11:26:17.02+00:00
Hi there Mohsin Khan

Here are some categories that follow format and checksum methods:

India Unique Identification (Aadhaar) Number

Format: 12 digits containing optional spaces or dashes.

Checksum: Yes.

U.S. Social Security Number (SSN)

Format: Nine digits, which may be formatted or unformatted.

Checksum: No.

U.S. Bank Account Number

Format: Specific to U.S. banking standards.

Checksum: Yes.

U.K. National Insurance Number (NINO)

Format: Specific to U.K. standards.

Checksum: Yes.

U.S. Drug Enforcement Agency (DEA) Number

Format: Specific to U.S. regulations.

Checksum: Yes.

Kindly refer below link: regex-classifications

government-and-country-region-specific-identification

Thank you.
Mohsin Khan 65 Reputation points

2025-05-08T16:47:07.48+00:00

Hey @Saideep Anchuri

Much appreciated for the sharing, following with that could be please yes/no for below categories as well ?

ABA routing number

Category: International Banking Account Number (IBAN)

Argentina National Identity (DNI) Number

Austria identity card

Austria tax identification number

Austria Value Added Tax (VAT) number

Australia bank account number

Australian business number

Australia Company Number

Australia driver's license

Australia medical account number

Australia passport number

Australia tax file number

Belgium national number

Belgium Value Added Tax (VAT) number

Brazil legal entity number (CNPJ)

Brazil CPF number

Brazil National ID Card (RG)

Canada bank account number

Canada driver's license number

Canada health service number

Canada passport number

Canada Personal Health Identification Number (PHIN)

Canada social insurance number

Chile identity card number

China Resident Identity Card (PRC) number

EU debit card number

EU driver's license number

EU GPS coordinates

EU national identification number

EU passport number

EU Social Security Number (SSN) or equivalent ID

EU Tax Identification Number (TIN)

France driver's license number

France health insurance number

France national ID card (CNI)

France passport number

France Social Security Number (INSEE)

France tax identification number (Numéro SPI)

France Value Added Tax (VAT) number

German Driver's License Number

Germany Identity Card Number

Germany passport number

Germany Tax Identification Number

Germany Value Added Tax Number

Hong Kong Identity Card (HKID) Number

Hungary Personal Identification Number

Hungary Tax identification Number

Hungary Value Added Tax Number

India Permanent Account Number (PAN)

Indonesia Identity Card (KTP) Number

Ireland Personal Public Service (PPS) Number

Israel National ID

Israel Bank Account Number

Italy Driver's License ID

Italy Fiscal Code

Italy Value Added Tax Number

Japan Bank Account Number

Japan Driver's License Number

Japan "My Number" (personal)

Japan "My Number" (Corporate)

Japan Resident Registration Number

Japan Residence Card Number

Japan Social Insurance Number (SIN)

Japan Passport Number

Luxembourg National Identification Number (Natural persons)

Luxembourg National Identification Number (Non-natural persons)

Malta Identity Card Number

Malta Tax Identification Number

New Zealand Bank Account Number

New Zealand Driver's License Number

New Zealand Inland Revenue Number

New Zealand Ministry of Health Number

New Zealand Social Welfare Number

Philippines Unified Multi-Purpose ID Number

Portugal Citizen Card Number

Portugal Tax Identification Number

Singapore National Registration ID Card (NRIC) Number

South Africa Identification Number

South Korea Resident Registration Number

Spain DNI

Spain Social Security Number (SSN)

Spain Tax Identification Number

Swiss Social Security Number AHV

Taiwan National ID

Taiwan Resident Certificate (ARC/TARC)

Taiwan Passport Number

Thanks
Mohsin Khan 65 Reputation points

2025-05-09T03:29:15.1366667+00:00

That's really great, Thank you for the sharing.