Always On AG secondary replicas showing DISCONNECTED despite endpoint and network checks passing

Question

Always On AG secondary replicas showing DISCONNECTED despite endpoint and network checks passing

Gowtham K 0

We are facing an issue with a SQL Server Always On Availability Group where the secondary replica remains DISCONNECTED. We have verified that the HADR endpoint is STARTED, uses TCP port 5022, and AES encryption. The SQL Server service accounts are running, and the required CONNECT permission is granted on the endpoint. Port 5022 connectivity was tested successfully in both directions using Test-NetConnection. The listener is reachable on port 1433 for one IP, while connectivity to another listener IP failed. DNS resolution is working, and port 5022 is listening under sqlservr.exe. No HADR-related certificates were found. What additional checks or resolution steps should we perform? User's image

Erland Sommarskog 137.4K Reputation points MVP Volunteer Moderator

2026-09-06T08:34:29.5066667+00:00

In addition to all the diagnostics suggested by the AI, have you checked the SQL Server errorlog on the three instances of messages that might be relevant?

For further assistance, it helps if you post the result of the diagnostics that the AI suggested. (I am not sure that all of them are on target, but some could be.)
Erland Sommarskog 137.4K Reputation points MVP Volunteer Moderator

2026-09-06T18:45:18.97+00:00

By the way, did you run ALTER AVAILABILITY JOIN on the secondaries? My apologies if this comes across as a stupid question, but I have a recollection of having made that mistake myself at some occasion.
Gowtham K 0 Reputation points

2026-09-08T09:55:45.49+00:00

Thank you for the suggestions.

Yes, we have reviewed the SQL Server error logs across the relevant instances and found repeated Database Mirroring/HADR endpoint connection handshake failures, indicating that the login TASQL\tasqldb was being denied CONNECT permission on the endpoint. We are currently validating the endpoint permissions and related Always On configuration on each replica.

We are also reviewing the diagnostics suggested earlier and will share the results once the validation is completed.

Regarding ALTER AVAILABILITY GROUP … JOIN, we will verify that the secondary replicas were properly joined to the availability group and confirm the configuration on each secondary.

We appreciate your guidance and will update the thread with our findings.
Erland Sommarskog 137.4K Reputation points MVP Volunteer Moderator

2026-09-08T12:13:55.97+00:00
indicating that the login TASQL\tasqldb was being denied CONNECT permission on the endpoint.

That sounds like a lead. :--)

You can run this:

SELECT * FROM sys.fn_my_permissions('hadr_endpoint', 'ENDPOINT') go EXECUTE AS LOGIN = 'hadr_login' go SELECT * FROM sys.fn_my_permissions('hadr_endpoint', 'ENDPOINT') go REVERT

To verify. Replace hadr_endpoint with the actual name of the endpoint. The first SELECT is only to verify that the name is correct. It is the second result set that is relevant. It must list CONNECT.

1 answer

Your answer

Erland Sommarskog 137.4K Reputation points MVP Volunteer Moderator

2026-09-06T08:34:29.5066667+00:00

In addition to all the diagnostics suggested by the AI, have you checked the SQL Server errorlog on the three instances of messages that might be relevant?

For further assistance, it helps if you post the result of the diagnostics that the AI suggested. (I am not sure that all of them are on target, but some could be.)
Erland Sommarskog 137.4K Reputation points MVP Volunteer Moderator

2026-09-06T18:45:18.97+00:00

By the way, did you run ALTER AVAILABILITY JOIN on the secondaries? My apologies if this comes across as a stupid question, but I have a recollection of having made that mistake myself at some occasion.
Gowtham K 0 Reputation points

2026-09-08T09:55:45.49+00:00

Thank you for the suggestions.

Yes, we have reviewed the SQL Server error logs across the relevant instances and found repeated Database Mirroring/HADR endpoint connection handshake failures, indicating that the login TASQL\tasqldb was being denied CONNECT permission on the endpoint. We are currently validating the endpoint permissions and related Always On configuration on each replica.

We are also reviewing the diagnostics suggested earlier and will share the results once the validation is completed.

Regarding ALTER AVAILABILITY GROUP … JOIN, we will verify that the secondary replicas were properly joined to the availability group and confirm the configuration on each secondary.

We appreciate your guidance and will update the thread with our findings.
Erland Sommarskog 137.4K Reputation points MVP Volunteer Moderator

2026-09-08T12:13:55.97+00:00

indicating that the login TASQL\tasqldb was being denied CONNECT permission on the endpoint.

That sounds like a lead. :--)

You can run this:

SELECT * FROM sys.fn_my_permissions('hadr_endpoint', 'ENDPOINT') go EXECUTE AS LOGIN = 'hadr_login' go SELECT * FROM sys.fn_my_permissions('hadr_endpoint', 'ENDPOINT') go REVERT

To verify. Replace hadr_endpoint with the actual name of the endpoint. The first SELECT is only to verify that the name is correct. It is the second result set that is relevant. It must list CONNECT.

Answer 1

Additional checks should focus on replica-to-replica endpoint configuration consistency and connection error details.

Check the last HADR connection error on both primary and secondary replicas.

SELECT r.replica_server_name,
       r.endpoint_url,
       rs.connected_state_desc,
       rs.last_connect_error_description,
       rs.last_connect_error_number,
       rs.last_connect_error_timestamp
FROM sys.dm_hadr_availability_replica_states AS rs
     INNER JOIN sys.availability_replicas AS r
         ON rs.replica_id = r.replica_id
WHERE rs.is_local = 1;

This identifies the most recent connection failure. If the description shows DNS lookup failure or an incorrect host, the replica endpoint_url is likely wrong.

Verify that the AG endpoint_url matches the actual database mirroring endpoint port on each replica.

Run on the primary and each disconnected secondary:

SELECT endpoint_url
FROM sys.availability_replicas;

Then verify the local endpoint definition:

SELECT tep.name AS EndPointName,
       sp.name AS CreatedBy,
       tep.type_desc,
       tep.state_desc,
       tep.port
FROM sys.tcp_endpoints AS tep
     INNER JOIN sys.server_principals AS sp
         ON tep.principal_id = sp.principal_id
WHERE tep.type = 4;

The port in endpoint_url must match the port defined for the endpoint on that replica. A mismatch can leave replicas in DISCONNECTED even when Test-NetConnection to 5022 succeeds.

Recheck endpoint configuration on both replicas and compare encryption settings.

$server_name = "server_instance"
sqlcmd -S $server_name -E -Q "SELECT name, state_desc, encryption_algorithm_desc, protocol_desc, type_desc FROM sys.database_mirroring_endpoints"

Run it on both servers and confirm the encryption and protocol values match.

Confirm there is no port conflict despite sqlservr.exe listening on 5022.

$server_name = "server_instance"
sqlcmd -S $server_name -E -Q "SELECT type_desc, port FROM sys.tcp_endpoints WHERE type_desc = 'DATABASE_MIRRORING'; "

$port = "5022"
Get-NetTCPConnection -LocalPort $port
Get-Process -Id (Get-NetTCPConnection -LocalPort $port).OwningProcess | Select-Object Name, ProductVersion, Path, Id

This confirms the mirroring endpoint is bound to the expected port and helps detect another process or instance using that port.

Validate connectivity by both server name and IP address for each replica endpoint.

$server_name = "your_server_name"
$IP_address = "your_ip_address"
$port_number = "your_port_number"

Test-NetConnection -ComputerName $server_name -Port $port_number
Test-NetConnection -ComputerName $IP_address -Port $port_number

Expected result is TcpTestSucceeded : True.

If IP works but server name fails, the issue is likely DNS or name resolution.
If server name works but IP fails, another endpoint or instance may be listening on that port.
If either fails intermittently, check firewall and, for Azure VMs, NSG rules.

If the endpoint exists but may not be responding, restart the endpoint carefully.

sqlcmd -S "server_instance" -E -Q "ALTER ENDPOINT hadr_endpoint STATE = stopped"
sqlcmd -S "server_instance" -E -Q "ALTER ENDPOINT hadr_endpoint STATE = started"

Warning: setting STATE = stopped temporarily interrupts Always On traffic flow.

If SQL Server is running under an account that is not a domain user, test connectivity while signed in as that service account.

Open Windows PowerShell with Run as Different User.
Sign in with the SQL Server service account.
Run whoami to confirm the account.
Then test the remote node on port 5022 with Test-NetConnection.

Given that one listener IP is reachable and another is not, also treat that as a sign to compare endpoint URLs, port bindings, and network path behavior per replica rather than assuming the AG endpoint is fully healthy because 5022 responds.

References: