HPC Pack 2019 cannot connect to headnode immediately after first install

Question

HPC Pack 2019 cannot connect to headnode immediately after first install

Ivan Voutchkov 1

I have just installed HPC Pack 2019 Update 1 on a brand new Windows Server 2022. Immediately after installation finishes and I attempt to start the HPC Cluster Manager I get this. Please help!

The connection to the management service failed. detail error: Microsoft.Hpc.RetryCountExhaustException: Retry Count of RetryManager is exhausted. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.HttpClientExtension.<>c__DisplayClass5_0.<

Ivan Voutchkov 1 Reputation point

2022-06-27T14:30:21.12+00:00

Log Name: Microsoft HPC Pack
Source: MonitoringServer
Date: 27/06/2022 15:20:43
Event ID: 1
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: SPITFIREHEAD.cedgcompute.local
Description:
Failed to initialize collector. Retrying in 60 seconds. System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.Monitoring.ManagementRestClient.<>c__DisplayClass19_0.<<GetApiCallAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Hpc.Monitoring.MonitoringRetryModule.<RetryTemporaryFailAsync>d__81.MoveNext() \--- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.Hpc.Monitoring.MonitoringRetryModule.<RetryTemporaryFailAsync>d__81.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
Ivan Voutchkov 1 Reputation point

2022-06-27T14:30:40.227+00:00

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.Monitoring.HpcRestMonitoringStore.<GetMetricsAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.Monitoring.HpcRestMonitoringStore.GetMetrics(Nullable`1 target)
at Microsoft.Hpc.Monitoring.MetricCollector.Initialize()

5 answers

Your answer

Ivan Voutchkov 1 Reputation point

2022-06-27T14:30:21.12+00:00

Log Name: Microsoft HPC Pack
Source: MonitoringServer
Date: 27/06/2022 15:20:43
Event ID: 1
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: SPITFIREHEAD.cedgcompute.local
Description:
Failed to initialize collector. Retrying in 60 seconds. System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.Monitoring.ManagementRestClient.<>c__DisplayClass19_0.<<GetApiCallAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Hpc.Monitoring.MonitoringRetryModule.<RetryTemporaryFailAsync>d__81.MoveNext() \--- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.Hpc.Monitoring.MonitoringRetryModule.<RetryTemporaryFailAsync>d__81.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
Ivan Voutchkov 1 Reputation point

2022-06-27T14:30:40.227+00:00

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.Monitoring.HpcRestMonitoringStore.<GetMetricsAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.Monitoring.HpcRestMonitoringStore.GetMetrics(Nullable`1 target)
at Microsoft.Hpc.Monitoring.MetricCollector.Initialize()

Answer 1

Yutong Sun 261 Microsoft Employee

For running HPC Pack 2019 on Windows Server 2022, please also check if the required .Net Framework 4.8 cumulative update August 9, 2022-KB5015733 or later is applied to all cluster nodes and clients that are on Windows Server 2022. If the updates are not available, you may need to disable TLS 1.3 on Windows Server 2022 before installing the head node as a workaround. This is a known issue in th release note.

0 comments

Answer 2

Ken Kam Hung, Lin 251

I have the same issue too, after installing HPC Pack 2019 Update 2 on Win 2022.

0 comments

Answer 3

Gurvinder Bindra 0

New deployment, SAME issue! Will appreciate any help we can get! Naming service IS running!

Nate Anderson 5 Reputation points

2023-09-12T14:27:31.5533333+00:00

I managed to resolve the issue, though I couldn't pinpoint the exact cause. While analyzing the log files (detailed steps on extracting logs can be found here), I identified a 404 error stemming from a web request to the Fabric service. This is confusing since I wanted a single head node which doesn't use Service Fabric.

To address this, I set up two more VMs, giving me a total of three head nodes. I then proceeded with the HPC Pack Installer and chose the "Microsoft Service Fabric cluster" option. This completed successfully and the Cluster Manager initialized without any hitches.

For the final setup, I uninstalled HPC Pack 2019 from the desired head node, reinstalled it, and this time chose the "Single Head Node mode". The Cluster Manager launched smoothly. I no longer needed the extra VMs, so I deleted them. Everything's functioning as expected now. Hope this solution assists others facing a similar issue!

Answer 4

Nate Anderson 5

@Ivan Voutchkov Did you ever get this working? I'm running into the same problem.

Regards,

Nate

0 comments

Answer 5

Yutong Sun 261 Microsoft Employee

Hi @Ivan Voutchkov ,

The call stack indicates there was a connection issue with the HpcNaming service to resolve the service instance. Could you check if the HpcNaming service is in running state on the head nodes and retry it again?

Regards,
Yutong Sun

0 comments