Azure CycleCloud
A Microsoft tool for creating, managing, operating, and optimizing high-performance computing (HPC) and big compute clusters in Azure.
Cyclecloud - Persistent connection Slurm accounting - connection refuse to my scheduler
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 48%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJG%3C/text%3E%3C/svg%3E)
Jason Gumarang
25
Reputation points
hi Support,
this is the error i am getting from my newly scheduler. I was able to deploy without any issue before to my new cluster but since the upgrade to cyclecloud 8, it seems newly created cluster has this issue whenever i enable the accounting.
I am not sure sure if there is a need to update in the jetpack or something as currently i am always getting this connection refuse.
Actions: Already tried to add port 6819 and 6817 in my network security group but no luck at all.
[2025-04-15T06:42:14.863] slurmscriptd: debug: _slurmscriptd_mainloop: started
[2025-04-15T06:42:14.863] debug: slurmctld: slurmscriptd fork()'d and initialized.
[2025-04-15T06:42:14.863] debug: _slurmctld_listener_thread: started listening to slurmscriptd
[2025-04-15T06:42:14.863] slurmctld version 24.05.4 started on cluster jason-test
[2025-04-15T06:42:14.863] cred/munge: init: Munge credential signature plugin loaded
[2025-04-15T06:42:14.863] select/cons_tres: init: select/cons_tres loaded
[2025-04-15T06:42:14.864] select/linear: init: Linear node selection plugin loaded with argument 20
[2025-04-15T06:42:14.864] debug: MPI: Loading all types
[2025-04-15T06:42:14.865] debug: mpi/pmix_v4: init: PMIx plugin loaded
[2025-04-15T06:42:14.865] debug: mpi/pmix_v4: init: PMIx plugin loaded
[2025-04-15T06:42:14.866] debug: _plugrack_foreach: serializer plugin type:serializer/json path:/usr/lib64/slurm/serializer_json.so
[2025-04-15T06:42:14.866] debug: _plugrack_foreach: serializer plugin type:serializer/url-encoded path:/usr/lib64/slurm/serializer_url_encoded.so
[2025-04-15T06:42:14.866] accounting_storage/slurmdbd: init: Accounting storage SLURMDBD plugin loaded
[2025-04-15T06:42:14.867] error: _open_persist_conn: failed to open persistent connection to host:jason-test-scheduler:6819: Connection refused
[2025-04-15T06:42:14.867] error: Sending PersistInit msg: Connection refused
[2025-04-15T06:42:14.867] accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Registering slurmctld at port 6817 with slurmdbd
[2025-04-15T06:42:14.867] error: Sending PersistInit msg: Connection refused
[2025-04-15T06:42:14.867] debug: Association database appears down, reading from state file.
[2025-04-15T06:42:14.867] debug: create_mmap_buf: Failed to open file /sched/jason-test/spool/slurmctld/last_tres, No such file or directory
[2025-04-15T06:42:14.867] debug: create_mmap_buf: Failed to open file /sched/jason-test/spool/slurmctld/assoc_mgr_state, No such file or directory
[2025-04-15T06:42:14.867] fatal: You are running with a database but for some reason we have no TRES from it. This should only happen if the database is down and you don't have any state files.
[2025-04-15T06:42:14.868] slurmscriptd: debug: _slurmscriptd_mainloop: finished
Azure CycleCloud
-
Arko 4,185 Reputation points Moderator2025-04-16T07:45:51.12+00:00 Hello Jason Gumarang,
Your error message
error: _open_persist_conn: failed to open persistent connection to host:jason-test-scheduler:6819: Connection refusedindicates that the Slurm controller (
slurmctld) is unable to establish a persistent connection to the Slurm Database Daemon (slurmdbd) on port 6819.So, first of all ensure that port 6819 is open on both the VM's firewall and the Azure NSG associated with the scheduler node.
Also, could you kindly check below pointers and let me know the outcomes?
do a
sudo firewall-cmd --list-portsand if port 6819 is not listed, add itsudo firewall-cmd --add-port=6819/tcp --permanent sudo firewall-cmd --reloadVerify
slurmdbdis Running-sudo systemctl status slurmdbdIf it's not running, start the service, restart it-
sudo systemctl start slurmdbdrun this-
sudo journalctl -u slurmdbd -xeand share the logsIf all these looks good, then validate Munge Key under /etc/munge/munge.key it should be same across all nodes. Compare it using
md5sum /etc/munge/munge.keyIf the keys differ, synchronize them across nodes and restart the Munge service
sudo systemctl restart munge
sudo systemctl restart slurmdbd
sudo systemctl restart slurmctld
Kindly check all the above pointers and let me know if the issue persists.
You can also refer to MS document- https://learn.microsoft.com/en-us/azure/cyclecloud/?view=cyclecloud-8
-
Jason Gumarang 25 Reputation points
2025-04-16T11:02:50.4933333+00:00 Hi Support,
Thanks for the feedback.
When i brought up a a new cluster, basically the cyclecloud will create a new vm and its very tricky to add the port as the azure will assign a default security group to my new scheduler. It fails to install the jetpack and I dont know how i can attached a the network security groups. Since the jetpack installation failed, my slurm installation also failed.
Both the slurmDB and the scheduler are on the same vlan and i assume that there will be no needed acl to be configured, or i may be wrong.
when i run firewall-cmd --list-ports, the system says FirewallD is not running.
-
Arko 4,185 Reputation points Moderator
2025-04-16T15:29:39.0933333+00:00 When a new scheduler VM is created by CycleCloud, Jetpack runs as a systemd unit (
jetpack.service) to install all necessary packages and configuration. If that fails, Slurm setup will also fail, includingslurmdbd, user creation, and Munge configuration.Could you please share the outputs of
sudo journalctl -u jetpack sudo tail -n 100 /opt/cycle/jetpack/logs/jetpackd.logIt’s okay that firewalld is not running, many Azure VM images default to using iptables or have no OS-level firewall enabled. Since the scheduler VM is auto-created, Azure assigns it a default NSG. If you haven’t explicitly allowed ports like
6817and6819in this NSG, communication betweenslurmctldandslurmdbdwill fail even if they are on the same subnet/VLAN. To fix this-Go to the Azure Portal. Locate the Network Interface attached to the scheduler VM. View its Effective NSG and update inbound security rules to allow:
TCP Port 6817 (slurmctld)
TCP Port 6819 (slurmdbd)
TCP Port 22 (SSH, if needed)
Even if the scheduler and database nodes are on the same subnet, Azure still applies NSG rules at the NIC and subnet level so intra-subnet traffic can still be blocked unless explicitly allowed in the NSG.
-
Arko 4,185 Reputation points Moderator
2025-04-17T09:25:24.8266667+00:00 Hello Jason, following up on this, hope I was able to clear your query here.
-
Jason Gumarang 25 Reputation points
2025-04-17T09:58:37.26+00:00 Hi Support,
please see command output below, also, i did setup a incoming acl to allow port 6817 and 6819 in the NSG. And when i restarted jetpack installation, the issue is still connection refuse. Also, i assume there will be no further configuration needed in my Slurm DB right?
===================================
[root@jason-test-scheduler ~]# sudo journalctl -u jetpack
-- Logs begin at Thu 2025-04-17 08:16:34 UTC, end at Thu 2025-04-17 09:54:26 UTC. --
-- No entries --
[root@jason-test-scheduler ~]# sudo tail -n 100 /opt/cycle/jetpack/logs/jetpackd.log
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:44:21,411 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:45:06,788 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:45:16,789 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:46:02,159 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:46:12,160 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:46:57,560 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:47:07,561 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:47:53,097 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:48:03,098 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:48:48,507 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:48:58,507 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:49:43,853 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:49:53,854 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:50:39,246 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:50:49,247 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:51:34,655 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:51:44,656 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:52:29,973 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:52:39,974 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:53:25,336 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:53:35,337 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:54:20,698 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
2025-04-17 09:54:30,699 INFO Forking
jetpack convergeto configure VM software2025-04-17 09:55:16,028 ERROR Failed to configure software
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 70, in main
subprocess.check_call('jetpack converge --mode=install', shell=True)
File "/opt/cycle/jetpack/system/python/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'jetpack converge --mode=install' returned non-zero exit status 1.
-
Arko 4,185 Reputation points Moderator
2025-04-17T13:45:08.8533333+00:00 Hello Jason Gumarang, the above error logs which you shared means the
jetpack convergecommand which runs during VM provisioning to install/configure software like Slurm, Munge, and the scheduler stack failed. And since no other error context is logged in/opt/cycle/jetpack/logs/jetpackd.log, it likely failed inside a specific Jetpack "project role" script (e.g.,slurmdbd,slurmctld,munge).From your message and logs, shared so far below is what I can understand or narrow down into-
CycleCloud deployed a new VM, but Jetpack failed to fully configure Slurm. You opened NSG ports, but that alone doesn’t guarantee slurmdbd is running
Why?
because it may not have been installed at all.How am I suspecting that?
The journal output is empty (journalctl -u jetpack), meaning Jetpack didn’t even register properly with systemd.Most importantly the repeated failure suggests something is broken in the custom install role, like missing repo, permission issue, or wrong package version.
I would recommend you try to install it manually once using
sudo /opt/cycle/jetpack/bin/jetpack converge --mode=install --verboseThis will show which specific part of the install is failing — whether it’s Munge, Slurm, Python package, or system dependencies. Based on that we can narrow down on the blocker
-
Jason Gumarang 25 Reputation points
2025-04-22T12:33:56.0066667+00:00 hi Support, i have time to do this tomorrow and i will provide a feedback on Thursday this week. Hope you can keep this case open for now.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 4%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator2025-04-23T22:26:32.48+00:00 Hello, @Jason Gumarang ! I've followed up via private message on this thread.
-
Jason Gumarang 25 Reputation points
2025-04-24T09:05:30.2966667+00:00 Hi Support,
Since we merge this with my other post (https://learn.microsoft.com/en-us/answers/questions/2245815/slurm-scheduler-cant-be-access-via-ssh-or-bastion?page=1#answer-2024996), i would suggest that we will focus first in the issue on the missing /share/home after reboot.
It seems that there was an issue running the jetpack run.
[root@jason-test-scheduler ~]# jetpack run --parts allUsage: jetpack [OPTIONS] COMMAND [ARGS]...
Error: No such command "run".
Per system, these are the only arguments that are available:
Commands:
autoscale Autoscales the cluster the node is part of
config Gets a configuration value
converge Triggers a converge on the current node
download Download a project file from cloud storage
keepalive Delays the healthcheck daemon
log Sends log message to CycleCloud
report_issue Upload logs to Azure Storage.
run_on_shutdown Add a script to be called prior to node termination
send Sends a message to CycleCloud
shutdown terminates the node
test Runs python style unittests on the node
users
Here are some logs:
2025-04-24 08:35:15,207 INFO Getting node configuration from /clusterlink/userdata/ea3a7cc839e2133c72eba8176d59cfe62025-04-24 08:35:15,246 INFO Scheduled event monitor thread started
2025-04-24 08:35:15,246 INFO Forking
jetpack convergeto configure VM software2025-04-24 08:37:32,670 INFO Software configuration complete!
2025-04-24 08:37:43,182 INFO Getting latest auth configuration from /clusterlink/auth/ea3a7cc839e2133c72eba8176d59cfe6
2025-04-24 08:38:43,195 INFO Getting latest auth configuration from /clusterlink/auth/ea3a7cc839e2133c72eba8176d59cfe6
2025-04-24 08:39:43,207 INFO Getting latest auth configuration from /clusterlink/auth/ea3a7cc839e2133c72eba8176d59cfe6
2025-04-24 08:40:43,220 INFO Getting latest auth configuration from /clusterlink/auth/ea3a7cc839e2133c72eba8176d59cfe6
2025-04-24 08:41:14,285 INFO Auto-acknowledging EventID D0B8F90C-4225-4EF8-B3CA-9A13B0C9B6BB for resources ['scheduler-GRSTQYTCMI3TQLJVGUYWMLJUGA'] (Reboot)
2025-04-24 08:41:43,233 INFO Getting latest auth configuration from /clusterlink/auth/ea3a7cc839e2133c72eba8176d59cfe6
2025-04-24 08:42:02,494 ERROR Failed to get instance status from CycleCloud
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 23, in initialize
config = jetpack.util.get_current_config() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/util/init.py", line 604, in get_current_config
r = query_cyclecloud(url, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/util/init.py", line 577, in query_cyclecloud
conn.request(method, path, body=body, headers=default_headers)File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1336, in request
self._send_request(method, url, body, headers, encode_chunked)File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1382, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1331, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1091, in _send_output
self.send(msg)File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1035, in send
self.connect()File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1470, in connect
super().connect()File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1001, in connect
self.sock = self._create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^File "/opt/cycle/jetpack/system/python/lib/python3.12/socket.py", line 865, in create_connection
raise exceptions[0]File "/opt/cycle/jetpack/system/python/lib/python3.12/socket.py", line 850, in create_connection
sock.connect(sa)ConnectionRefusedError: [Errno 111] Connection refused
2025-04-24 08:42:02,511 ERROR Failed to get instance status from CycleCloud
Traceback (most recent call last):
File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/service.py", line 23, in initialize
config = jetpack.util.get_current_config() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/util/init.py", line 604, in get_current_config
r = query_cyclecloud(url, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/opt/cycle/jetpack/system/python/lib/python3.12/site-packages/jetpack/util/init.py", line 577, in query_cyclecloud
conn.request(method, path, body=body, headers=default_headers)File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1336, in request
self._send_request(method, url, body, headers, encode_chunked)File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1382, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1331, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)File "/opt/cycle/jetpack/system/python/lib/python3.12/http/client.py", line 1091, in _send_output
ConnectionRefusedError: [Errno 111] Connection refused
2025-04-24 08:44:11,673 WARNING While querying instance metadata, encountered <urlopen error [Errno 110] Operation timed out> retrying in 2 seconds
2025-04-24 08:44:13,716 INFO Scheduled event monitor thread started
2025-04-24 08:44:13,720 ERROR The home directory mounted at /shared/home is not available
2025-04-24 08:45:13,723 ERROR The home directory mounted at /shared/home is not available
2025-04-24 08:46:13,727 ERROR The home directory mounted at /shared/home is not available
2025-04-24 08:47:13,730 ERROR The home directory mounted at /shared/home is not available
2025-04-24 08:48:13,733 ERROR The home directory mounted at /shared/home is not available
2025-04-24 08:49:13,737 ERROR The home directory mounted at /shared/home is not available
2025-04-24 08:50:13,740 ERROR The home directory mounted at /shared/home is not available
2025-04-24 08:51:13,744 ERROR The home directory mounted at /shared/home is not available
Sign in to comment
Sign in to answer