Slurmctld service stopped with fatal glibc error

Question

Slurmctld service stopped with fatal glibc error

Anonymous

A cyclecloud cluster shows this error in cyclecloud GUI.

The logs for slurmctld shows glib errors.

Restarting the slrumctld, resolved temporatily, however the error occurs again.
<redacted Logs PII info>

Manish Deshpande 8,135 Reputation points Microsoft External Staff Moderator

2026-01-27T17:43:29.9833333+00:00
Hello @Anonymous

Thank you for contacting us about the cycle cloud issue.

The failure occurs when Slurm controller cannot reliably resolve hostnames, usually due to one of the following:

DNS misconfiguration inside Azure VMs (systemd‑resolved / incorrect resolv.conf)

CycleCloud cluster hostname not resolvable via DNS

/etc/hosts missing static entries expected by Slurm

Intermittent DNS failures causing glibc name resolution to fail

Because glibc relies strictly on working name resolution, Slurm fails hard when DNS breaks even briefly.

Steps to confirm :

1.Validate hostname resolution on the Slurm controller :

hostname hostname -f getent hosts $(hostname)

These commands must return an IP address.

2.Check DNS configuration

Run : cat /etc/resolv.conf

At least one reachable DNS server

No invalid or removed nameservers

No looping entries caused by systemd-resolved misconfiguration

If systemd‑resolved is in use:

Run : resolvectl status

3.Add static hostname mappings :

Edit /etc/hosts on all Slurm nodes (controller + compute)
run : sudo vi /etc/hosts

Add entries similar to:
10.0.0.4 slurm-controller

10.0.0.5 compute0001

10.0.0.6 compute0002

Hostnames must match exactly what is defined in slurm.conf
This ensures Slurm never depends solely on DNS for critical resolution.

4.Verify slurm.conf

Confirm hostname consistency:

grep -E "ControlMachine|NodeName" /etc/slurm/slurm.conf

ControlMachine must match the system hostname

NodeName entries must match DNS or /etc/hosts

5.Restart Slurm services

sudo systemctl restart slurmctld sudo systemctl status slurmctld

Restarting slurmctld works only until the next DNS resolution failure occurs. Once glibc fails to resolve a hostname again, Slurm terminates.

Correcting name resolution permanently is the only lasting fix.

The slurmctld failures are caused by intermittent hostname resolution issues, not by glibc itself. Slurm requires consistent name resolution; when DNS fails, slurmctld exits. Configuring stable DNS or adding static /etc/hosts entries for the controller and compute nodes permanently resolves the issue. This fix has successfully addressed the same problem in previous CycleCloud Slurm clusters.

Links:
https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances?tabs=redhat
https://slurm.schedmd.com/faq.html#hostname

Thanks,
Manish Deshpande.
Manish Deshpande 8,135 Reputation points Microsoft External Staff Moderator

2026-01-30T15:09:28.5066667+00:00

Hello @Mario Vicente Quinto (Ascendion Inc) ,

I wanted to check if my last response made sense. I’d be glad to assist further or explain anything in more detail

Answer accepted by question author

0 additional answers

Your answer

Manish Deshpande 8,135 Reputation points Microsoft External Staff Moderator

2026-01-30T15:09:28.5066667+00:00

Hello @Mario Vicente Quinto (Ascendion Inc) ,

I wanted to check if my last response made sense. I’d be glad to assist further or explain anything in more detail

Answer 1

Hello @Mario Vicente Quinto (Ascendion Inc) ,

Thank you for your patience while we worked through this issue together and collaborated with the product group to reach a resolution.

Sharing the summary of the issue, the mitigations we attempted, and what ultimately resolved it.

Issue Description:

The customer was running Azure CycleCloud with Slurm (Ubuntu 22.x) and began seeing intermittent scheduler errors:

_xgetaddrinfo: getaddrinfo(<node>:6818) failed: Temporary failure in name resolution error: slurm_set_addr: Unable to resolve "<node>" Fatal glibc error: malloc.c:2599 (sysmalloc): assertion failed

The DNS resolution failures were followed by fatal malloc assertion errors in slurmctld, causing scheduler instability.

Initial slurm version: Slurm 25.05.2

Mitigations Attempted

Based on guidance from the Azure CycleCloud Slurm documentation (https://github.com/Azure/cyclecloud-slurm?tab=readme-ov-file#ubuntu-22-or-greater-and-dns-hostname-resolution), we applied:

[[[configuration]]]
slurm.ubuntu22_waagent_fix = false

This suppressed the issue temporarily, but the errors reappeared after several days.

Later, we engaged the Engineering team for deeper investigation. Their findings indicated:

Repeated hostname resolution failures can cause slurmctld to enter a corrupted internal state.
This may surface as glibc malloc assertion failures.
A historical workaround was modifying NodeHostname and NodeAddr for the problematic node.
However, newer Slurm versions include fixes addressing this memory corruption behavior

Final Resolution

The customer upgraded:

CycleCloud -> 8.8.2
Slurm -> 25.05.5

After upgrading from Slurm 25.05.2 to 25.05.5, the issue was fully resolved and has not reoccurred.

Steps taken to upgrade version:

Below are the high-level steps used to perform the upgrade:

Upgrade CycleCloud

yum upgrade -y cyclecloud8

Export Existing Cluster Parameters

cyclecloud export_parameters -o CLUSTER_NAME.json CLUSTER_NAME

Edit CLUSTER_NAME.json to set the new Slurm version.

Update Project Versions

If you want the latest versions of azslurm and healthagent, update your cluster template before re-importing:

Edit the cluster template.
Update all [[[cluster-init ...]]] lines to reference the latest project versions.

Re-Import the Cluster

cyclecloud import_cluster CLUSTER_NAME -c slurm -f ./slurm.txt -p ./CLUSTER_NAME.json --force

Drain and Down Execute Partitions

In Slurm:

Drain the partitions.
Wait until nodes reach the DOWN state.

Reimage Nodes

Once nodes are down, use the Actions -> Reimage option in the CycleCloud UI to apply the upgraded configuration to the nodes.