Parallel cyclecloud slrum process stacked while reading input file

Question

Parallel cyclecloud slrum process stacked while reading input file

Eran Arad 0

I am running a parallel job, using openmpi, on cycle cloud, using slrum (batch) on 5 nodes, 120 cores in each.

The job starts with reading the computational mesh, by each core. While reading, a mesh lock file appears for each core and disappears when it finishes reading. The problem: a new mesh lock file then appears. The process is stacked, though squeue shows as if it keeps running.

Running on a single node with 120 cores using mpirun directly (without sbatch) works fine, so it is slrum issue. Used to work for many runs. Actually worked yesterday. But now, though I restarted the scheduler several times, the process get stacked. Anyone has an idea how to resolve the issue?

Anonymous

2024-08-23T15:06:34.0866667+00:00

Hi Eran Arad,
Thank you for reaching out to the Microsoft Q&A forum.

If the job works fine on a single node but gets stuck when running on multiple nodes, the issue might be related to how file locking or synchronization is handled across multiple nodes. The appearance of new mesh lock files suggests there could be conflicts when multiple processes access the same files simultaneously. Check for any race conditions or conflicts in file handling.

Additionally, ensure that the nodes have sufficient resources, including CPU, memory, and disk I/O. Network issues or MPI configuration problems might also be contributing factors, so verify network stability and compatibility of your MPI version. Review your SLURM configuration and job dependencies for any recent changes or issues. Finally, adding more detailed logging to your code could help identify the root cause of the problem.

Please refer to this document for SLURM configuration

If you found it helpful, please click "Upvote" on my post let us know. If you have any additional questions or need further assistance, feel free to reach out.

Thank you!
Eran Arad 0 Reputation points

2024-08-23T19:52:30.5566667+00:00

Screenshot_2024-08-23_22-32-15.png

Thanks for your answer

This is the lock file that shows up when the code gets stuck. basically it is empty

The slrum.out file is empty till I stop the run. Then, its last lines are:

StripedMesh::read_mles shIsol_62_cut_10.mles on 600 ranks...

slurmstepd: error: *** JOB 5 ON slurm-cluster-hpc-1 CANCELLED AT 2024-08-23T19:32:36 ***

That is, the problem indeed occurred during reading. The last line is the result of my cancellation command.

Just yesterday, on the same files with minor differences in parameters, the same line showed up in the slrum out file but the run continued:

StripedMesh::read_mles shIsol_62_cut_10.mles on 600 ranks...

... code messages about input ...

StripedMesh::buildExtendedFaces()

... run and printup continued.

Now I have not changed anything about slrum, or the code. I ran it many times using a script as follows:

#!/bin/bash

#SBATCH --job-name=rlc05 # Job name

#SBATCH --partition=hpc # Submit queue

#SBATCH --nodes=5 # Total number of nodes requested

#SBATCH --ntasks-per-node=120

source /etc/profile

module load mpi/openmpi

mpirun -n 600 [execute] -i rlc05.in 2>&1 | tee out.log

and it worked well (including yesterday). The only difference between yesterday and today is that I restarted the scheduler, and it reallocated the cyclecloud cluster. Could it be that there has been in the last 24 hours a change in the global slrum setting in Azure-cyclcloud ?

Thanks,

Eran
Anonymous

2024-08-26T12:55:20.9+00:00

Hi Eran Arad,
Thank you for getting back to us.

After the scheduler was restarted and the cluster was reallocated, it's possible that some new settings or configurations might be affecting your job. The restart could have introduced changes in how resources are allocated or managed, which might be causing the job to get stuck. There might also be new network or file system issues, or differences in the software environment that are impacting the job's execution. Reviewing these aspects should help determine why the job is failing now when it previously ran without issues.

And also, If possible, you can, try it on test env running the job on a different cluster or with a different setup to see if the problem continues

If this reply was helpful, please don’t forget "Upvote" on my post let us know. If you need further assistance, feel free to reach out.
Anonymous

2024-08-27T12:32:21.83+00:00

Hi Eran Arad,
Just checking in to see if you had a chance to see my comment to your question.

If it was helpful, please click "Upvote" on my post let us know. If you have any additional questions or need further assistance, feel free to reach out.

Thank You.

Your answer

Anonymous

2024-08-23T15:06:34.0866667+00:00

Hi Eran Arad,
Thank you for reaching out to the Microsoft Q&A forum.

If the job works fine on a single node but gets stuck when running on multiple nodes, the issue might be related to how file locking or synchronization is handled across multiple nodes. The appearance of new mesh lock files suggests there could be conflicts when multiple processes access the same files simultaneously. Check for any race conditions or conflicts in file handling.

Additionally, ensure that the nodes have sufficient resources, including CPU, memory, and disk I/O. Network issues or MPI configuration problems might also be contributing factors, so verify network stability and compatibility of your MPI version. Review your SLURM configuration and job dependencies for any recent changes or issues. Finally, adding more detailed logging to your code could help identify the root cause of the problem.

Please refer to this document for SLURM configuration

If you found it helpful, please click "Upvote" on my post let us know. If you have any additional questions or need further assistance, feel free to reach out.

Thank you!
Eran Arad 0 Reputation points

2024-08-23T19:52:30.5566667+00:00

Screenshot_2024-08-23_22-32-15.png

Thanks for your answer

This is the lock file that shows up when the code gets stuck. basically it is empty

The slrum.out file is empty till I stop the run. Then, its last lines are:

StripedMesh::read_mles shIsol_62_cut_10.mles on 600 ranks...

slurmstepd: error: *** JOB 5 ON slurm-cluster-hpc-1 CANCELLED AT 2024-08-23T19:32:36 ***

That is, the problem indeed occurred during reading. The last line is the result of my cancellation command.

Just yesterday, on the same files with minor differences in parameters, the same line showed up in the slrum out file but the run continued:

StripedMesh::read_mles shIsol_62_cut_10.mles on 600 ranks...

... code messages about input ...

StripedMesh::buildExtendedFaces()

... run and printup continued.

Now I have not changed anything about slrum, or the code. I ran it many times using a script as follows:

#!/bin/bash

#SBATCH --job-name=rlc05 # Job name

#SBATCH --partition=hpc # Submit queue

#SBATCH --nodes=5 # Total number of nodes requested

#SBATCH --ntasks-per-node=120

source /etc/profile

module load mpi/openmpi

mpirun -n 600 [execute] -i rlc05.in 2>&1 | tee out.log

and it worked well (including yesterday). The only difference between yesterday and today is that I restarted the scheduler, and it reallocated the cyclecloud cluster. Could it be that there has been in the last 24 hours a change in the global slrum setting in Azure-cyclcloud ?

Thanks,

Eran
Anonymous

2024-08-26T12:55:20.9+00:00

Hi Eran Arad,
Thank you for getting back to us.

After the scheduler was restarted and the cluster was reallocated, it's possible that some new settings or configurations might be affecting your job. The restart could have introduced changes in how resources are allocated or managed, which might be causing the job to get stuck. There might also be new network or file system issues, or differences in the software environment that are impacting the job's execution. Reviewing these aspects should help determine why the job is failing now when it previously ran without issues.

And also, If possible, you can, try it on test env running the job on a different cluster or with a different setup to see if the problem continues

If this reply was helpful, please don’t forget "Upvote" on my post let us know. If you need further assistance, feel free to reach out.
Anonymous

2024-08-27T12:32:21.83+00:00

Hi Eran Arad,
Just checking in to see if you had a chance to see my comment to your question.

If it was helpful, please click "Upvote" on my post let us know. If you have any additional questions or need further assistance, feel free to reach out.

Thank You.