When running an MPI job on HPC, it executes successfully when the resource is set to “Node,” but fails when set to “Socket.”

The behavior difference between “Node” and “Socket” resources for an MPI job on HPC is explained by how the job scheduler allocates hardware and how MPI processes are placed and pinned.

On Windows HPC, an MPI job is typically expressed as a parallel task launched via mpiexec [mpi_options] <myapp.exe> [arguments]. The scheduler then allocates cores/sockets/nodes and may also set processor affinity for tasks.

Key points from the platform behavior:

1. MPI jobs and resource granularity
   • An MPI task usually runs a single executable concurrently on multiple cores with communication between processes.
   • When the resource is set to “Node,” the scheduler allocates whole nodes, which matches the recommended configuration for MPI jobs that manage their own process placement and affinity.
   • When the resource is set to “Socket,” the scheduler allocates at the socket level, which can interact differently with how mpiexec expects to place ranks and how many cores are available per allocation.
2. Affinity interaction with MPI
   • The job scheduler’s Affinity Type setting controls whether it pins tasks to cores and how.
   • For MPI jobs, the recommended Affinity Type is Non-Exclusive jobs (recommended) so that MPI jobs that run exclusively on a node can manage their own affinity.
   • If the scheduler is setting affinity while the job is constrained to “Socket,” it can override mpiexec’s placement and cause mismatches between requested ranks and available cores on that socket, leading to failures.
   • If the job scheduler sets affinity, any affinity specified in mpiexec is ignored, which can degrade performance or cause failures when the MPI application assumes full-node control.
3. Practical implication for the failure
   • When using “Node,” the MPI job likely receives full-node allocations, and mpiexec can place ranks freely across all cores, consistent with how MS-MPI and HPC Pack are designed to run parallel tasks.
   • When using “Socket,” the allocation may be partial (per socket), and combined with scheduler affinity, this can restrict or conflict with MPI rank placement, causing the job to fail even though it runs successfully with node-level allocation.

To avoid this issue for MPI jobs on HPC 2019:

• Prefer node-level allocation (resource = Node) for MPI jobs.
• Ensure the job scheduler Affinity Type is set to Non-Exclusive jobs (recommended) or No jobs if the MPI application manages its own affinity and runs exclusively on nodes.
• Keep mpiexec as the launcher for MPI tasks and avoid configurations where the scheduler’s socket-level allocations conflict with the number of ranks or cores requested by mpiexec.

References:

• Understanding Parallel Computing Jobs
• Understanding Affinity
• mpiexec
• job submit
• Guidelines for Running MPI Applications in Azure
• Guidelines for Running HPC Applications on Azure Nodes