MPI - MPI deep dive - Hello world! extended and job submission

Objectives

  • Figure out the version of the MPI library in use

  • See an extended version of the “Hello world!” program that also prints the core ID (on the node) and the processor (node) name where the MPI processes are running

  • Explore job submission on a cluster

Instructor note

  • 0 min teaching

  • 10 min exercises or demo

MPI - version test

Run the version test to figure out the version of the MPI library and of the header in use:

#include <stdio.h>
#include <mpi.h>

int main (int argc, char *argv[])
{
  int lib_version, lib_subversion;

  MPI_Init(&argc, &argv);

  MPI_Get_version(&lib_version, &lib_subversion);
  printf ("Version: Library: %d.%d, mpi.h: %d.%d\n", 
           lib_version, lib_subversion, MPI_VERSION, MPI_SUBVERSION);

  MPI_Finalize();
}

Compile the version test:

mpicc version_test.c -o version_test

Run the version test:

mpirun -np 1 ./version_test

 

Extended version of the MPI - Hello world!

This is an extended version of the “Hello world!” program that also prints the core ID (on the node) and the processor (node) name where the MPI processes are running.

#include <stdio.h>
#include <sched.h>                                                // sched_getcpu()
#include <mpi.h>                                                  // MPI header

int main(int argc, char *argv[])
{

int rank = 0, size = 1;                                            // initialize rank and size
int core_id;                                                       // ... core_id
int comm_name_length;                                              // ... MPI communicator name
char comm_name[MPI_MAX_OBJECT_NAME];                               // ... MPI communicator name
int node_name_length;                                              // ... MPI processor/node name
char node_name[MPI_MAX_PROCESSOR_NAME];                            // ... MPI processor/node name

MPI_Init(&argc, &argv);                                            // MPI initialization

MPI_Comm_rank(MPI_COMM_WORLD, &rank);                              // MPI rank
MPI_Comm_size(MPI_COMM_WORLD, &size);                              // MPI size

MPI_Comm_get_name(MPI_COMM_WORLD, comm_name, &comm_name_length);   // ... MPI communicator name
MPI_Get_processor_name(node_name, &node_name_length);              // ... MPI processor/node name
core_id = sched_getcpu();                                          // ... core_id

if (rank == 0)
{
printf ("\n");
printf ("he-mpi - MPI program to print core_id & node_name (cb)\n");
printf ("Hello world! from %4i MPI processes in %s\n", size, comm_name);
}

printf ("MPI process %4i / %4i ON core %4i of node %s\n", rank, size, core_id, node_name);

MPI_Finalize();                                                    // MPI finalization

}

Compile the extended version (note the additional option -D_GNU_SOURCE (needed for sched.h)):

mpicc he-mpi.c -o he-mpi -D_GNU_SOURCE

Run the extended version (note additional options and sorting of the output):

mpirun -np 4 --map-by core --rank-by slot --bind-to hwthread --report-bindings ./he-mpi | sort +2n -3 | cut -c 1-54

Resources available (within the notebook) (How many and which cores are available, how many MPI processes can be started):

import os
print(os.uname()[1], os.sched_getaffinity(0))

Expected output on the VSC-5 cluster in the MPI courses will be something similar to:

n3503-022 {0, 1, 2, 3, 128, 129, 130, 131}

nodename { 4 physical cores (out of 0 - 127) + their hardware hyperthreads (+128) }

Note, on VSC-5 there are 128 physical cores per node where first all hwt 0 are numbered consecutively (0 - 127) and then all hwt 1 (128 - 255). Therefore, e.g., processing unit number 0 (hwt 0) and processing unit number 128 (hwt 1) are both on the same physical core.
On VSC-5 for each pysical core:
(processing unit number of hwt 1)   =   (processing unit number of hwt 0) + 128

Note

A note on pinning: mpirun –bind-to core OR hwthread?

  • First of all, with running MPI programs you should always take care of a good pinning! Pinning is related to the performance that you will get in the end. The recommendation is to always take care of a proper pinning and not rely on any defaults. Without pinning, it might happen that some MPI processes share the same hardware resources (are put on the same physical core) or that the MPI processes are moved around and even accross NUMA domains by the operationg system - both will slow down execution.

  • The recommendation with pure MPI applications is to pin the MPI processes at least to physical cores and to use all physical cores available on a node if there are no other constraints. (Other constraints could be, e.g., memory-bandwidth bound applications, not enough memory on the node, or very poor scalability of the application. The former two could benefit from having a lower number of MPI processes per node, i.e., not using all physical cores of the node, but with an equal spread accross the NUMA domains of the node. If the code does not scale up to the number of available cores (nowadays the nodes in HPC systems have a quite high core count) the only way out is to use only a part of the node (on VSC-5 you can request a part of a node if instead of --nodes (-N) in your job scripts you use --ntasks (-n) together with --mem with a rule of thumb of 4G per core).)

  • Depending on the applications’ internal communication patterns, some applications might benefit from a specific way of pinning (e.g., consecutive pinning if the MPI processes communicate mainly with their neighbors) while other applications do not have any special needs as long as the MPI processes are pinned.

  • With --bind-tohwthread the MPI processes are pinned (bound) to the hwthread (hwt 0 and “B.”), see above.

  • With --bind-to core the MPI processes are pinned (bound) to physical cores (hwt 0-1 and “BB”), i.e., an MPI process can move between the two hardware hyperthreads available on the physical core, it even could use both hardware hyperthreads simultaneously by other means of parallelzation (e.g., shared-memory parallelization with OpenMP).

  • It does not really matter on which hardware hyperthread (hwt) of the core an MPI process is running. With --bind-to core you will see in the output of he-mpi that some MPI processes are running on hwt 0 and some on hwt 1, that’s just fine.

Job submission

We are going to use the he-mpi.c from above and will run it on two nodes of VSC-5.

Job script using the Open-MPI library

%%writefile job_openmpi.sh
#!/bin/bash

#SBATCH --job-name        openmpi         # job name
#SBATCH --nodes           2               # number of nodes (exclusive)
#SBATCH --clusters        vsc5            # submit to VSC-5
#SBATCH --qos             zen3_0512       # use a qos
#SBATCH --partition       zen3_0512       # use partition that fits to the qos
#SBATCH --ntasks-per-node 128             # SLURM_NTASKS_PER_NODE  [1 mpi/core]
#SBATCH --reservation     jh_training_mpi # reservation DURING COURSE ONLY
#SBATCH --time            0-00:05:00      # max runtime (5 min during course)

module purge                          # recommended to be done in all jobs !!!!!
# module load <modules>               # load only modules actually needed by job
module load openmpi/4.1.6-gcc-12.2.0-exh7lqk # Open-MPI (including gcc)

echo
echo 'Hello from node: '$HOSTNAME
echo 'Number of nodes: '$SLURM_JOB_NUM_NODES
echo 'Tasks per node:  '$SLURM_TASKS_PER_NODE
echo 'Partition used:  '$SLURM_JOB_PARTITION
echo 'QOS used:        '$SLURM_JOB_QOS
echo 'Using the nodes: '$SLURM_JOB_NODELIST
echo

# <do_my_work>

echo 'Compiling he-mpi & running it on all available cores & sort for output ordering'

mpicc he-mpi.c -o he-mpi -D_GNU_SOURCE   # could/should be done outside of job

mpirun -np $SLURM_NTASKS --map-by core --rank-by slot --bind-to hwthread ./he-mpi | sort +2n -3 | cut -c 1-54

Submit the job script with sbatch

sbatch job_openmpi.sh

Note, on the JupyterHub, we have to get rid of the JupyterHub environment before we can submit a job

source ../tooling/unload_jupyter_env.sh; sbatch job_openmpi.sh

Check the job status as often as needed (job is queued (PD), running (R), or done (empty list))

squeue --me

Click on the slurm-JOB_ID.out file in the left menu (if you did not submit a job you can look at the provided job output).

Job script using the Intel-MPI library

%%writefile job_intelmpi.sh
#!/bin/bash

#SBATCH --job-name        intelmpi        # job name
#SBATCH --nodes           2               # number of nodes (exclusive)
#SBATCH --clusters        vsc5            # submit to VSC-5
#SBATCH --qos             zen3_0512       # use a qos
#SBATCH --partition       zen3_0512       # use partition that fits to the qos
#SBATCH --ntasks-per-node 128             # SLURM_NTASKS_PER_NODE  [1 mpi/core]
#SBATCH --reservation     jh_training_mpi # reservation DURING COURSE ONLY
#SBATCH --time            0-00:05:00      # max runtime (5 min during course)

module purge                          # recommended to be done in all jobs !!!!!
# module load <modules>               # load only modules actually needed by job
module load intel intel-mpi           # VERY OLD Intel compiler and Intel-MPI

echo
echo 'Hello from node: '$HOSTNAME
echo 'Number of nodes: '$SLURM_JOB_NUM_NODES
echo 'Tasks per node:  '$SLURM_TASKS_PER_NODE
echo 'Partition used:  '$SLURM_JOB_PARTITION
echo 'QOS used:        '$SLURM_JOB_QOS
echo 'Using the nodes: '$SLURM_JOB_NODELIST
echo

# <do_my_work>

echo 'Compiling he-mpi & running it on all available cores & sort for output ordering'

mpiicc he-mpi.c -o he-mpi-intel -D_GNU_SOURCE   # could/should be done outside of job

export I_MPI_PIN=1
export I_MPI_PIN_RESPECT_CPUSET=0
export I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=0
export I_MPI_PIN_PROCESSOR_LIST=0-127

mpirun -np $SLURM_NTASKS ./he-mpi-intel | sort +2n -3 | cut -c 1-54

Submit the job script with sbatch

sbatch job_intelmpi.sh

Note, on the JupyterHub, we have to get rid of the JupyterHub environment before we can submit a job

source ../tooling/unload_jupyter_env.sh; sbatch job_intelmpi.sh

Check the job status as often as needed (job is queued (PD), running (R), or done (empty list))

squeue --me

Click on the slurm-JOB_ID.out file in the left menu (if you did not submit a job you can look at the provided job output).

Keypoints

  • Version of the MPI library in use

  • Extended version of the “Hello world!” with information about the hardware ressources

  • Job submission on a cluster