MPI - Rank and size

Objectives

• Use rank and size in MPI_COMM_WORLD to identify individual MPI processes

• Explain a communicator and its purpose

Instructor note

• 15 min teaching

• 10 min exercises

 

MPI - rank and size in MPI_COMM_WORLD

MPI programming paradigm

Note

• Each MPI process runs the same code (written in a conventional sequential language)

• The code is the same in each MPI process –> SPMD (single program multiple data)

  • the varables have the same names on each MPI process but hold different data

  • variables can be directly accessed only by the MPI process that owns the data

  • all MPI processes have their own memory (distributed memory), i.e., all variables are private

  • MPI is mostly done in this SPMD (single program multiple data) way

  • but if needed MPMD (multiple program multiple data) is possible as well

• MPI processes (will later on) communicate their data via special communication routines
   

• Data & work has to be distributed among the MPI processes

  • a system of size processes is started by a special MPI initialization program

  • the values of size and rank are returned by a special library routines

  • all distribution decisions (data & work) are based on rank
     

• All processes of one MPI program are combined in the communicator MPI_COMM_WORLD
  (MPI_COMM_WORLD is a predefined handle in the mpi.h header)

  • the communicator is the communication context,
    communication can be done only within a communicator

  • size is the number of processes in a communicator

  • each process has its own rank in a communicator, rank = 0 - (size-1)

World communicator:   MPI_COMM_WORLD

Getting the rank of a process:

CFortranPython

MPI_Comm_rank(comm, rank)

• returns the rank of this process in a communicator

• rank – identifies the different processes within a communicator – it is the basis for any work and data distribution

• IN       comm   communicator (handle)

• OUT   rank      rank of the calling MPI process in group of comm (integer)

int MPI_Comm_rank(MPI_Comm comm, int *rank)

• Usage:   MPI_Comm_rank(MPI_COMM_WORLD, &rank);

Getting the number of processes:

CFortranPython

MPI_Comm_size(comm, size)

• returns the number of processes (size) in a communicator

• size – is the number of processes in a communicator

• IN       comm   communicator (handle)

• OUT   size       number of MPI processes in the group of comm (integer)

int MPI_Comm_size(MPI_Comm comm, int *size)

• Usage:   MPI_Comm_size(MPI_COMM_WORLD, &size);

Exercise

Modify the program below so that – every process writes its rank and the size of MPI_COMM_WORLD,
– only the process with rank = 0 in MPI_COMM_WORLD prints “Hello world!”.

What happens if you do NOT modify the cell below? Try it out!

In C and Fortran you can compile and execute without modifying the code below.
Give it a try before you actually modify.
Note, if you don’t change the Python code below the mpirun will show errors.
What happens here? Why is this possible at all?
Of course, in the end you have to modify the code below.

CFortranPython

#include <stdio.h>
#include <mpi.h>

int main(int argc, char *argv[])
{
   int rank, size;

   MPI_Init(&argc, &argv);

// QUERY rank of your communicator

// QUERY size of your communicator

// ONLY 1 PROCESS (rank == 0) prints "Hello world!"
    
     printf ("Hello world!\n");


   printf("I am process %i out of %i.\n", rank, size);

   MPI_Finalize();
}

Compile it as an MPI program:

CFortran

mpicc myrank.c -o myrank

Run it with 4 MPI processes (run it several times to see run to run variations):

C, FortranPython

mpirun -np 4 ./myrank

Expected output with 4 MPI processes:

I am process 2 out of 4
Hello world!
I am process 0 out of 4
I am process 3 out of 4
I am process 1 out of 4

Tip

Getting only zeros or getting strange numbers?

• Did you forget to modify the code above?
  

• Each process needs to query for rank and size!

Exercise

Try different numbers of MPI processes (run it several times to see run to run variations):

C, FortranPython

mpirun -np 8 --oversubscribe ./myrank

Solution (please try to solve the exercise by yourself before looking at the solution)

The correct MPI program:

Solution

CFortranPython

%%writefile tmp/myrank.c
                                                        // Modify 5 lines:
#include <stdio.h>
#include <mpi.h>

int main(int argc, char *argv[])
{
   int rank, size;

   MPI_Init(&argc, &argv);

   MPI_Comm_rank(MPI_COMM_WORLD, &rank);                // Figure out rank

   MPI_Comm_size(MPI_COMM_WORLD, &size);                // Figure out size

   if (rank == 0)                                       // Output done by only one MPI process
   {                                                    // {
     printf ("Hello world!\n");
   }                                                    // }

   printf("I am process %i out of %i.\n", rank, size);

   MPI_Finalize();
}

Discussion

Did you notice / can you explain?

• Why is the sequence of the output non-deterministic?

• Is there a way to get the output ordered by the rank number?

• It’s not determined by the different speed of the MPI processes (this might play a role, but it’s not the reason why we see this non-deterministic behavior with our little “Hello world!” program.

• The explanation is simple:
  There are no sequence rules in place, neither by MPI nor by the OS!

• The MPI Standard does not require any specific order of the output from different MPI processes.
  

• The Linux Operating System does not have any rules for merging the output from serveral processes into one output stream that will be printed on the screen or into a file.

• Therefore, we see a non-deterministic output sequence. Any order just happens by chance.
   

• Why should you know about this?
  Simply to not get confused when doing “poor man’s debugging” later on and to prevent you from believing that strange things are going on when your MPI processes seem to write their debugging output in a weird order…
   

• How to get the output ordered by the rank number?
  The only way to achieve this is that all processes communicate their result to one process (e.g., rank 0) and only this one process prints the output. Of course, this involves serialization and from the point of view of performance one would prefer to not do this.

Keypoints

• Communicator MPI_COMM_WORLD

• Use rank and size in MPI_COMM_WORLD to identify individual MPI processes

See also

Details and recommended reading
MPI 5.0 Section 7.4 - Communicator Management

• pages 317-318: 7.4.1 Communicator Accessors - MPI_Comm_size, MPI_Comm_rank