Point-to-Point blocking Send and Receive in MPI

Point-to-point communication
- The term point-to-point communication is used to describe a communication between one source with one destination
- The term broadcast communication is used to describe a communication between one source with all destinations

Blocking and non-blocking communication

There are 2 (two) communication primitives in MPI:
The communication primitives in MPI can be

Blocking communications

blocking send operation:

A blocking send operation terminates when the message is received by the destination
I.e., a program that invokes a blocking send primitive will block until the message is received by the destination

blocking receive operation:

A blocking receive operation terminates when a message is received by the caller
I.e., a program that invokes a blocking receive primitive will block until a message is received by the caller

NON-Blocking communications

non-blocking send operation:

A non-blocking send operation terminates when the message is sent by the sender
Note that the message has not been received yet
I.e., a program that invokes a non-blocking send primitive will NOT wait until the message is received by the destination
The non-blocking send primitive will return a status variable with which the sender can query the status of the message sent

non-blocking receive operation:

A blocking receive operation always terminates immediately
Note that a message may not been received
The non-blocking receive primitive will return a status variable with which the caller can test whether a message was received

The reason for providing NON-blocking send and receive operations is to allow the process to do something else while waiting for the message to be transmitted or for some message to arrive...
Obviously, after invoking NON-blocking send or receive operations, the process needs to have a way to make sure that the data has been sent or some data has arrived before proceeding...
In this webpage, we will look at (the easier) blocking send and receive MPI operation first
In the next webpage, we will study the (more difficult) NON-blocking send and receive.

Point-to-point communication primitives

Recall that the MPI point-to-point blocking communication primitives are:

Function Name	Usage
MPI_Send(void *buff, int count, MPI_Datatype type, int dest, int tag, int comm)	Send a point-to-point message to process dest in the communication group comm The message is stored at memory location buff and consists of count items of datatype type The message will be tagged with the tag value tag The MPI_Send() function will only return if the message sent has been received by the destination. (It's safe to reuse the buffer buff right away).
MPI_Recv(void buff, int count, MPI_Datatype type, int source, int tag, int comm, MPI_Status status)	Receive a point-to-point message The message MUST BE from the process source in the communication group comm AND the message MUST BE tagged with the tag value tag The message received will be stored at memory location buff which will have space to store count items of datatype type When the function exits, the exit status is stored in status. Information about the received message is returned in a status variable, e.g.,: The received message tag is status.MPI_TAG and The rank of the sending process (of the message) is status.MPI_SOURCE. In most cases, you know the structure of data received and then you can ignore the status value... If you pass NULL as status parameter, MPI will not return the status value. The MPI_Recv() function will only return if the desired message (from source with tag tag) has been received - or exits with an error code...

Function Name

Usage

MPI_Send(void *buff,
     int count,
     MPI_Datatype type,
     int dest,
     int tag,
     int comm)

Send a point-to-point message to process dest in the communication group comm

The message is stored at memory location buff and consists of count items of datatype type

The message will be tagged with the tag value tag

The MPI_Send() function will only return if the message sent has been received by the destination. (It's safe to reuse the buffer buff right away).

MPI_Recv(void *buff,
     int count,
     MPI_Datatype type,
     int source,
     int tag,
     int comm,
     MPI_Status *status)

Receive a point-to-point message

The message MUST BE from the process source in the communication group comm AND the message MUST BE tagged with the tag value tag

The message received will be stored at memory location buff which will have space to store count items of datatype type

When the function exits, the exit status is stored in status.

Information about the received message is returned in a status variable, e.g.,:

The received message tag is status.MPI_TAG and
The rank of the sending process (of the message) is status.MPI_SOURCE.

In most cases, you know the structure of data received and then you can ignore the status value... If you pass NULL as status parameter, MPI will not return the status value.

The MPI_Recv() function will only return if the desired message (from source with tag tag) has been received - or exits with an error code...

Be careful that:

The MPI_Recv() function is a selective receive function.
It will ONLY receive (and return) a message from the matching source ID and with the matching tag
If you want to receive a message from any source , then use the value:
for "source"
If you want to receive a message with any tag value, then use the value
for "tag"

Examples of point-to-point communication in MPI

Example MPI point-to-point communication:

#include "mpi.h" int main(int argc, char **argv) { char reply[100]; char buff[128]; int numprocs; int myid; int i; MPI_Status stat; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); /* ----------------------------------------- Master process ----------------------------------------- */ if(myid == 0) { printf("WE have %d processors\n", numprocs); /* ----------------------------------------- Master process: send msg with tag 1234 ----------------------------------------- */ for(i=1;i < numprocs;i++) { sprintf(buff, "Hello %d", i); MPI_Send(buff, 128, MPI_CHAR, i, 1234, MPI_COMM_WORLD); } /* --------------------------------------------- Master process: wait for msg with tag 4444 --------------------------------------------- */ for(i=1;i < numprocs;i++) { MPI_Recv(buff, 128, MPI_CHAR, i, 4444, MPI_COMM_WORLD, &stat); cout << buff << endl; } } else /* ----------------------------------------- Slave process: receive msg with tag 1234 ----------------------------------------- */ { /* ----------------------------------------- Slave process: receive msg with tag 1234 ----------------------------------------- */ MPI_Recv(buff, 128, MPI_CHAR, 0, 1234, MPI_COMM_WORLD, &stat); sprintf(reply, " |--> Hello 0, Processor %d is present and accounted for !", myid); strcat(buff, reply); /* -------------------------------------------- Slave process: send back msg with tag 4444 -------------------------------------------- */ MPI_Send(buff, 128, MPI_CHAR, 0, 4444, MPI_COMM_WORLD); } MPI_Finalize(); }

Example Program:

Prog file: click here

To compile the program:

ssh2 -X rockscluster
cd cs561
mpiCC -o Comm01 Comm01.C

To run the program:

mpirun -np 4 ./Comm01

Experiment:

Change sending tag to 1111 and recompile and run
The receiver will NOT receive the message
Change receive tag to MPI_ANY_TAG and recompile and run
The receiver will receive the message

The MPI_Status data structure
- The MPI_Status data structure is used by MPI mesage receiving functions to return information about a message RECEIVED.
- The MPI_Status type contains at least the following fields:
- There may also be other fields in the stucture, but these are reserved for the implementation and you are not supposed to access them directly.
Sending/Receiving SIMPLE data types in MPI
- The MPI communication primitives allow you to send simple data types without much effort
- To send N items of the type TYPE stored at location buff, use the call:
  Effect of the function:
- Use the following pre-defined MPI symbolic constants to indicate the most commonly used C/C++ type:
- To receive N items of the type TYPE stored at location buff, use the call:
  Effect of the function:

Name of field	Usage
`MPI_SOURCE`	id of processor sending the message (integer)
`MPI_TAG`	tag of the message (integer)
`MPI_ERROR`	error code (integer)

C/C++ Type	MPI symbolic constant
`char`	`MPI_CHAR`
`int`	`MPI_INT`
`float`	`MPI_FLOAT`
`double`	`MPI_DOUBLE`

Example: sending and receiving when types are different

Process 1 sends 4 characters to process 0

Process 0 receives an integer (4 bytes)

int main(int argc, char **argv) { char in[4]; // Send 4 characters int out; // Interprete as an integer int numprocs; int myid; int i; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); if(myid == 0) { cout << "We have " << numprocs << " processors" << endl; MPI_Recv(&out, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, NULL); cout << "Received this number from proc 1: " << out << endl; } else if ( myid == 1 ) { in[0] = '2'; in[1] = 1; in[2] = 0; in[3] = 0; MPI_Send(in, 4, MPI_CHAR, 0, 0, MPI_COMM_WORLD); } MPI_Finalize(); }

Result:

Program will print (256+50 = 306)

Because:

ASCII code for '2' is 50, weight of digit = 1 => 1 x 50 = 50
Binary value 1, weight of digit = 256 => 1 x 256 = 256
Binary value 0, weight of digit = 256*256 => 0 x 256*256 = 0
Binary value 0, weight of digit = 256*256*256 => 0 x 256*256*256 = 0
Total = 256+50 = 306

(Note: puma is a Intel based machine and uses little endian storage, that's why we need to put the '2' in the first byte)

Example Program: (Demo above code)
- Prog file: click here
To compile the program:
To run the program:

Cooperation to complete a common task: COmpute PI in MPI programming

Consider the problem of computing the value of Pi by numerical integration

Paralel Algorithm:

The computational labor is subdivided into N portions
Each processor (inclusing the master) will compute its own (partial) sum.
Each slave processor send partial sum to master processor (0)
Then at the end, the master (processor 0) collects all (partial) sums and forms a total sum (Pi).

MPI program to compute Pi by integration:

double f(double a) { return( 2.0 / sqrt(1 - a*a) ); } int main(int argc, char *argv[]) { int N; // Number of intervals double w, x; // width and x point int i, myid; double mypi, others_pi; MPI_Init(&argc,&argv); // Initialize MPI_Comm_size(MPI_COMM_WORLD, &num_procs); // Get # processors MPI_Comm_rank(MPI_COMM_WORLD, &myid); N = atoi(argv[1]); w = 1.0/(double) N; mypi = 0.0; for (i = myid; i < N; i = i + num_procs) { x = w*(i + 0.5); mypi = mypi + w*f(x); } /* ---------------------------------------------------- Proc 0 collects and others send data to proc 0 ---------------------------------------------------- */ if ( myid == 0 ) { for (i = 1; i < num_procs; i++) { MPI_Recv(&others_pi, 1, MPI_DOUBLE, i, 0, MPI_COMM_WORLD, NULL); mypi += others_pi; } cout << "Pi = " << mypi<< endl << endl; } else { MPI_Send(&mypi, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD); } MPI_Finalize(); }

NOTE:
Example Program: (Demo above code)
- Prog file: click here
To compile the program:
To run the program:

Advanced communication: unknown sender
- The MPI_Recv() requires the user to specify a specific sender ID and a specific tag:
- If the programmer does not know in advance what values source and tag the message will be, you can use the wildcard values to receive the message, and then use the status variable to find out who the sender was:
- After the MPI_Recv() returns with a message, you can use these fields to find the source id of the sender and the tag of the message:
- NOTE: there is still one thing you must know to use MPI_Recv():

Advanced communication: unknown message length

If you do not know the length of the data, then you cannot use the MPI_Recv() function (because you must specify a buffer variable and the size of the buffer may be too small)
Probing message arrivals:
The MPI_Probe() function will block a process until a message with matching source and matching tag has been received.
The MPI_Probe() function will not receive the message.
Instead, it initializes a status variable containing all information about the pending message

To receive a message of unknown length, we must proceed in 3 steps:

Wait for a message to arrive and find out the SIZE of the message (using MPI_Probe())
Find the length (number of bytes) of the message. (using MPI_Get_count())
Allocate a buffer of sufficient size to receive the message (using MPI_Get_count())

Example:

MPI_Status status; int nbytes; MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &status); // Allocate memory to receive data MPI_Get_count(&status, MPI_CHAR, &nbytes); if ( nbytes != MPI_UNDEFINED ) buff = malloc( nbytes ); // Allocate buffer to receive message MPI_Recv(buff, nbytes, MPI_CHAR, status.SOURCE, status.TAG, MPI_COMM_WORLD, &status);

Example Program: (Demo above code)
- Prog file: click here
To compile the program:
To run the program:

Final Note
- If you want to transmit data that have arbitrary format , then you must transmit META DATA along with the actual data
- This approach is similar to the meta data used in Database Systems
- For an in dept study of meta data , please see: click here