Communicate between separate MPI-Programs

I have the following problem:

Program 1 has a huge amount of data, say 10GB. The data in question consists of la开发者_开发技巧rge integer- and double-arrays. Program 2 has 1..n MPI processes that use tiles of this data to compute results.

How can I send the data from program 1 to the MPI Processes?

Using File I/O is out of question. The compute node has sufficient RAM.

---

It should be possible, depending on your MPI implementation, to run several different programs in the same MPI job. For instance, using OpenMPI you can run

 mpirun -n 1 big_program : -n 20 little_program

and you should be able to access both programs using MPI_COMM_WORLD. From there you'd then be able to use the usual MPI functions to pass your data from the big program to the little ones.

---

One answer might be to have the two programs reside in separate communicators; a single executable could launch both sets of apps by utilizing MPI-2's dynamic process management, and the "producer" program communicate through MPI_COMM_WORLD to the "consumer" application. Subsequently, all IPC for the consumer app would have to run inside a subcommunicator that excluded the producer portion. This would mean rewriting to avoid direct calls to MPI_COMM_WORLD, however.

---

Based on your description "Program 1" is not an MPI application, and "Program 2" is an MPI application. The shortest path to a solution is likely to open a socket between the two programs and send the data that way. This does not require that "Program 1" be modified to be an MPI Program. I would begin with a socket between "Program 1" and " Program 2 : Rank 0", with Rank 0 distributing the data to the remaining ranks.

Several suggestions so far have involved launching a heterogeneous set of executables as one possible solution. There is no requirement that all the ranks in a single MPI job be the same executable. This requires that both executables be "MPI Programs" (e.g. include at least MPI_Init, and MPI_Finalize calls). The level of modification required to "Program 1", and the inability to run it outside of the MPI environment, may make this option unattractive.

I would recommend that you avoid the "dynamic process" approach, unless you are using a commercial implementation that offers support. Support for connect/accept tends to be spotty in the open source implementations of MPI. It may "just work", but getting technical help if it does not can be an open ended problem.

---

It is not a good idea to mix sockets & MPI. The easiest way to achieve this is moving both process 1 & process 2 in to a single MPI application.

The best way to implement this is to use a programming model called - MPMD or Multi-Program Multi-Data. As the name implies, your MPI application will have multiple programs operating on multiples of data. Even if Program 1 is not an MPI application, you need not make too many changes. Just call MPI_Init & add the routines for send/recv data. You can think of this as a sort of a Master-Slave model where Prg1 is master and rest are slaves getting pieces of data to work on from the master.

Another method could be to implement a pool of workers by making program 1 same as program 2 and everyone reads a part of the data file and starts working. But you ruled out file IO, so I assume prog2-n do not have access to the file at run time. Master-Slave will work best for your needs.