Slashdot Mirror
Looking for Portable MPI I/O Implementation?
rikt writes "I am trying to implement MPI I/O for our CFD product. I am facing a problem with the portability of the generated data files. MPI2 interface describes a way to achieve this either by using 'external32' or user defined data representations. The problem is that ROMIO, the most widely available MPI I/O implementation, has not implemented support for any data representation other than 'native'. Do you know of any MPI I/O implementation that supports this, and is available on various platforms? I know IBM and Sun supports this, but I am looking for a solution on Linux and Windows (both 32 & 64 bit) as well."
Just explain a few of the TLAs, mmkay?
Am I the only one who thinks this is way too specific a question for Ask Slashdot, and that the subject at hand is not one that most people would know offhand? This seems much more appropriate to a topic-specific newsgroup or mailing list, not the general tech community.
ttuttle is a rankmaniac
I'm a geek who does administration and programming in Windows and Linux realms, am fairly aware of my acronym soup and yet this left me, um, cold. For those who don't feel like doing the research:
MPI: Message Passing Interface, a standard for parallel processing environment message passing.
MPI-2: Extended version of MPI.
MPI-IO: Parallel input/output extensions for MPI, included in MPI-2
ROMIO: An implementation of these extensions.
CFD: Computational Fluid Dynamics (a good candidate for parallel processing, thus the interest in the above).
Of course, the fact I had to look them up means I have no idea about implementations, but at least others won't have to wonder what all that was about.
Sig under construction since 1998.
Think of it this way.
typedef struct MessageStruct {
int messageType;
int messageSize;
} Message;
Message msg;
msg.type = messageType;
msg.size = sizeof(Message);
msg.data = someData; /* repeat for each part of the message struct*/
SendMessage(&msg, msg.size);
MPIMessage msg;
if (msg.size == sizeof(Message)) {
Message *msg = &msg;
}
I think something along these lines should work. Just make a struct for each type of message your app has. Then check the size and type elements of the structs to determine which type of message you have recieved. You can also just make a struct with just a type and size field and copy the first 8 bytes of the message into that and use that to determine the type of message. I'm sure I am missing some implementation details, but something like this should handle your problem.
"Those that start by burning books, will end by burning men."
I take it you've aleady read section 7.5 in MPIv2. If you haven't, now's the time!
Unfortunately, I know of no other MPI I/O implementations, other than ROMIO, that can simply be plugged into an existing MPI stack. You might want to ask around at the new project OpenMPI, a new-from-the-ground-up MPI implementation that is currently in development. I'd be curious to learn the level of MPI I/O support that they claim!
Assuming you are stuck with a MPI stack that only supports the "native" representation, the problem you face becomes one of data representation in general. As you know, there's bajillions of different ways of storing floating-point numbers, and if you write them to disk, the files will be only valid for exactly that CPU.
As a last resort, a brute-force solution is to write the numbers as human-readable text, and then parse them in again accoringly. It's a waste of file space, but there's no ambiguity in the datatype representation, and it is very tolerant of floating point differences between machines.
-1.2345234523452345
2.345634563456365e+13
-3.2121212121e-24
And so on.
This shouldn't be much of a hotspot in your code, since ideally it would only be done at start, stop, and checkpoint time. Also, if you need paralellism, and don't care about wasted file space or future precision improvements, you could use a fixed-length string for each number (with much padding), thus allowing you to read your numbers random-access instead of sequential.
Hope this helps!
Josh
Dr. Demento On The 'Net!
Now, we move onto the portable I/O. The vast majority of scientific software (which is, in turn, the bulk of MPI-based software) uses the Heirarchical Data Format. There are two versions worthy of mention - HDF5 and Parallel HDF. Both support MPI in operations. Compile HDF5 with MPI support, and you have something that will support platform-independent atomic and compound data types.
Of all the options, HDF5 (from the NCSA) is the most widely used. I would say that the majority of scientific and distributed software out there that uses platform-independent typing uses HDF. So does the grid computing system Globus. The other platform-independent complex data typing libraries, CDF (from NASA) and NetCDF (from UniData), are rarely used. Indeed, the next generation of NetCDF - version 4 - will be built on top of HDF5. There's a link to the development site and the source code on Freshmeat.
Less-widely used, but still very significant, is the Transparent Parallel I/O Environment. I am not 100% sure if this supports MPI, it's been a while since I've used it and I never put in the dependencies on Freshmeat for it.
Depending on what is being done, PETSc may also be worth checking out. This supports MPI-based differential equations.
Globus can use MPI for communication and then handle the I/O directly. This means you only have to write your interface for one API, not one API per type of operation. Main problem is that Globus has a fairly large footprint, so you might not want to do that unless the project is large enough to warrant that kind of sophistication.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
I was going to suggest string representations, too... I am working an MPI project that deals with passing a lot of stuff around and found that the method of structure passing in MPI caused us to have to represent the structure specifically byte-by-byte anyway, so we have just stuck with doing everything as character arrays in specific formats...
The main benefit for us was that our message passing code became generic and we got the side effect of passing large values between machines without respect for endianess or word size.
hope that helps,
dave
Have you tried reversing the polarity?
"It's too bad that stupidity isn't painful." - Anton LaVey
http://www.verarisoft.com/
I work there and I worked on our MPI-IO implementation. I'm sure we'd like to find a way to help you out if you aren't against paying for the software.
you just got added to my friends list, and I just got added to your fans.
Filled a row on my Buzzword Bingo card in just the summary, didn't even need to follow the links.
Thanks for the definitions, now the summary makes a little more sense.
The living have better things to do than to continue hating the dead.
- We properly wrapped it such that I/O requests are of type MPI_Request, not MPIO_Request. Hence, you can actually progress IO requests, generalized requests, and point-to-point requests in a single MPI_WAITALL (or MPI_TESTALL, or any of the other variants)
- Our MPI-2 IO support is based on a component framework -- so replacing ROMIO is not only easy, it's encouraged! We had always intended ROMIO to be a stopgap soltuon until we could implement "something better" (as yet to be defined). We would love to have someone with expertise in this area to a) help define what a "better" component interface should be (our ROMIO interface is a simple one-to-one function mapping), and b) write one or more components to implement this in a generic and/or proprietary way.
That's a long way of saying: "E-mail me and let's talk."I think his post is a little off the mark, but jd (1658) has the right idea -- if you are concerned about portable representation of your data, you might want to use a higher-level library. HDF5 and NetCDF are good choices.
Even beter might be Parallel-NetCDF. It has all the benefits of a high-level library (portable, self-describing data representation), but it has a much simpler interface than HDF5. Unlike serial NetCDF, you'll probably see much better performance as all processes can carry out I/O collectively instead of forwarding to a master .
Strings are fine for mashalling/unmarshalling numbers if the remote computation is very long compared to the time it takes to parse/encode the numbers. But if your remote computation is relatively short lived then you could waste 90% of your CPU resources. Binary is the way to go if you can.
Nobody has mentioned XDR yet. Although it's probably not what you'd really prefer (as NetCDF seems to be very popular with CFD projects, and there's presumably a good reason), I'll describe it for completeness. EXternal Data Represenation it sits nicely in the midpoint between pure native data and human-readable text of the numbers (including XML).
Free libxdr code is available everyplace, although often quite ancient (some written in 1982 or so). Just run your data structs through xdr calls, write it out with the MPI native interface, and apply reverse versions of the same calls when loading. It's a lot like the htonl/ntohl functions you may be familiar with (but more elaborate, of course, because it handles marshalling and padding in addition to just endianness).
Advantage of XDR: files are portable, but hardly any bigger than the original data in RAM (prehaps even a little smaller)
Disadvantage of XDR: files are not human-readable, decellerating the processes of debugging or informing others how to inspect your files. Plus, it simply lacks the cool-factor of XML.
http://www.faqs.org/rfcs/rfc1832.html
Hello friends Thank you all for your responses. I have looked into your suggestions. As for HDF5, I agree that would be a good one, but due to some other constraints, I had to choose MPI. Now, the question is, If I can't get a portable MPI I/O implementation, what can I do to make it portable.The software I am working upon, can do it by storing the macine information in the file and then having its own converter [for serial I/O ]. I wanted to avoid all that task and was relying on MPI interface to do the job. [ That is why the external32 representation. ] I am sorry if I am posting it too late, coz I get access to net only from my office.