MPI: is there mpi libraries capable of message compression?

Posted by osgx on Stack Overflow See other posts from Stack Overflow or by osgx
Published on 2012-06-01T12:08:20Z Indexed on 2012/12/16 11:06 UTC
Read the original article Hit count: 363

Filed under:

compression

|

mpi

Sometimes MPI is used to send low-entropy data in messages. So it can be useful to try to compress messages before sending it. I know that MPI can work on very fast networks (10 Gbit/s and more), but many MPI programs are used with cheap network like 0,1G or 1Gbit/s Ethernet and with cheap (slow, low bisection) network switch. There is a very fast Snappy (wikipedia) compression algorithm, which has

Compression speed is 250 MB/s and decompression speed is 500 MB/s

so on compressible data and slow network it will give some speedup.

Is there any MPI library which can compress MPI messages (at layer of MPI; not the compression of ip packets like in PPP).

MPI messages are also structured, so there can be some special method, like compression of exponent part in array of double.

© Stack Overflow or respective owner

Related posts about compression

Converting linear colors to SRGB shows banding in FFmpeg

as seen on Super User - Search for 'Super User'
When I convert an EXR file sequence with x264 using FFmpeg and convert the colorspace from linear to SRGB (with gamma 0.45454545) I get some heavy banding issues (most visible on a dark gradient). Here is the ffmpeg command I use: C:/ffmpeg.exe -y -i C:/seq_v001.%04d.exr -vf lutrgb=r=gammaval(0… >>> More
IIS7 Compression CSS files only compressed when dynamic compression is enabled

as seen on Server Fault - Search for 'Server Fault'
If anyone can help it would be appreciated. I would like to enable compression for static files within IIS7 (for the sake of simplicity I'll just refer to static css files for the time being). The problem I'm getting is that css files are only compressed when both dynamic and static compression… >>> More
Built-in GZip/Deflate Compression on IIS 7.x

as seen on West-Wind - Search for 'West-Wind'
IIS 7 improves internal compression functionality dramatically making it much easier than previous versions to take advantage of compression that’s built-in to the Web server. IIS 7 also supports dynamic compression which allows automatic compression of content created in your own applications (ASP… >>> More
Domino Document data compression and design compression

as seen on Server Fault - Search for 'Server Fault'
I was thinking of turning this on some large databases not just mail files - we have around 8 - 10GB of large databases as well as small databases of couple of hundred MB in size. But after reading this post I am not too sure: http://www-10.lotus.com/ldd/nd85forum.nsf/4b9931b774db788c85256bf0006b5e6d/1f4e67b569720e54852576c0003cb8ac… >>> More
I need to choose a compression algorithm

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to choose a compression algorithm to compress some data. I don't know the type of data I'll be compressing in advance (think of it as kinda like the WinRAR program). I've heard of the following algorithms but I don't know which one I should use. Can anyone post a short list of pros and cons… >>> More

Related posts about mpi

Message Passing Interface (MPI)

as seen on Daniel Moth - Search for 'Daniel Moth'
So you have installed your cluster and you are done with introductory material on Windows HPC. Now you want to develop an application with the most common programming model: Message Passing Interface.The MPI programming model is a standard with implementations from many vendors. For newbies (like… >>> More
How do I debug an MPI program?

as seen on Stack Overflow - Search for 'Stack Overflow'
I have an MPI program which compiles and runs, but I would like to step through it to make sure nothing bizarre is happening. Ideally, I would like a simple way to attach GDB to any particular process, but I'm not really sure whether that's possible or how to do it. An alternative would be having… >>> More
trying to build Boost MPI, but the lib files are not created. What's going on?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to run a program with Boost MPI, but the thing is I don't have the .lib. So I try to create one by following the instruction at http://www.boost.org/doc/libs/1_43_0/doc/html/mpi/getting_started.html#mpi.config The instruction says "For many users using LAM/MPI, MPICH, or OpenMPI, configuration… >>> More
MPI: is there mpi libraries capable of message compression?

as seen on Stack Overflow - Search for 'Stack Overflow'
Sometimes MPI is used to send low-entropy data in messages. So it can be useful to try to compress messages before sending it. I know that MPI can work on very fast networks (10 Gbit/s and more), but many MPI programs are used with cheap network like 0,1G or 1Gbit/s Ethernet and with cheap (slow,… >>> More
Slides and code for MPI Cluster Debugger

as seen on Daniel Moth - Search for 'Daniel Moth'
I've blogged before about the MPI Cluster Debugger in VS2010 that facilitates launching the application on the cluster and attaching the debugger (btw, a shorter version of the screencast I link to there, is here).There have been requests for the code I use in the screencast, so please find a ZIP… >>> More