Read non-blocking from multiple fifos in parallel
- by Ole Tange
I sometimes sit with a bunch of output fifos from programs that run in parallel. I would like to merge these fifos. The naïve solution is:
cat fifo* > output
But this requires the first fifo to complete before reading the first byte from the second fifo, and this will block the parallel running programs.
Another way is:
(cat fifo1 & cat fifo2 & ... ) > output
But this may mix the output thus getting half-lines in output.
When reading from multiple fifos, there must be some rules for merging the files. Typically doing it on a line by line basis is enough for me, so I am looking for something that does:
parallel_non_blocking_cat fifo* > output
which will read from all fifos in parallel and merge the output on with a full line at a time.
I can see it is not hard to write that program. All you need to do is:
open all fifos
do a blocking select on all of them
read nonblocking from the fifo which has data into the buffer for that fifo
if the buffer contains a full line (or record) then print out the line
if all fifos are closed/eof: exit
goto 2
So my question is not: can it be done?
My question is: Is it done already and can I just install a tool that does this?
