Minimal "Task Queue" with stock Linux tools to leverage Multicore CPU
        Posted  
        
            by Manuel
        on Stack Overflow
        
        See other posts from Stack Overflow
        
            or by Manuel
        
        
        
        Published on 2009-05-06T23:14:57Z
        Indexed on 
            2010/06/10
            20:13 UTC
        
        
        Read the original article
        Hit count: 344
        
What is the best/easiest way to build a minimal task queue system for Linux using bash and common tools?
I have a file with 9'000 lines, each line has a bash command line, the commands are completely independent.
command 1 > Logs/1.log
command 2 > Logs/2.log
command 3 > Logs/3.log
...
My box has more than one core and I want to execute X tasks at the same time. I searched the web for a good way to do this. Apparently, a lot of people have this problem but nobody has a good solution so far.
It would be nice if the solution had the following features:
- can interpret more than one command (e.g. command; command)
- can interpret stream redirects on the lines (e.g. ls > /tmp/ls.txt)
- only uses common Linux tools
Bonus points if it works on other Unix-clones without too exotic requirements.
© Stack Overflow or respective owner