Minimal "Task Queue" with stock Linux tools to leverage Multicore CPU
- by Manuel
What is the best/easiest way to build a minimal task queue system for Linux using bash and common tools?
I have a file with 9'000 lines, each line has a bash command line, the commands are completely independent.
command 1 > Logs/1.log
command 2 > Logs/2.log
command 3 > Logs/3.log
...
My box has more than one core and I want to execute X tasks at the same time. I searched the web for a good way to do this. Apparently, a lot of people have this problem but nobody has a good solution so far.
It would be nice if the solution had the following features:
can interpret more than one command (e.g. command; command)
can interpret stream redirects on the lines (e.g. ls > /tmp/ls.txt)
only uses common Linux tools
Bonus points if it works on other Unix-clones without too exotic requirements.
