deciding between subprocess, multiprocesser and thread in Python?

Posted by user248237 on Stack Overflow See other posts from Stack Overflow or by user248237
Published on 2010-04-13T13:01:31Z Indexed on 2010/04/13 13:02 UTC
Read the original article Hit count: 341

I'd like to parallelize my Python program so that it can make use of multiple processors on the machine that it runs on. My parallelization is very simple, in that all the parallel "threads" of the program are independent and write their output to separate files. I don't need the threads to exchange information but it is imperative that I know when the threads finish since some steps of my pipeline depend on their output.

Portability is important, in that I'd like this to run on any Python version on Mac, Linux and Windows. Given these constraints, which is the most appropriate Python module for implementing this? I am tryign to decide between thread, subprocess and multiprocessing, which all seem to provide related functionality.

Any thoughts on this? I'd like the simplest solution that's portable.

Thanks.

© Stack Overflow or respective owner

Related posts about python

Related posts about multithreading