Python IPC, popen too slow
- by UnableToLoad
i need to run a subprocess (./myProgram) form python script and get output, actually i do this:
import subprocess
proc = subprocess.Popen('./generate_out',
shell=False,
stdout=subprocess.PIPE,
)
while proc.poll() is None:
out = proc.stdout.readline()
data = doStuff(out)
print(data)
but is slow, sometimes pass a lot of time between the output produced by ./generate_out and the print(data), knowing that my doStuff() function is very fast, i think there is some buffer slowing down my pipe...
Notes:
./generate_out, generates potentially an unlimited number of lines of finite length each.
It seems that when too few chars are put in the pipe between the two processes nothing happens, then when enough is produced i get a huge print (non the expected behaviour!)
sometimes i wait many seconds (10-20 and more) between generate_out print and python print)
what can i do? maybe communicate() is faster? anithing else?
Thank you a lot!
