Long running, polling, queueing process for Python. What's the best stuff to use?
- by Bialecki
Feel free to close and/or redirect if this has been asked, but here's my situation:
I've got an application that will require doing a bunch of small units of work (polling a web service until something is done, then parsing about 1MB worth of XML and putting it in a database). I want to have a simple async queueing mechanism that'll poll for work to do in a queue, execute the units of work that need to be done, and have the flexibility to allow for spawning multiple worker processes so these units of work can be done in parallel. (Bonus if there's some kind of event framework that would also me to listen for when work is complete.)
I'm sure there is stuff to do this. Am I describing Twisted? I poked through the documentation, I'm just not sure exactly how my problems maps onto their framework, but I haven't spent much time with it. Should I just look at the multiprocess libraries in Python? Something else?
