Advice: Python Framework Server/Worker Queue management (not Website)

Posted by Muppet Geoff on Stack Overflow See other posts from Stack Overflow or by Muppet Geoff
Published on 2011-11-21T22:22:33Z Indexed on 2011/11/22 1:50 UTC
Read the original article Hit count: 186

Filed under:
|
|
|

I am looking for some advice/opinions of which Python Framework to use in an implementation of multiple 'Worker' PCs co-ordinated from a central Queue Manager.

For completeness, the 'Worker' PCs will be running Audio Conversion routines (which I do not need advice on, and have standalone code that works).

The Audio conversion takes a long time, and I need to co-ordinate an arbitrary number of the 'Workers' from a central location, handing them conversion tasks (such as where to get the source files, or where to ask for the job configuration) with them reporting back some additional info, such as the runtime of the converted audio etc.

At present, I have a script that makes a webservice call to get the 'configuration' for a conversion task, based on source files located on the worker already (we manually copy the source files to the worker, and that triggers a conversion routine). I want to change this, so that we can distribute conversion tasks ("Oy you, process this: xxx") based on availability, and in an ideal world, based on pending tasks too.

There is a chance that Workers can go offline mid-conversion (but this is not likely).

All the workers are Windows based, the co-ordinator can be WIndows or Linux.

I have (in my initial searches) come across the following - and I know that some are cross-dependent:

Celery (with RabbitMQ) Twisted Django

Using a framework, rather than home-brewing, seems to make more sense to me right now. I have a limited timeframe in which to develop this functional extension.

An additional consideration would be using a Framework that is compatible with PyQT/PySide so that I can write a simple UI to display Queue status etc.

I appreciate that the specifics above are a little vague, and I hope that someone can offer me a pointer or two.

Again: I am looking for general advice on which Python framework to investigate further, for developing a Server/Worker 'Queue management' solution, for non-web activities (this is why DJango didn't seem the right fit).

© Stack Overflow or respective owner

Related posts about python

Related posts about message-queue