Queue remote calls to a Python Twisted perspective broker?

Posted by agartland on Stack Overflow See other posts from Stack Overflow or by agartland
Published on 2010-05-18T23:22:21Z Indexed on 2010/05/19 2:20 UTC
Read the original article Hit count: 303

Filed under:

python

|

twisted

|

asynchronous

The strength of Twisted (for python) is its asynchronous framework (I think). I've written an image processing server that takes requests via Perspective Broker. It works great as long as I feed it less than a couple hundred images at a time. However, sometimes it gets spiked with hundreds of images at virtually the same time. Because it tries to process them all concurrently the server crashes.

As a solution I'd like to queue up the remote_calls on the server so that it only processes ~100 images at a time. It seems like this might be something that Twisted already does, but I can't seem to find it. Any ideas on how to start implementing this? A point in the right direction? Thanks!

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about twisted

Has Twisted changed its dependencies?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I'm currently working on a Python/Twisted project which is to be distributed and tested on Planetlab. For some reason my code was working on friday and now that I wanted to test a minor change it refuses to work at all: Traceback (most recent call last): File "acn_a4/src/node.py", line… >>> More
Python (Twisted) - reading from fifo and sending read data to multiple protocols

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, Im trying to write some kind of multi protocol bot (jabber/irc) that would read messages from fifo file (one liners mostly) and then send them to irc channel and jabber contacts. So far, I managed to create two factories to connect to jabber and irc, and they seem to be working. However, I've… >>> More
Start Python from Twisted

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I have learnt Python for about a month as a one year's PHPer.And I started from Twisted as I'm working in a corporation supplying webservice.I have finished some simple application such as data transferring service,page images-fetch service etc.But the problem is ,I don't understand the struture… >>> More
Python twisted Reactor class

as seen on Stack Overflow - Search for 'Stack Overflow'
What is the significance of the decorators @reactor.callWhenRunning, @results_deferred.addCallback @results_deferred.addErrback. Also what are deferred strings, for example in the twisted.internet.utils.getProcessOutput() returns a deferred string what exactly is happening here? I am new… >>> More
How to step through debug twisted?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'd like to be able to debug Punjab, a twisted python application, in Netbeans so that I can step through the code. How can I do that? Alternatively, how could I do it in a different debugger? >>> More