I'm trying to answer the following question out of personal interest:
What is the fastest way to send 100,000 HTTP requests in Python?
And this is what I have came up so far, but I'm experiencing something very stange.
When installSignalHandlers is True, it just hangs. I can see that the DelayedCall instances are in reactor._newTimedCalls, but processResponse never gets called.
When installSignalHandlers is False, it throws an error and works.
from twisted.internet import reactor
from twisted.web.client import Agent
from threading import Semaphore, Thread
import time
concurrent = 100
s = Semaphore(concurrent)
reactor.suggestThreadPoolSize(concurrent)
t=Thread(
target=reactor.run,
kwargs={'installSignalHandlers':True})
t.daemon=True
t.start()
agent = Agent(reactor)
def processResponse(response,url):
print response.code, url
s.release()
def processError(response,url):
print "error", url
s.release()
def addTask(url):
req = agent.request('HEAD', url)
req.addCallback(processResponse, url)
req.addErrback(processError, url)
for url in open('urllist.txt'):
addTask(url.strip())
s.acquire()
while s._Semaphore__value!=concurrent:
time.sleep(0.1)
reactor.stop()
And here is the error that it throws when installSignalHandlers is True:
(Note: This is the expected behaviour! The question is why it doesn't work when installSignalHandlers is False.)
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 396, in fireEvent
DeferredList(beforeResults).addCallback(self._continueFiring)
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 224, in addCallback
callbackKeywords=kw)
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 213, in addCallbacks
self._runCallbacks()
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 371, in _runCallbacks
self.result = callback(self.result, *args, **kw)
--- <exception caught here> ---
File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 409, in _continueFiring
callable(*args, **kwargs)
File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1165, in _reallyStartRunning
self._handleSignals()
File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1105, in _handleSignals
signal.signal(signal.SIGINT, self.sigInt)
exceptions.ValueError: signal only works in main thread
What am I doing wrong and what is the right way? I'm new to twisted.
