Logging in worker threads spawned from a pylons application does not seem to work
- by TimM
I have a pylons application where, under certain cirumstances I want to spawn multiple worker threads to process items in a queue. Right now we aren't making use of a ThreadPool (would be ideal, but we'll add that in later). The main problem is that the worker threads logging does not get written to the log files.
When I run the code outside of the pylons application the logging works fine. So I think its something to do with the pylons log handler but not sure what.
Here is a basic example of the code (trimmed down):
import logging
log = logging.getLogger(__name__)
import sys
from Queue import Queue
from threading import Thread, activeCount
def run(input, worker, args = None, simulteneousWorkerLimit = None):
queue = Queue()
threads = []
if args is not None:
if len(args) > 0:
args = list(args)
args = [worker, queue] + args
args = tuple(args)
else:
args = (worker, queue)
# start threads
for i in range(4):
t = Thread(target = __thread, args = args)
t.daemon = True
t.start()
threads.append(t)
# add ThreadTermSignal
inputData = list(input)
inputData.extend([ThreadTermSignal] * 4)
# put in the queue
for data in inputData:
queue.put(data)
# block until all contents are downloaded
queue.join()
log.critical("** A log line that appears fine **")
del queue
for thread in threads:
del thread
del threads
class ThreadTermSignal(object):
pass
def __thread(worker, queue, *args):
try:
while True:
data = queue.get()
if data is ThreadTermSignal:
sys.exit()
try:
log.critical("** I don't appear when run under pylons **")
finally:
queue.task_done()
except SystemExit:
queue.task_done()
pass
Take note, that the log lin within the RUN method will show up in the log files, but the log line within the worker method (which is run in a spawned thread), does not appear. Any help would be appreciated. Thanks
** EDIT: I should mention that I tried passing in the "log" variable to the worker thread as well as redefining a new "log" variable within the thread and neither worked.
** EDIT: Adding the configuration used for the pylons application (which comes out of the INI file). So the snippet below is from the INI file.
[loggers]
keys = root
[handlers]
keys = wsgierrors
[formatters]
keys = generic
[logger_root]
level = WARNING
handlers = wsgierrors
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = WARNING
formatter = generic
[handler_wsgierrors]
class = pylons.log.WSGIErrorsHandler
args = ()
level = WARNING
format = generic
