Dynamically loading modules in Python (+ threading question)
- by morpheous
I am writing a Python package which reads the list of modules (along with ancillary data) from a configuration file.
I then want to iterate through each of the dynamically loaded modules and invoke a do_work() function in it which will spawn a new thread, so that the code runs in a separate thread.
At the moment, I am importing the list of all known modules at the beginning of my main script - this is a nasty hack I feel, and is not very flexible, as well as being a maintenance pain.
This is the function that spawns the threads. I will like to modify it to dynamically load the module when it is encountered. The key in the dictionary is the name of the module containing the code:
def do_work(work_info):
for (worker, dataset) in work_info.items():
#import the module defined by variable worker here...
t = threading.Thread(target=worker.do_work, args=[dataset])
# I'll NOT dameonize since spawned children need to clean up on shutdown
# Since the threads will be holding resources
#t.daemon = True
t.start()
Question 1
When I call the function in my script (as written above), I get the following error:
AttributeError: 'str' object has no
attribute 'do_work'
Which makes sense, since the dictionary key is a string (name of the module to be imported).
When I add the statement:
import worker
before spawning the thread, I get the error:
ImportError: No module named worker
This is strange, since the variable name rather than the value it holds are being used - when I print the variable, I get the value (as I expect) whats going on?
Question 2
As I mentioned in the comments section, I realize that the do_work() function written in the spawned children needs to cleanup after itself. My understanding is to write a clean_up function that is called when do_work() has completed successfully, or an unhandled exception is caught - is there anything more I need to do to ensure resources don't leak or leave the OS in an unstable state?
Question 3
If I comment out the t.daemon flag statement, will the code stil run ASYNCHRONOUSLY?. The work carried out by the spawned children are pretty intensive, and I don't want to have to be waiting for one child to finish before spawning another child. BTW, I am aware that threading in Python is in reality, a kind of time sharing/slicing - thats ok
Lastly is there a better (more Pythonic) way of doing what I'm trying to do?
