Waiting for thread to finish Python

Posted by lunchtime on Stack Overflow See other posts from Stack Overflow or by lunchtime
Published on 2014-06-08T02:54:08Z Indexed on 2014/06/08 3:25 UTC
Read the original article Hit count: 118

Alright, here's my problem.

I have a thread that creates another thread in a pool, applies async so I can work with the returned data, which is working GREAT. But I need the current thread to WAIT until the result is returned.

Here is the simplified code, as the current script is over 300 lines. I'm sure i've included everything for you to make sense of what I'm attempting:

from multiprocessing.pool import ThreadPool
import threading
pool = ThreadPool(processes=1)

class MyStreamer(TwythonStreamer):
#[...]
    def on_success(self, data): #### Everytime data comes in, this is called
    #[...]
#<Pseudocode> 
    if score >= limit
        if list exists: Do stuff
        elif list does not exist:              
#</Pseudocode>
            dic = []
            dic.append([k1, v1])
            did = dict(dic)
            async_result = pool.apply_async(self.list_step, args=(did))
            return_val = async_result.get()
            slug = return_val[0]
            idd = return_val[1]
#[...]
    def list_step(self, *args):
            ## CREATE LIST
            ## RETURN 2 VALUES

class threadStream (threading.Thread):
    def __init__(self, auth):
        threading.Thread.__init__(self)
        self.auth = auth

    def run(self):
        stream = MyStreamer(auth = auth[0], *auth[0])
        stream.statuses.filter(track=auth[1])

t = threadStream(auth=AuthMe)
t.start()

I receive the results as intended, which is great, but how do I make it so this thread t waits for the async_result to come in?? My problem is everytime new data comes in, it seems that the ## CREATE LIST function is called multiple times if similar data comes in quickly enough. So I'm ending up with many lists of the same name when I have code in place to ensure that a list will never be created if the name already exists.

So to reiterate: How do I make this thread wait on the function to complete before accepting new data / continuing. I don't think time.sleep() works because on_success is called when data enters the stream. I don't think Thread.Join() will work either since I have to use a ThreadPool.apply_async to receive the data I need. Is there a hack I can make in the MyStreamer class somehow? I'm kind of at a loss here. Am I over complicating things and can this be simplified to do what I want?

© Stack Overflow or respective owner

Related posts about python

Related posts about multithreading