Alright, here's my problem.
I have a thread that creates another thread in a pool, applies async so I can work with the returned data, which is working GREAT. But I need the current thread to WAIT until the result is returned.
Here is the simplified code, as the current script is over 300 lines. I'm sure i've included everything for you to make sense of what I'm attempting:
from multiprocessing.pool import ThreadPool
import threading
pool = ThreadPool(processes=1)
class MyStreamer(TwythonStreamer):
#[...]
def on_success(self, data): #### Everytime data comes in, this is called
#[...]
#<Pseudocode>
if score >= limit
if list exists: Do stuff
elif list does not exist:
#</Pseudocode>
dic = []
dic.append([k1, v1])
did = dict(dic)
async_result = pool.apply_async(self.list_step, args=(did))
return_val = async_result.get()
slug = return_val[0]
idd = return_val[1]
#[...]
def list_step(self, *args):
## CREATE LIST
## RETURN 2 VALUES
class threadStream (threading.Thread):
def __init__(self, auth):
threading.Thread.__init__(self)
self.auth = auth
def run(self):
stream = MyStreamer(auth = auth[0], *auth[0])
stream.statuses.filter(track=auth[1])
t = threadStream(auth=AuthMe)
t.start()
I receive the results as intended, which is great, but how do I make it so this thread t waits for the async_result to come in?? My problem is everytime new data comes in, it seems that the ## CREATE LIST function is called multiple times if similar data comes in quickly enough. So I'm ending up with many lists of the same name when I have code in place to ensure that a list will never be created if the name already exists.
So to reiterate: How do I make this thread wait on the function to complete before accepting new data / continuing. I don't think time.sleep() works because on_success is called when data enters the stream. I don't think Thread.Join() will work either since I have to use a ThreadPool.apply_async to receive the data I need. Is there a hack I can make in the MyStreamer class somehow? I'm kind of at a loss here. Am I over complicating things and can this be simplified to do what I want?
