problems with unpickling a 80 megabyte file in python

Posted by tipu on Stack Overflow See other posts from Stack Overflow or by tipu
Published on 2010-06-01T22:07:10Z Indexed on 2010/06/01 22:13 UTC
Read the original article Hit count: 283

Filed under:
|

I am using the pickle module to read and write large amounts of data to a file. After writing to the file a 80 megabyte pickled file, I load it in a SocketServer using

class MyTCPHandler(SocketServer.BaseRequestHandler):

    def handle(self):
        print("in handle")
        words_file_handler = open('/home/tipu/Dropbox/dev/workspace/search/words.db', 'rb')
        words = pickle.load(words_file_handler)
        tweets = shelve.open('/home/tipu/Dropbox/dev/workspace/search/tweets.db', 'r');
        results_per_page = 25
        query_details = self.request.recv(1024).strip()
        query_details = eval(query_details)
        query = query_details["query"]
        page = int(query_details["page"]) - 1
        return_ = []
        booleanquery = BooleanQuery(MyTCPHandler.words)
        if query.find("(") > -1:
            result = booleanquery.processAdvancedQuery(query)
        else:
            result = booleanquery.processQuery(query)
        result = list(result)
        i = 0
        for tweet_id in result and i < 25:
            #return_.append(MyTCPHandler.tweets[str(tweet_id)])
            return_.append(tweet_id)
            i += 1
        self.request.send(str(return_))

However the file never seems to load after the pickle.load line and it eventually halts the connection attempt. Is there anything I can do to speed this up?

© Stack Overflow or respective owner

Related posts about python

Related posts about pickle