problems with unpickling a 80 megabyte file in python
- by tipu
I am using the pickle module to read and write large amounts of data to a file. After writing to the file a 80 megabyte pickled file, I load it in a SocketServer using
class MyTCPHandler(SocketServer.BaseRequestHandler):
def handle(self):
print("in handle")
words_file_handler = open('/home/tipu/Dropbox/dev/workspace/search/words.db', 'rb')
words = pickle.load(words_file_handler)
tweets = shelve.open('/home/tipu/Dropbox/dev/workspace/search/tweets.db', 'r');
results_per_page = 25
query_details = self.request.recv(1024).strip()
query_details = eval(query_details)
query = query_details["query"]
page = int(query_details["page"]) - 1
return_ = []
booleanquery = BooleanQuery(MyTCPHandler.words)
if query.find("(") > -1:
result = booleanquery.processAdvancedQuery(query)
else:
result = booleanquery.processQuery(query)
result = list(result)
i = 0
for tweet_id in result and i < 25:
#return_.append(MyTCPHandler.tweets[str(tweet_id)])
return_.append(tweet_id)
i += 1
self.request.send(str(return_))
However the file never seems to load after the pickle.load line and it eventually halts the connection attempt. Is there anything I can do to speed this up?
