Converting python collaborative filtering code to use Map Reduce
- by Neil Kodner
Using Python, I'm computing cosine similarity across items.
given event data that represents a purchase (user,item), I have a list of all items 'bought' by my users.
Given this input data
(user,item)
X,1
X,2
Y,1
Y,2
Z,2
Z,3
I build a python dictionary
{1: ['X','Y'], 2 : ['X','Y','Z'], 3 : ['Z']}
From that dictionary, I generate a bought/not bought matrix, also another dictionary(bnb).
{1 : [1,1,0], 2 : [1,1,1], 3 : [0,0,1]}
From there, I'm computing similarity between (1,2) by calculating cosine between (1,1,0) and (1,1,1), yielding 0.816496
I'm doing this by:
items=[1,2,3]
for item in items:
for sub in items:
if sub >= item: #as to not calculate similarity on the inverse
sim = coSim( bnb[item], bnb[sub] )
I think the brute force approach is killing me and it only runs slower as the data gets larger. Using my trusty laptop, this calculation runs for hours when dealing with 8500 users and 3500 items.
I'm trying to compute similarity for all items in my dict and it's taking longer than I'd like it to. I think this is a good candidate for MapReduce but I'm having trouble 'thinking' in terms of key/value pairs.
Alternatively, is the issue with my approach and not necessarily a candidate for Map Reduce?
