Network bandwidth bottleneck for sorting of mapreduce intermediate keys?

Posted by Zubair on Stack Overflow See other posts from Stack Overflow or by Zubair
Published on 2010-03-11T08:42:42Z Indexed on 2010/03/13 5:55 UTC
Read the original article Hit count: 180

Filed under:

mapreduce

I have been learning the mapreduce algorithm and how it can potentially scale to millions of machines, but I don't understand how the sorting of the intermediate keys after the map phase can scale, as there will be:

1,000,000 x 1,000,000

: potential machines communicating small key / value pairs of the intermediate results with each other? Isn't this a bottleneck?

Related posts about mapreduce

Chaining multiple MapReduce jobs in Hadoop.

as seen on Stack Overflow - Search for 'Stack Overflow'
In many real-life situations where you apply MapReduce, the final algorithms end up being several MapReduce steps. I.e. Map1 , Reduce1 , Map2 , Reduce2 , etc. So you have the output from the last reduce that is needed as the input for the next map. The intermediate data is something you (in general)… >>> More
Error in using Hadoop MapReduce in Eclipse

as seen on Stack Overflow - Search for 'Stack Overflow'
When I executed a MapReduce program in Eclipse using Hadoop, I got the below error. It has to be some change in path, but I'm not able to figure it out. Any idea? 16:35:39 INFO mapred.JobClient: Task Id : attempt_201001151609_0001_m_000006_0, Status : FAILED java.io.FileNotFoundException: File C:/tmp/hadoop-Shwe/mapred/local/taskTracker/jobcache/job_201001151609_0001/attempt_201001151609_0001_m_000006_0/work/tmp… >>> More
Help converting java program to MapReduce job

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I would like to convert the following Java program to a MapReduce job. I have read about MapReduce and feel like this would be a good problem to solve using it, but I cannot figure out what to do. This basically loops through a directory of html files and parses them into a CSV file. http://www… >>> More
Can a webserver be implemented using mapreduce?

as seen on Stack Overflow - Search for 'Stack Overflow'
Could mapreduce be used to implement a webserver? I'm thinking something like when a request comes in then the request sits on a queue, until a server is free to process it? Or am I missing the point here? >>> More
Big Data – Buzz Words: What is MapReduce – Day 7 of 21

as seen on SQL Authority - Search for 'SQL Authority'
In yesterday’s blog post we learned what is Hadoop. In this article we will take a quick look at one of the four most important buzz words which goes around Big Data – MapReduce. What is MapReduce? MapReduce was designed by Google as a programming model for processing large data sets with… >>> More

Developer IT

Network bandwidth bottleneck for sorting of mapreduce intermediate keys? - Developer IT

Network bandwidth bottleneck for sorting of mapreduce intermediate keys?

mapreduce

Related posts about mapreduce

Chaining multiple MapReduce jobs in Hadoop.

Error in using Hadoop MapReduce in Eclipse

Help converting java program to MapReduce job

Can a webserver be implemented using mapreduce?

Big Data – Buzz Words: What is MapReduce – Day 7 of 21

Categories cloud