Would it be good, if you complete this example for naive userd like me, by deploying this in one of the cloud providers like Windows Azure.

Regards
Mohit

You specify
Map –> Intermediates –> Reduce as
(K1, V1) –-> (K2, V2) –-> V3

where google has
Map –> Intermediates –> Reduce as
(K1, V1) –-> (K2, list(V2)) –-> V2

I thought Hadoop was the same as Google?

See also a C++ implementation of MapReduce at http://www.craighenderson.co.uk/mapreduce which I hope to get through a submission to Boost.org sometime in the future.

First of all hi there, nice post..

As I was reading the post you I didn’t quite understand the “(Yes, I realize I could use .NET’s Func instead but that would just result in horribly long ugly code…)” part. I kind of disagree on that.

Here’s my *naive* implementation of MapReduce to reason my disagreement, in which I use Func:

public static IEnumerable MapReduce(
this IEnumerable source,
Func mapper,
Func grouper,
Func<IGrouping, TReduceResult> reducer)
{
return source.Select(mapper).GroupBy(grouper).Select(reducer);
}


that’s it..
and a small example, say I want to get word counts from a list of documents:

var documents = new[] {
new {
Name = "doc1.txt",
Data = "word1 word2 word3 word4 word2 word2 word4 word4 word4"
},
new {
Name = "doc2.txt",
Data = "word5 word6 word7 word8 word6 word6 word7 word7 word7"
}
};

foreach (var d in docs)
{
Console.WriteLine(d.FileName);
foreach (var w in d.WordOccurences)
Console.WriteLine("\t{0}: {1}", w.Word, w.Occurence);
}


Note: I use MapReduce to iterate over the list of documents as well over the list of words of each document.
I don’t think you get long and ugly code, on the contrary, in my opinion it’s more beautiful.

Anyhow, it was a nice article of yours and if only we had a Querable version of a MapReduce method that could not only run on a bunch of threads but on a cluster too, that would be great .

The Map function:
Gets a key and value where the value is a text to scan. It iterates over the words in the text and outputs values

The Reduce function:
Gets a key (word) and list of values (lots of 1’s) and return a sum of the values which is the number of the word was encountered.
Its the code calling the reduce delegate function that uses the key:

// Reduce on each group
var reduced = from g in groups
let k2 = g.Key
from reducedValue in _reduce(k2, g)
select new KeyValuePair(k2, reducedValue);

So the output of the entire process is an list of pairs…

The key matters to the result of each operation but not to the actual code of the map\reduce delegates as the framework running the delegates takes care of the work related to the keys (grouping etc.)

Can you explain why your MapFromMem and Reduce methods at the end don’t use the key argument they are passed?

“When the map operation outputs its pairs they are already available in memory. For efficiency reasons, sometimes it makes sense to take advantage of this fact by supplying a combiner class to perform a reduce-type function. If a combiner is used then the map key-value pairs are not immediately written to the output. Instead they will be collected in lists, one list per each key value. When a certain number of key-value pairs have been written, this buffer is flushed by passing all the values of each key to the combiner’s reduce method and outputting the key-value pairs of the combine operation as if they were created by the original map operation.

For example, a word count MapReduce application whose map operation outputs (word, 1) pairs as words are encountered in the input can use a combiner to speed up processing. A combine operation will start gathering the output in in-memory lists (instead of on disk), one list per word. Once a certain number of pairs is output, the combine operation will be called once per unique word with the list available as an iterator. The combiner then emits (word, count-in-this-part-of-the-input) pairs. From the viewpoint of the Reduce operation this contains the same information as the original Map output, but there should be far fewer pairs output to disk and read from disk. “

The reducer gets as input an intermediate-key and all the values mapped to it in the mapping phase.
It runs over those values and outputs a result (V3).
You can’t run the reduce until you have that list of all the values mapped to a certain key – so you have to wait for the mapping phase to finish.

The first statement is obviously correct because you can’t start running our reducer method before you have all the values for the intermediate key – which means the mapper has to finish running.

However, if you download Hadoop and run one of its samples you’ll see that it start running the Reducer tasks before the Mapper tasks finish.
This happens because the reducer task is composed of two phases:
1. Shuffle phase: collect the map outputs and group by key
2. Reduce phase: the actual reducing

Phase one of the reducer can execute in parallel with the mapping process and that is what the 2nd statement is referring to – you can start streaming and grouping values from the mappers to the reducers during the mapping process.

Hope this clarifies your confusion…

Regards,
Eran

First you say:
“Therefore both phases can be parallelized with the only bottleneck being the fact that the map phase has to finish for the reduce phase to start.”

Then you say:
“Streamline values emitted by map function to the worker (or workers) doing the reduce. Note that we don’t have to wait for a certain map run to finish going over the entire file in order to start sending its emitted values to the reducer, so the map and reduce functions can process data in parallel.”

Am I missing something, or are these two statements contradictory?

BTW’ the line
MapReduceProgram master = new MapReduceProgram(MapFromMem, Reduce);
is cut