MapReduce Job Execution Process – Reduce Function

Hi Hadoopers,

We are in 6th circle today, which is the reducer function. A job is submitted by the user, which has been initiated in 2nd circle for which the setup is completed in 3rd circle.

Map Task was executed in 4th circle and sort & shuffle was completed in 5th circle.

hadoop037-job-submission-1

The reducer will collect the output from all the mappers to apply the user defined reduce function.

hadoop043-hadoop-reducer

  1. Task tracker launches the reduce task
  2. Reduce task (not reduce function) read the jar and xml of the job.
  3. It execute the shuffle. Because the time the reducer task started, all the mappers may not have completed the job. So it goes to individual mapper machines to collect the output and shuffles them.
  4. Once all the mapping activity is finished it invokes the user reducer function (one more reducers).
  5. Each reducers will complete their jobwrite the output records to HDFS.
  6. Those output would be stored in temporary output file first.
  7. Once all the reducers have completed their job, final output would be written to the reducer partition file.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s