We shall talk about 3rd circle today, as we talk about Job submission and Job initialilzation already.
Scheduling the jobs is an interesting concept. I’m really excited to see the communication between Scheduler, Job tracker and Task tracker.
- The task tracker keeps on sending heartbeats to Job Tracker about the status of the job. So, it says to Job Tracker that job is completed and it wants more jobs.
- Job Tracker updates the task status and make a note of Task Tracker’s message.
- Job Tracker goes to Scheduler asking for tasks.
- Scheduler updates the tasks scheduler record. Based on job scheduling policy, either it makes the job client to wait or process the job. It is based on execution policy, priority etc.
- Job tracker gets the task.
- It submits the task to the task tracker.
Yesterday I wrote my note on MR v1. After perceiving the problems, YARN, which is MR v2 is released.
It has lot of advantages as given below.
- MR v1 has two major daemons – Job Tracker and Task Tracker. As all the tasks are aggregated under these daemons, it may hang when we process large set of data
- It does not provide HA.
Yarn – MR2 – Process flow
- JOB SUMISSION: User executes his YARN code. You are going to launch a job. A new JVM is launched.
- JOB SUMISSION: The job is submitted to App Manager of Resource Manager.
- JOB SUMISSION: In turn, Resource manager gives you back an job identifier. Job resources are copied to the HDFS. Name node has the meta data of the resources. (It is not the file or content. It is the job detail)
- RESOURCE MGMT: Resource Manager talks to one of the machines where the node manager is running – Machine A. It will ask the node manager to start an Application Master in that machine.
- RESOURCE MGMT: Node manager accepts the call and spawns Application Master.
- RESOURCE MGMT: Application master goes to Name node to get the meta data of the job details. It calculates the resource usage, split information etc to execute the task.
- RESOURCE MGMT: It requests the RM for the application master instance ID.
- RESOURCE MGMT: The Application Manager identify machines and calculates the resource information of them. It informs the Resource Manager about the identification of the Node Manager to execute the task.
- RESOURCE MGMT: The Resource Manager gives an acknowledgement to Application manager to allocate container from the identified Node Manager.
- RESOURCE MGMT: Application Master talks to Node Manager of Machine B to inform the approval of containers to Node Manager on its box where the data lies.
- TASK EXECUTION: It creates the container. The task will be executed inside the same.
- TASK EXECUTION: The Task Container will ge the job resources from HDFS, we copied in Step #3.
- TASK EXECUTION: Task starts.
- TASK EXECUTION: It is the Application Master’s duty to monitor the container. Hence the container sends the MR status to the same. (Task related information)
- TASK EXECUTION: Node manager of Machine B will update the Resource Manager about the resource consumption (CPU/MEM..) Based on this data, Resource manager decides the Machines for another job execution.
- TASK COMPLETION: Container informs the Application Master about the job completion. Resource Manager is updated. Job queue is updated. Application master de-registers the container and the container is terminated.
Happy Weekend, Hadoopers!