Partitioner is a powerful concept in MapReduce. I have given an example in Lab 06 – A simple hadoop partitioner
Let’s see how to implement same concept using Pig – It’s simple – using split!
Let’s take a simple employee table as below
grunt> fs -cat employee.csv 1, Dharma, 45, Sr Manager 2, Bheema, 43, Cook 3, Arjuna, 41, Instructor 4, Nakula, 35, Jr Instructor 5, Sahadeva, 33, Jr Instructor
Let’s create the employee relation as given below.
grunt> employee = load 'employee.csv' using PigStorage (',') as (eid:int,ename:chararray,eage:int,edes:chararray);
Now I’ll split employees based on age. Above 40, Below 40.
grunt> split employee into employee1 if eage>=40, employee2 if (eage<40); grunt> dump employee1 (1, Dharma,45, Sr Manager) (2, Bheema,43, Cook) (3, Arjuna,41, Instructor) grunt> dump employee2 (4, Nakula,35, Jr Instructor) (5, Sahadeva,33, Jr Instructor)
Note – You need to specify the else column – it is must.