java.io.IOException: Illegal partition for Big Data (3)

Hi,

I was writing a sample MR program with Partitioner for 6th lab exercise. My job failed with below given teasing exception.

16/09/10 18:28:40 WARN mapred.LocalJobRunner: job_local938197532_0001
java.lang.Exception: java.io.IOException: Illegal partition for Big Data (3)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.io.IOException: Illegal partition for Big Data (3)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
        at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
        at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
        at org.grassfield.hadoop.FeedCategoryCountMapper.map(FeedCategoryCountMapper.java:32)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
16/09/10 18:28:41 INFO mapreduce.Job: Job job_local938197532_0001 failed with state FAILED due to: NA

logo-mapreduce

Actually there were only 3 reducers 0, 1 and 2. I mentioned 3 in my partitioner instead of 2. Correcting the right reducer number in the partitioner solved this problem.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s