So we are looking at the 7th circle today – which is the job clean up.
MR job writes many intermediate results and junk files during the operation. Once the job is completed, these junks would occupy space on HDFS which is of no benefit any more. Hence the clean up task is launched.
- Job tracker informs all the task trackers to perform the cleanup.
- Individual task tracker cleans up the work folders
- They clean up the temporary directory
- Once the cleanup task is successful, Task Tracker ends the job by writing _SUCCESS file
This is the continuation of my Hadoop series. This will follow the folder structures created earlier. Pls refer to the following posts.
Download and Extract
hadoop@gandhari:~$ wget http://download.nus.edu.sg/mirror/apache/pig/pig-0.16.0/pig-0.16.0.tar.gz
hadoop@gandhari:~$ gunzip pig-0.16.0.tar.gz
hadoop@gandhari:~$ tar -xvf pig-0.16.0.tar
hadoop@gandhari:~$ ln -s pig-0.16.0 pig
Add the following environmental variables to your ~/.bashrc
Sample Pig Job
Let’s copy a file using hadoop and read it with Pig
hadoop@gandhari:~$ ls -alt>test.txt
hadoop@gandhari:~$ hadoop fs -mkdir /pigdata
hadoop@gandhari:~$ hadoop fs -put test.txt /pigdata
Let’s enure the jobs are running.
grunt> a = LOAD ‘/data/test.txt’ as (name:chararray);
grunt> dump a;
2016-08-24 14:01:27,845 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil – Total input paths to process : 1
(drwxr-xr-x 15 hadoop hadoop 4096 Aug 24 06:00 .)
(-rw-rw-r– 1 hadoop hadoop 0 Aug 24 06:00 text.txt)
(drwxrwxr-x 3 hadoop hadoop 4096 Aug 24 05:58 logs)
(-rw-rw-r– 1 hadoop hadoop 139 Aug 23 16:09 .hivehistory)
(-rw-r–r– 1 hadoop hadoop 4567 Aug 23 15:59 .bashrc)
Here is a scary exception thrown out when I started Hive.
org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver (“com.mysql.jdbc.Driver”) was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
I forgot to copy the mysql driver to hive lib folder. Here is the command to copy the same
cp /usr/share/java/mysql-connector-java-5.1.38.jar /opt/hadoop/hive/lib/
I like this font and embedded in my site.
<link rel='stylesheet' id='superhero-carrois-gothic-css' href='https://fonts.googleapis.com/css?family=Carrois+Gothic&ver=4.4.1-alpha-36109' type='text/css' media='all' />
When I say someone, this font is not readable on the screen, I used to get a reply ‘this font is used in xxxxx device’. My reaction then would be like this!
Use the following menu based command line to change the timezone in Ubuntu
$ sudo dpkg-reconfigure tzdata
Screenshots are given below for reference.
Here is the snippet for jQuery date picker.
<link rel="stylesheet" href="//code.jquery.com/ui/1.11.4/themes/smoothness/jquery-ui.css">
Date (Click for calendar): <input type="text" name="mydate" id="mydate"/>
We have added a simple CSS using the ResourceHandlers. Now, it is the time to test jQuery on a Spring MVC. First Step is to display the calendar object on the jsp.
Insert the scripts and CSS to head part of jsp
<link rel="stylesheet" href="http://code.jquery.com/ui/1.11.0/themes/smoothness/jquery-ui.css">
Thats all. The calendar object should rendered on the browser window.