Sqoop Installation and Configuration

2000px-wikipedia-logo-v2-en-svg1

Sqoop is a command-line interface application for transferring data between relational databases and Hadoop. It supports incremental loads of a single table or a free form SQL query as well as saved jobs which can be run multiple times to import updates made to a database since the last import. Imports can also be used to populate tables in Hive or HBase. Exports can be used to put data from Hadoop into a relational database. Sqoop got the name from sql+hadoop. Sqoop became a top-level Apache project in March 2012.

Download and Extract

hadoop@gandhari:~$ wget http://download.nus.edu.sg/mirror/apache/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

hadoop@gandhari:~$ gunzip sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

hadoop@gandhari:~$ tar -xvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar

hadoop@gandhari:~$ ln -s sqoop-1.4.6.bin__hadoop-2.0.4-alpha/ sqoop

.bashrc and other environmental changes

#SQOOP VARIABLES
export SQOOP_HOME=/opt/hadoop/sqoop
export PATH=$PATH:$SQOOP_HOME/bin

hadoop@gandhari:~$ source ~/.bashrc

Sqoop config

hadoop@gandhari:~$ cd sqoop
hadoop@gandhari:~/sqoop$ cd conf/
hadoop@gandhari:~/sqoop/conf$ ls
oraoop-site-template.xml  sqoop-env-template.sh    sqoop-site.xml
sqoop-env-template.cmd    sqoop-site-template.xml

hadoop@gandhari:~/sqoop/conf$ cp sqoop-env-template.sh sqoop-env.sh

export HADOOP_COMMON_HOME=/opt/hadoop
export HADOOP_MAPRED_HOME=/opt/hadoop

hadoop@gandhari:~/sqoop/conf$ cp /usr/share/java/mysql-connector-java-5.1.38.jar /opt/hadoop/sqoop/lib/

Execution

hadoop@gandhari:~/sqoop/conf$ cd ..
hadoop@gandhari:~/sqoop$ sqoop-version
Warning: /opt/hadoop/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/hadoop/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/hadoop/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/hadoop/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
16/08/24 15:24:09 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015

 

 

 

Advertisements

One thought on “Sqoop Installation and Configuration

  1. Pingback: Hadoop Eco System Installation – Contents | JavaShine

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s