Lab12 – Writing local file to HDFS

Hi Hadoopers,

You might have seen my earliest post on how to read from HDFS using APIs. Here is the post that can tell you how to write to HDFS.

Input file in HDD – /opt/hadoop/feed/output/2016-09-18
HDFS file – /user/hadoop/feed/2016-09-18


The following code needs two inputs. One is the local file and another one is HDFS file.

package org.grassfield.hadoop.input;


import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

 * upload local file to HDFS
 * @author pandian
public class LoadItemsHdfs {

     * @param args    localFile remoteFile
     * @throws IOException
    public static void main(String[] args) throws IOException {
        Path path = new Path("hdfs://gandhari:9000"+args[1]);
        FileSystem fs = FileSystem.get(new Configuration());
        BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fs.create(path, true)));
        BufferedReader br = new BufferedReader(new FileReader(args[0]));
        String line = null;


Let’s execute it.

hadoop@gandhari:~/jars$ hadoop jar FeedCategoryCount-9.jar org.grassfield.hadoop.input.LoadItemsHdfs ../feed/output/2016-09-18 /user/hadoop/feed/2016-09-18
hadoop@gandhari:~/jars$ hadoop fs -ls /user/hadoop/feed
Found 1 items
-rw-r--r--   3 hadoop supergroup     120817 2016-09-18 06:51 /user/hadoop/feed/2016-09-18
hadoop@gandhari:~/jars$ hadoop fs -cat /user/hadoop/feed/2016-09-18

application/rss+xml     Today Online - Hot news null null    Rosberg in pole position to claim victory on Sunday
 Today       Sat Sep 17 23:44:59 MYT 2016    []
 application/rss+xml     Today Online - Hot news null null    No slowing Tang down despite qualifying setback Today  Sat Sep 17 22:09:20 MYT 2016    []

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s