Java program to copy a file from local disk to Hadoop Cluster with progress bar

Hadoop

You need to look at the following blog posts to understand this post in a better way.

  1. Copying the File to HDFS file system
  2. A java program to read the file from HDFS
  3. A java program to read the file from HDFS – 2
  4. Java program to read a file from Hadoop Cluster 2 (with file seek)

We copied the file using CLI in example 1 given above. Lets use java to do the same.

Here comes the full java program to copy the local file to HDFS. You can get it from https://github.com/tomwhite/hadoop-book/blob/3e/ch03/src/main/java/FileCopyWithProgress.java

import java.io.*;
import java.net.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.util.*;

public class FileCopyWithProgress{
    public static void main(String [] args) throws Exception{
        String localSrc=args[0];
        String dst=args[1];
        InputStream in=new BufferedInputStream(new FileInputStream(localSrc));
        Configuration conf=new Configuration();
        FileSystem fs = FileSystem.get(URI.create(dst), conf);
        OutputStream out=fs.create(new Path(dst),new Progressable(){

View original post 61 more words

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s