Store files in two disks in Hadoop – Storage Reliability

Hi BigDs,

As I’m using single node cluster, I’m trying to store the copy in two different disks in my server, as directed by my instructor. Let’s see how.

Initially following is my data directory configured in hdfs-site.xml.

<property>
<name>dfs.data.dir</name>
<value>/var/lib/hadoop/cache/hadoop/dfs/data</value>
</property>

I’ll change it as below.

<property>
<name>dfs.data.dir</name>
<value>/var/lib/hadoop/cache/hadoop/dfs/data,/var/lib/hadoop/cache/hadoop/dfs/data2</value>
</property>

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ sudo mkdir data2

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ sudo chown hadoop:hadoop data2/

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ ls -alt
total 24
drwx------ 3 hadoop hadoop 4096 Sep  2 15:20 data
drwxr-xr-x 3 hadoop hadoop 4096 Sep  2 15:20 name
drwxr-xr-x 6 root   root   4096 Sep  2 15:19 .
drwxr-xr-x 2 hadoop hadoop 4096 Sep  2 15:19 data2
drwxr-xr-x 3 hadoop hadoop 4096 Aug 26 20:16 namesecondary
drwxr-xr-x 4 root   root   4096 Aug 20 09:52 ..

Stop and start the dfs. We can see the data2 folder is getting replicated.

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ ls /var/lib/hadoop/cache/hadoop/dfs/data2
current  in_use.lock

hadoop@gandhari:/var/lib/hadoop/cache/hadoop/dfs$ ls /var/lib/hadoop/cache/hadoop/dfs/data2/current/
BP-419586781-192.168.0.169-1471713542064  VERSION

data node log shows it is scanning in two folders now.

2016-09-02 15:29:48,346 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-419586781-192.168.0.169-1471713542064 on volume /var/lib/hadoop/cache/hadoop/dfs/data/current…

2016-09-02 15:29:48,407 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-419586781-192.168.0.169-1471713542064 on volume /var/lib/hadoop/cache/hadoop/dfs/data2/current..

 

Advertisements

One thought on “Store files in two disks in Hadoop – Storage Reliability

  1. Pingback: Anotomy of Read & Write in HDFS | JavaShine

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s