HDFS in Detail

HDFS concepts are interesting. After I highlighted the basic features in HDFS – A birds eye view, I’ll write a note on some of the other important features.

Rack Aware Replication

Hadoop is Rack Aware. When the HDFS client wants to write a block, it goes to name node. Name node directs the client to write in a data node 1 running in rack 1.

The client in data node 1 of Rack 1 makes a replica on another name node – say data node 1 on rack 2.

To make the 3rd replica, client on data node 2, makes a copy in data node 1 of rack 3. If all these are completed successfully, we shall commit this transaction.

hadoop018 - DFS rack aware


I have been told that High Availability is not part of Hadoop 1.x. But Hadoop 2.x has implemented the same.

Name node has an standby, which updates the memory copy of name nodes specification using the update logs. It has an replica of the memory as that of the active name node. when name node goes down, stand-by picks up to serve the HDFS clients.

When there is any change to name space, active node durably logs it to the Journal Node. Standby name node will update the edits using the journal node entries.

ZKFC – ZooKeeper Failover Control along with ZooKeeper Quorum. ZK is used for HA in hadoop which has small amount of data about coordination, notify clients about changes, monitor them for availability etc.

Both name nodes, maintain a session on ZK. When the session crashes, ZK takes it as a name node crash.

hadoop018 - HA

ZK gives flexibility to specifically elect a namenode. When the active node crashes, the standby shall elect itself as the active name node.

ZKFC helps to check the availability of the nodes with ping and availability command. When the name node doesn’t give a healthy signal or frozen or failed to give a response, ZKFC will mark it as unavailable. Each node will run its own ZKFC daemons.

ZKFC also helps to maintain the ZooKeeper sessions of name nodes. The active node acquires a lock on ZK. When it is not reachable, the lock is deleted from ZK.If the namenode is healthy and no other name node has acquired the lock, it will win the election and acquire the lock.

Hadoop Federation & HA

Hadoop 1.x has one name node which has the block management system. When we manage high volume of data and the name node doesn’t have enough memory to hold the meta data, it gives a scalability issue to a Hadoop Admin.

hadoop019 - Hadoop1x - Federation

Hadoop 2.x overcomes this issue using the changes to its name node architecture. Instead of having one nodes, we have multiple nodes now. Instead of one block management, we have pool of block management system. If one of the name node goes down, the content would be delivered by other name nodes until we bring the faulty name node up.

hadoop020 - Hadoop2x - Federation

I need to write about a data replication and various HDFS clients. I need time. Hope I shall do it this weekend.

Good day, guys and gals. Have a wonderful weekend.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s