'How to transfer data from local file system (linux) to a Hadoop Cluster made on Google Cloud Platform

I am a beginner in Hadoop, I made a Hadoop Cluster (one master and two slaves) on Google Cloud Platform.

I accessed the master of the cluster using from the local file system (Linux): ssh -i key key@public_ip_of_master

Then I did sudo su - inside the cluster because Hadoop functions only appears while being root.

Then I initiated the HDFS using start-dfs.sh and start-all.sh

Now the problem is that I want to tranfer files from the local Linux file system to the Hadoop Cluster and vice versa using the following command (inserting the command inside the cluster while being root):

root@master:~# hdfs dfs -put   /home/abas1/Desktop/chromFa.tar.gz     /Hadoop_File

The problem is that the local path which is: /home/abas1/Desktop/chromFa.tar.gz is never recognized and I can not seem to know what to do.

I am sure I am missing something trivial but I do not know what it is. I have to use either -copyFromLocal or -put.

linuxhadoopgoogle-cloud-platformhdfs


Solution 1:[1]

local path is never recognized

That is not a Hadoop problem, then. You are on the master node (over SSH), as the root user. There is a /root folder with files, and probably no /home/abas1.

In other words, run ls -l /home, and you see what local files are available.

To get files to the master server to upload from that terminal session, you will want to SCP files first to there from a different machine.

  1. Exit the SSH session
  2. scp -i key root@master-ip home/abas1/Desktop/chromFa.tar.gz /tmp
  3. ssh -i key root@master-ip
  4. Then you can do this
hdfs mkdir /Hadoop_File
ls -l /tmp | grep chromFa  # for example, to check file
hdfs -put /tmp/chromFa.tar.gz /Hadoop_file/

Hadoop functions only appears while being root.

Please do not use root for interacting with Hadoop services. Create unique user accounts for HDFS, YARN, Zookeeper, etc. with restricted permissions like you would for any other Unix process.

Using DataProc will do this... And you can still SSH to it, so you should really considering using it instead of manual GCE cluster.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1

Related Questions