Skip to content

Week1 Virtual Machine Setup

Jump to bottom
Kaloyan Pavlov edited this page Nov 30, 2022 · 3 revisions

Task 1

Objectives

  • Create small cluster with 3 virtual machines (1 Master -> 2 slaves like old times)
  • Install hadoop
  • Config automation using ansible (optional ?)
  • Write report on wiki

Creating virtual machines

First step is to create all instances, for this we follow the instructions from here. We create 3 instances with the same setup for this example:

  • Ubuntu LTS
  • Python 3.8
  • lrz.small
  • Non-Internet node

Note! : When adding SSH to the security group, in the Remote dropdown menu select Security Group

InstancesThis is how the instance tab should look like

Configure master node

installing some features

First make sure you have updated apt package manager

sudo apt update

Install the following packages using the apt package manager:

  • Java
  • Yarn
sudo apt install openjdk-8-jdk openjdk-8-jre
sudo apt install yarn
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Note: You have to install java and hadoop on all nodes.

Configure hadoop

All configurations can be found here!.

After configuring hadoop, copy all configuration files using

scp -i <ssh key> ~/hadoop/etc/hadoop/* <IP slave>:hadoop/etc/hadoop/

Run configuration with the following command:

hdfs namenode -format
Output example 
2022-11-06 13:43:27,359 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/192.168.2.5
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.3.4
STARTUP_MSG:   classpath = /home/ubuntu/hadoop/etc/hadoop:/home/ubuntu/hadoop/share/hadoop/common/lib/jsch-0.1.55.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/dnsjava-2.1.7.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/hadoop-annotations-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/snappy-java-1.1.8.2.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jetty-security-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jersey-json-1.19.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/reload4j-1.2.22.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jakarta.activation-api-1.2.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jackson-core-2.12.7.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/guava-27.0-jre.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-daemon-1.0.13.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-text-1.4.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jul-to-slf4j-1.7.36.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/zookeeper-jute-3.5.6.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/woodstox-core-5.3.0.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/curator-client-4.2.0.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/curator-recipes-4.2.0.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/checker-qual-2.5.2.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jetty-util-ajax-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jetty-util-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerb-client-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jackson-databind-2.12.7.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-lang3-3.12.0.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerb-util-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/accessors-smart-2.4.7.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerby-config-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/httpcore-4.4.13.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jetty-webapp-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jetty-server-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/token-provider-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-codec-1.15.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/stax2-api-4.2.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/zookeeper-3.5.6.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jsr305-3.0.2.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerb-common-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jersey-servlet-1.19.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jackson-annotations-2.12.7.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerb-core-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jersey-core-1.19.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/re2j-1.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/json-smart-2.4.7.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-compress-1.21.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-io-2.8.0.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jetty-xml-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/failureaccess-1.0.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerb-server-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/netty-3.10.6.Final.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/slf4j-api-1.7.36.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/httpclient-4.5.13.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jetty-servlet-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/commons-net-3.6.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/metrics-core-3.2.4.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jersey-server-1.19.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/hadoop-auth-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jetty-http-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/gson-2.8.9.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/avro-1.7.7.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-9.8.1.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/asm-5.0.4.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/jetty-io-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/common/lib/curator-framework-4.2.0.jar:/home/ubuntu/hadoop/share/hadoop/common/hadoop-registry-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/common/hadoop-common-3.3.4-tests.jar:/home/ubuntu/hadoop/share/hadoop/common/hadoop-common-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/common/hadoop-kms-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/common/hadoop-nfs-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/hadoop-annotations-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/snappy-java-1.1.8.2.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-native-unix-common-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-http2-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/paranamer-2.3.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jetty-security-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/reload4j-1.2.22.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-resolver-dns-native-macos-4.1.77.Final-osx-x86_64.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jakarta.activation-api-1.2.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jackson-core-2.12.7.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-text-1.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/zookeeper-jute-3.5.6.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/woodstox-core-5.3.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/curator-client-4.2.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kotlin-stdlib-1.4.10.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-sctp-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/curator-recipes-4.2.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-classes-epoll-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-smtp-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jetty-util-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jackson-databind-2.12.7.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-rxtx-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-resolver-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-lang3-3.12.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-redis-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kotlin-stdlib-common-1.4.10.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/accessors-smart-2.4.7.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/httpcore-4.4.13.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-resolver-dns-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-handler-proxy-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jetty-webapp-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-handler-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jetty-server-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-haproxy-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-codec-1.15.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/stax2-api-4.2.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-common-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-buffer-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/zookeeper-3.5.6.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/hadoop-shaded-guava-1.1.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/okhttp-4.9.3.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jettison-1.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jackson-annotations-2.12.7.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/re2j-1.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-memcache-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/okio-2.8.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-dns-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-resolver-dns-native-macos-4.1.77.Final-osx-aarch_64.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-native-epoll-4.1.77.Final-linux-aarch_64.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/json-smart-2.4.7.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-http-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-compress-1.21.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-resolver-dns-classes-macos-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-io-2.8.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jetty-xml-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-native-kqueue-4.1.77.Final-osx-aarch_64.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-classes-kqueue-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-stomp-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/httpclient-4.5.13.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-native-kqueue-4.1.77.Final-osx-x86_64.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jetty-servlet-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/commons-net-3.6.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-transport-udt-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/hadoop-auth-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jetty-http-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/gson-2.8.9.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-all-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/avro-1.7.7.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/nimbus-jose-jwt-9.8.1.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-xml-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/asm-5.0.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/jetty-io-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/curator-framework-4.2.0.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-mqtt-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/lib/netty-codec-socks-4.1.77.Final.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.4-tests.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.4-tests.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.3.4-tests.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-3.3.4-tests.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.4-tests.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn:/home/ubuntu/hadoop/share/hadoop/yarn/lib/javax-websocket-server-impl-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/javax-websocket-client-impl-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/snakeyaml-1.26.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/guice-4.0.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/websocket-common-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/json-io-2.5.1.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.12.7.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jetty-client-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jersey-client-1.19.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/websocket-api-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-base-2.12.7.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/websocket-server-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/javax.websocket-client-api-1.0.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/websocket-client-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/bcprov-jdk15on-1.60.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/asm-tree-9.1.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/websocket-servlet-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/asm-analysis-9.1.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jetty-annotations-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/objenesis-2.6.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/asm-commons-9.1.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/java-util-1.9.0.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jetty-plus-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/javax.websocket-api-1.0.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jline-3.9.0.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.12.7.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/fst-2.50.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jetty-jndi-9.4.43.v20210629.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/home/ubuntu/hadoop/share/hadoop/yarn/lib/jna-5.2.0.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-services-api-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-client-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-services-core-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-registry-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-server-router-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-api-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-applications-mawo-core-3.3.4.jar:/home/ubuntu/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.3.4.jar
STARTUP_MSG:   build = https://github.com/apache/hadoop.git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb; compiled by 'stevel' on 2022-07-29T12:32Z
STARTUP_MSG:   java = 11.0.16
************************************************************/
2022-11-06 13:43:27,374 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2022-11-06 13:43:27,625 INFO namenode.NameNode: createNameNode [-format]
2022-11-06 13:43:28,460 INFO common.Util: Assuming 'file' scheme for path $HOME/hadoop/data/nameNode in configuration.
2022-11-06 13:43:28,461 INFO common.Util: Assuming 'file' scheme for path $HOME/hadoop/data/nameNode in configuration.
2022-11-06 13:43:28,475 INFO namenode.NameNode: Formatting using clusterid: CID-9ffeed55-0f0b-4eb2-a0e6-f3216fb4e58b
2022-11-06 13:43:28,542 INFO namenode.FSEditLog: Edit logging is async:true
2022-11-06 13:43:28,597 INFO namenode.FSNamesystem: KeyProvider: null
2022-11-06 13:43:28,598 INFO namenode.FSNamesystem: fsLock is fair: true
2022-11-06 13:43:28,600 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2022-11-06 13:43:28,639 INFO namenode.FSNamesystem: fsOwner                = ubuntu (auth:SIMPLE)
2022-11-06 13:43:28,639 INFO namenode.FSNamesystem: supergroup             = supergroup
2022-11-06 13:43:28,639 INFO namenode.FSNamesystem: isPermissionEnabled    = true
2022-11-06 13:43:28,639 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
2022-11-06 13:43:28,640 INFO namenode.FSNamesystem: HA Enabled: false
2022-11-06 13:43:28,721 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2022-11-06 13:43:28,748 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2022-11-06 13:43:28,748 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2022-11-06 13:43:28,752 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2022-11-06 13:43:28,753 INFO blockmanagement.BlockManager: The block deletion will start around 2022 Nov 06 13:43:28
2022-11-06 13:43:28,757 INFO util.GSet: Computing capacity for map BlocksMap
2022-11-06 13:43:28,758 INFO util.GSet: VM type       = 64-bit
2022-11-06 13:43:28,759 INFO util.GSet: 2.0% max memory 1.0 GB = 21.5 MB
2022-11-06 13:43:28,759 INFO util.GSet: capacity      = 2^21 = 2097152 entries
2022-11-06 13:43:28,774 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
2022-11-06 13:43:28,775 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2022-11-06 13:43:28,784 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
2022-11-06 13:43:28,788 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2022-11-06 13:43:28,788 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2022-11-06 13:43:28,789 INFO blockmanagement.BlockManager: defaultReplication         = 2
2022-11-06 13:43:28,789 INFO blockmanagement.BlockManager: maxReplication             = 512
2022-11-06 13:43:28,789 INFO blockmanagement.BlockManager: minReplication             = 1
2022-11-06 13:43:28,789 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
2022-11-06 13:43:28,790 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
2022-11-06 13:43:28,790 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
2022-11-06 13:43:28,790 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
2022-11-06 13:43:28,838 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2022-11-06 13:43:28,838 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2022-11-06 13:43:28,838 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2022-11-06 13:43:28,838 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
2022-11-06 13:43:28,854 INFO util.GSet: Computing capacity for map INodeMap
2022-11-06 13:43:28,854 INFO util.GSet: VM type       = 64-bit
2022-11-06 13:43:28,855 INFO util.GSet: 1.0% max memory 1.0 GB = 10.7 MB
2022-11-06 13:43:28,855 INFO util.GSet: capacity      = 2^20 = 1048576 entries
2022-11-06 13:43:28,856 INFO namenode.FSDirectory: ACLs enabled? true
2022-11-06 13:43:28,860 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2022-11-06 13:43:28,860 INFO namenode.FSDirectory: XAttrs enabled? true
2022-11-06 13:43:28,861 INFO namenode.NameNode: Caching file names occurring more than 10 times
2022-11-06 13:43:28,865 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2022-11-06 13:43:28,870 INFO snapshot.SnapshotManager: SkipList is disabled
2022-11-06 13:43:28,878 INFO util.GSet: Computing capacity for map cachedBlocks
2022-11-06 13:43:28,878 INFO util.GSet: VM type       = 64-bit
2022-11-06 13:43:28,878 INFO util.GSet: 0.25% max memory 1.0 GB = 2.7 MB
2022-11-06 13:43:28,878 INFO util.GSet: capacity      = 2^18 = 262144 entries
2022-11-06 13:43:28,889 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2022-11-06 13:43:28,893 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2022-11-06 13:43:28,893 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2022-11-06 13:43:28,901 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2022-11-06 13:43:28,902 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2022-11-06 13:43:28,904 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2022-11-06 13:43:28,908 INFO util.GSet: VM type       = 64-bit
2022-11-06 13:43:28,908 INFO util.GSet: 0.029999999329447746% max memory 1.0 GB = 329.6 KB
2022-11-06 13:43:28,909 INFO util.GSet: capacity      = 2^15 = 32768 entries
Re-format filesystem in Storage Directory root= /home/ubuntu/$HOME/hadoop/data/nameNode; location= null ? (Y or N) y
2022-11-06 13:43:31,400 INFO namenode.FSImage: Allocated new BlockPoolId: BP-514907876-192.168.2.5-1667742211380
2022-11-06 13:43:31,402 INFO common.Storage: Will remove files: [/home/ubuntu/$HOME/hadoop/data/nameNode/current/fsimage_0000000000000000000, /home/ubuntu/$HOME/hadoop/data/nameNode/current/VERSION, /home/ubuntu/$HOME/hadoop/data/nameNode/current/fsimage_0000000000000000000.md5, /home/ubuntu/$HOME/hadoop/data/nameNode/current/seen_txid]
2022-11-06 13:43:31,441 INFO common.Storage: Storage directory /home/ubuntu/$HOME/hadoop/data/nameNode has been successfully formatted.
2022-11-06 13:43:31,484 INFO namenode.FSImageFormatProtobuf: Saving image file /home/ubuntu/$HOME/hadoop/data/nameNode/current/fsimage.ckpt_0000000000000000000 using no compression
2022-11-06 13:43:31,634 INFO namenode.FSImageFormatProtobuf: Image file /home/ubuntu/$HOME/hadoop/data/nameNode/current/fsimage.ckpt_0000000000000000000 of size 401 bytes saved in 0 seconds .
2022-11-06 13:43:31,652 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2022-11-06 13:43:31,686 INFO namenode.FSNamesystem: Stopping services started for active state
2022-11-06 13:43:31,687 INFO namenode.FSNamesystem: Stopping services started for standby state
2022-11-06 13:43:31,695 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2022-11-06 13:43:31,695 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.2.5
************************************************************/

Run hadoop

You can initialize hadoop by executing the start-dfs.sh file. Because my ssh key configuration in hadoop is wrong I get the following error:

Starting namenodes on [node-master]
node-master: ssh: Could not resolve hostname node-master: Temporary failure in name resolution
Starting datanodes
-i: Warning: Identity file -- not accessible: No such file or directory.
-i: ssh: Could not resolve hostname /home/ubuntu/hadoop/bin/hdfs: Name or service not known
sed: -e expression #1, char 13: unknown option to `s'
sed: -e expression #1, char 13: unknown option to `s'
-i: Warning: Identity file -- not accessible: No such file or directory.
sed: -e expression #1, char 13: unknown option to `s'
-i: Warning: Identity file -- not accessible: No such file or directory.
-i: ssh: Could not resolve hostname /home/ubuntu/hadoop/bin/hdfs: Name or service not known
-i: ssh: Could not resolve hostname /home/ubuntu/hadoop/bin/hdfs: Name or service not known
localhost: ubuntu@localhost: Permission denied (publickey).
10.195.7.66: ubuntu@10.195.7.66: Permission denied (publickey).
10.195.4.218: ubuntu@10.195.4.218: Permission denied (publickey).
Starting secondary namenodes [master]
master: ubuntu@master: Permission denied (publickey).

Installing Hadoop filesystem

A complete tutorial for setting up a cluster with three nodes can be found here!.

To install hadoop on the master node on the master node we run the following commands:

wget http://apache.cs.utah.edu/hadoop/common/current/hadoop-3.3.4.tar.gz
tar -xzf hadoop-3.3.4.tar.gz
mv hadoop-3.3.4 hadoop

Warning: installation can take some time

hadoop

Add hadoop to PATH

Add hadoop to PATH in .bashrc file by adding the following commands:

# Add hadoop to PATH
export HADOOP_HOME=${HOME}/hadoop
export PATH=${PATH}:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Clone this wiki locally