How to Deploy Hadoop with Ambari
2019-12-23 · 9 min read
1. BUILD TWO VM(CentOS 7)
2. SSH CONNECTION
Create public key in Xshell,then copy it to the google cloud server.(For Users)
Change root code(For Root)
3. REMOTE CONTROL
Install xfce for master
# sodu yum groupinstall xfce
Use xstart to load this command /usr/bin/xfce4-session and use Xmanager to remote connect it.
Choose default setting to set up the start window.
4. HOSTNAME AND HOSTS CONFIG
Change hostname to the name in /ect/hosts
Add node's hostname and ip in master's hosts file and add master's hostname and ip in node's hosts file.
centos 7
[root@localhost ~]# hostnamectl set-hostname master #your hostname
[root@localhost ~]# vim /etc/sysconfig/network
-------------------------------------------------------------------
# Created by anaconda
NETWORKING=yes
HOSTNAME=master #your hostname
--------------------------------------------------------------------
[root@localhost ~]# vi /etc/hosts
--------------------------------------------------------------------
192.168.192.154 master
192.168.192.155 node
#your master and nodes ip and hostname
5. Close Firewall
centos7
[root@localhost ~]# systemctl disable firewalld.service
[root@localhost ~]# systemctl stop firewalld.service
6. Close SElinux
[root@localhost ~]# vi /etc/sysconfig/selinux
----------------------------------------------------------------------
SELINUX=disabled
7. Clock Synchronise
[root@localhost ~] yum install -y ntp
[root@localhost ~] chkconfig --list ntpd
[root@localhost ~] systemctl is-enabled ntpd
[root@localhost ~] systemctl enable ntpd
[root@localhost ~] systemctl start ntpd
8. Install JDK1.8
[root@localhost ~]# cd /usr/local
[root@localhost local]# mkdir jdk8
[root@localhost local]# cd jdk8
[root@master jdk8]# tar -zxvf jdk-8u231-linux-x64.tar.gz
[root@master jdk8]# vim /etc/profile
-----------------------------------------------------------------------
#java
export JAVA_HOME=/usr/local/jdk8/jdk1.8.0_231
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin
------------------------------------------------------------------------
[root@master jdk8]# source /etc/profile
[root@master jdk8]# java -version #Check it
9.Install Other Services
[root@localhost ~]# umask 0022
[root@ambari01 ~]# yum -y install lrzsz
[root@ambari01 ~]# yum install -y openssh-clients
Steps 1~9 should be done in both master and node server
10. SSH login Without Using Password
[root@master ~]# ssh-keygen -t rsa
[root@master ~]# cd .ssh
[root@master .ssh]# ls
id_rsa id_rsa.pub
[root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.192.155 #node ip address
[root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.192.156 #master ip address
[root@master ~]# ssh 192.168.192.155 #test
[root@node ~]# exit
11. Local Repo Configration
Download these three files to master.
CentOS 7：
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari-2.5.0.3-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.0.3/HDP-2.6.0.3-centos7-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz
[root@master ~]# yum install yum-utils createrepo yum-plugin-priorities -y
#Add gpgcheck=0
[root@master ~]# vi /etc/yum/pluginconf.d/priorities.conf
---------------------------------------------------------------------------
gpgcheck = 0
---------------------------------------------------------------------------
#check httpd
[root@master ~]# yum list httpd
#install httpd
[root@master ~]# yum install httpd
#config HTTP
[root@master ~]# chkconfig httpd on
[root@master ~]# service httpd start
#Creat local repo
[root@master ~]# cd /var/www/html/
[root@master html]# mkdir ambari
[root@master ambari]# cd ..
[root@master html]# mkdir hdp
#creat catalog and unzip
root@master Downloads]# ls
ambari-2.5.0.3-centos7.tar.gz HDP-UTILS-1.1.0.21-centos7.tar.gz
HDP-2.6.0.3-centos7-rpm.tar.gz
[root@master Downloads]# tar -zxvf ambari-2.5.0.3-centos7.tar.gz -C /var/www/html/ambari
[root@master Downloads]# tar -zxvf HDP-2.6.0.3-centos7-rpm.tar.gz -C /var/www/html/hdp
[root@master Downloads]# tar -zxvf HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/hdp
#creat repo
[root@master Downloads]# cd /var/www/html/ambari/
[root@master ambari]# createrepo ./
[root@master Downloads]# cd /etc/yum.repos.d/
[root@master yum.repos.d]# ls
CentOS7-Base-163.repo CentOS-Debuginfo.repo CentOS-Sources.repo
CentOS-Base.repo.backup CentOS-fasttrack.repo CentOS-Vault.repo
CentOS-CR.repo CentOS-Media.repo
[root@master yum.repos.d]# wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari.repo
[root@master yum.repos.d]# vim ambari.repo
------------------------------------------------------------------------------------
#VERSION_NUMBER=2.5.0.3-7
[ambari-2.5.0.3]
name=ambari Version - ambari-2.5.0.3
baseurl=http://master/ambari/ambari/centos7/
gpgcheck=0
gpgkey=http://master/ambari/ambari/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
------------------------------------------------------------------------------------
[root@master yum.repos.d]# wget -nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.0.3/hdp.repo
[root@master yum.repos.d]# vim hdp.repo
------------------------------------------------------------------------------------
#VERSION_NUMBER=2.6.0.3-8
[HDP-2.6.0.3]
name=HDP Version - HDP-2.6.0.3
baseurl=http://master/hdp/HDP/centos7/
gpgcheck=0
gpgkey=http://master/hdp/HDP/centos7/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.21]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.21
baseurl=http://master/hdp/
gpgcheck=0
gpgkey=http://master/hdp/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
-------------------------------------------------------------------------------------
# yum clean list
# yum update list
# yum makecache
# yum repolist
12. Check master/hdp/ and master/ambari/ambari/centos7/
###13. Install MySQL
# wget http://repo.mysql.com/mysql57-community-release-el7-8.noarch.rpm
# rpm -ivh mysql57-community-release-el7-8.noarch.rpm --nodeps --force
# cd /etc/yum.repos.d
# ls -l
# yum install -y mysql-server
# systemctl start mysqld
# service mysqld status
# grep 'temporary password' /var/log/mysqld.log #It will return a random password
# mysql -u root -p
mysql> Enter password: # type the random passwor you got before
mysql> set global validate_password_policy=0;
mysql> set global validate_password_length=4;
mysql> alter user 'root'@'localhost' identified by 'root-1234'; #Set your own password to replace root-1234
mysql> flush privileges;
mysql> quit;
# mysql -u root -p # type the password you set before
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY ' root-1234' WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;
# yum install -y mysql-connector-java
mysql> create database ambari character set utf8;
mysql> CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari-1234'; #Set your own password to replace ambari-1234
mysql> GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
mysql> FLUSH PRIVILEGES;
mysql> create database hive character set utf8;
mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hive-1234'; #Set your own password to replace hive-1234
mysql> GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
mysql> FLUSH PRIVILEGES;
14. Install Ambari
# yum install -y ambari-server
# mysql -uroot -p
mysql> use ambari;
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
# ambari-server setup
--------------------------------------------------------------------------------
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root): root
Adjusting ambari-server permissions and ownership...
Checking JDK...Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==========================================
Enter choice (1): 3
JAVA_HOME=usr/local/jdk8/jdk1.8.0_231/
Configuring database...Enter advanced database configuration [y/n] (n)? y
Configuring database...
========================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
==========================================
Enter choice (1): 3
Hostname (localhost): ambari.server
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (ambari-1234):#password you set before
-------------------------------------------------------------------------------
15. Start Ambari
# ambari-server start
http://master:8080/#/login #USE firefox to open it
Username:admin
Password:admin
Type cluster name: any name your want
Chose HDP-2.6(Defaut Version)
Use local Repository
Target Hosts :Paste both master and node FQDN hostname here
For ssh private key: chose id_rsa from /root/.ssh(It is a hide file,you need click show hide files)
Next
Find : SSLError:Failed to connect.Please check openssl library versions.
Use below set for both master and node
vim /etc/python/cert-verification.cfg
[https]
verify=disable
vim /etc/ambari-agent/ambari-agent
[security]
force_https_protocol=PROTOCOL_TLSv1_2
# service ambari-server restart
And restart the webpage to retry before steps.
-
-
- 1. BUILD TWO VM(CentOS 7)
- 2. SSH CONNECTION
- 3. REMOTE CONTROL
- 4. HOSTNAME AND HOSTS CONFIG
- 5. Close Firewall
- 6. Close SElinux
- 7. Clock Synchronise
- 8. Install JDK1.8
- 9.Install Other Services
- 10. SSH login Without Using Password
- 11. Local Repo Configration
- 12. Check master/hdp/ and master/ambari/ambari/centos7/
- 14. Install Ambari
- 15. Start Ambari
-