面向生产环境的大集群模式安装Hadoop-1

2019-12-17 18:02:30

一、实验说明

1、本实验将使用DNS而不是hosts文件解析主机名；

2、使用NFS共享密钥文件，而不是逐个手工拷贝添加密钥；

3、复制Hadoop时使用批量拷贝脚本而不是逐台复制。

测试环境：

HostnameIPHadoop版本Hadoop功能系统

hadoop1192.168.1.1610.20.0namenodenfs服务器端 rhel5.4x86

hadoop2192.168.1.1620.20.0datanodedns+nfs客服端 rhel5.4 x86

hadoop3192.168.1.1630.20.0datanodenfs客户端 rhel5.4 x86

二、DNS的安装与配置

1、上传dns目录：

1[root@hadoop2 dns]# ls2dnsmasq.conf dnsmasq.hosts dnsmasq.resolv.conf pid start.sh stop.sh

2、修改dns目录中的文件：

----dnsmasq.conf为dnsmasq的配置文件----

[

root@hadoop2 dns]# cat dnsmasq.conf

cache-size=50000dns-forward-max=1000resolv-file=/dns/dnsmasq.resolv.conf

addn-hosts=/dns/dnsmasq.hosts----dnsmasq缓存下来的域名，不使用/etc/hosts----

[root@hadoop2 dns]# cat dnsmasq.hosts 192.168.1.161 hadoop1192.168.1.162 hadoop2192.168.1.163hadoop3----在dnsmasq.resolv.conf添加上游dns的地址----

[root@hadoop2 dns]# cat dnsmasq.resolv.conf

### /etc/resolv.conffileautogeneratedby netconfig!

# Before you change this filemanually, considerto define the

# static DNS configuration using the following variables in the

# /etc/sysconfig/network/configfile:

# NETCONFIG_DNS_STATIC_SEARCHLIST

# NETCONFIG_DNS_STATIC_SERVERS

# NETCONFIG_DNS_FORWARDER

# ordisable DNS configuration updates via netconfigby setting:

# NETCONFIG_DNS_POLICY=''#

# See also the netconfig(8) manual pageand other documentation.

# Note: Manual change ofthisfile disables netconfig too, but

# may get lost whenthisfilecontainscommentsor empty lines

# only, the netconfig settings are samewithsettingsin this

# fileandincaseofa "netconfigupdate-f" call.

#nameserver 218.108.248.228

nameserver 218.108.248.200

[root@hadoop2 dns]# cat start.sh

#!/bin/sh

killall dnsmasq

dnsmasq --port=53 --pid-file=/dns/pid --conf-file=/dns/dnsmasq.conf[root@hadoop2 dns]# cat stop.sh

#!/bin/sh

killall dnsmasq

3、启动dns，并在hadoop2上进行测试：

[root@hadoop2 dns]# dig@hadoop2 www.qq.com

; <<>>DiG9.3.6-P1-RedHat-9.3.6-4.P1.el5<<>>@hadoop2 www.qq.com

; (1 server found)

;; global options: printcmd

;; Got answer:

;; ->>HEADER<<-opcode: QUERY, status: NOERROR, id:41272;; flags: qr rd ra; QUERY: 1, ANSWER:1, AUTHORITY:3, ADDITIONAL:7;; QUESTION SECTION:

;www.qq.com. IN A

;; ANSWER SECTION:

www.qq.com. 5INA182.254.8.146;; AUTHORITY SECTION:

www.qq.com. 5INNS ns-cmn1.qq.com.

www.qq.com. 5INNS ns-cnc1.qq.com.

www.qq.com. 5INNS ns-os1.qq.com.

;; ADDITIONAL SECTION:

ns-os1.qq.com.5INA184.105.66.196ns-os1.qq.com.5INA202.55.2.226ns-os1.qq.com.5INA202.55.2.230ns-os1.qq.com.5INA114.134.85.106ns-cmn1.qq.com.5INA120.204.202.200ns-cnc1.qq.com.5INA125.39.127.27ns-cnc1.qq.com.5INA61.135.167.182;; Query time: 33 msec

;; SERVER: 192.168.1.162#53(192.168.1.162)

;; WHEN: Sat Aug2419:40:252013;; MSG SIZE rcvd: 221

4、配置hadoop1和hadoop3的/etc/resolv.conf

----dns客户端需要对/etc/resolv.conf文件中注释掉search localdomain，否则无法解析dnsmasq缓存下来的域名----

[

hadoop@hadoop3 ~]$ cat/etc/resolv.conf

; generated by/sbin/dhclient-script

#search localdomain

#nameserver 192.168.11.2

nameserver 192.168.1.162[root@hadoop3 ~]# dig@hadoop2 www.qq.com

; <<>>DiG9.3.6-P1-RedHat-9.3.6-4.P1.el5<<>>@hadoop2 www.qq.com

; (1 server found)

;; global options: printcmd

;; Got answer:

;; ->>HEADER<<-opcode: QUERY, status: NOERROR, id:2061;; flags: qr rd ra; QUERY: 1, ANSWER:1, AUTHORITY:3, ADDITIONAL:7;; QUESTION SECTION:

;www.qq.com. IN A

;; ANSWER SECTION:

www.qq.com. 5INA182.254.8.146;; AUTHORITY SECTION:

www.qq.com. 5INNS ns-cnc1.qq.com.

www.qq.com. 5INNS ns-cmn1.qq.com.

www.qq.com. 5INNS ns-os1.qq.com.

;; ADDITIONAL SECTION:

ns-os1.qq.com.5INA202.55.2.226ns-os1.qq.com.5INA202.55.2.230ns-os1.qq.com.5INA114.134.85.106ns-os1.qq.com.5INA184.105.66.196ns-cmn1.qq.com.5INA120.204.202.200ns-cnc1.qq.com.5INA61.135.167.182ns-cnc1.qq.com.5INA125.39.127.27;; Query time: 24 msec

;; SERVER: 192.168.1.162#53(192.168.1.162)

;; WHEN: Sat Aug2419:44:432013;; MSG SIZE rcvd: 221

三、配置NFS

1、查看nfs是否已经安装

[root@hadoop1 ~]# rpm-qa|grep nfs

nfs-utils-1.0.9-42.el5

nfs-utils-lib-1.0.8-7.6.el5

2、编辑/etc/exports

[root@hadoop1 ~]# cat/etc/exports /home/hadoop/.ssh/*(rw,sync,no_root_squash)

3、创建hadoop用户

[root@hadoop1 ~]# useradd hadoop[root@hadoop1 ~]# passwd hadoop

Changing password foruser hadoop.