Linux学习21-hadoop续及hadoop高可用部署

yarn部署新增节点node4下载nfs执行yum install nfs-utils命令下载安装NFS相关工具包。node1做免密与本机也做免密在node1上生成SSH密钥对并将公钥分发到各节点实现免密登录。将node1的公钥添加到本机authorized_keys中实现本机免密登录。创建目录并挂载在node1上创建共享目录并配置NFS导出规则执行exportfs -r命令使NFS导出配置生效在其他节点上执行挂载命令将NFS共享目录挂载到本地切目录启动切换到NFS挂载目录准备启动相关服务执行启动命令启动NFS相关服务进程查看成功查看NFS服务状态确认挂载成功hdfs高可用部署下载zookeeper下载并解压ZooKeeper安装包。配置文件中加入datanode的ip在ZooKeeper配置文件中添加各datanode节点的IP地址。所有datanode写入id在每个datanode节点的myid文件中写入对应的节点ID。三个node分别为123三个节点的myid文件内容分别为1、2、3用于标识ZooKeeper集群中的节点编号。设置java路径在ZooKeeper配置文件中设置JAVA_HOME环境变量路径。确认Java环境变量配置正确确保ZooKeeper可以正常启动。启动zookeeper在第一个节点上执行启动ZooKeeper的命令。在第二个节点上启动ZooKeeper服务。在第三个节点上启动ZooKeeper服务。查看ZooKeeper进程状态确认服务已启动。查看ZooKeeper集群各节点的运行状态确认集群正常。修改hdfs-site.xml注意master和node的ip以实际为准configurationpropertynamedfs.replication/namevalue3/value/propertypropertynamedfs.nameservices/namevaluemasters/value/propertypropertynamedfs.ha.namenodes.masters/namevalueh1,h2/value/propertypropertynamedfs.namenode.rpc-address.masters.h1/namevalue192.168.36.166:9000/value/propertypropertynamedfs.namenode.http-address.masters.h1/namevalue192.168.36.166:9870/value/propertypropertynamedfs.namenode.rpc-address.masters.h2/namevalue192.168.36.170:9000/value/propertypropertynamedfs.namenode.http-address.masters.h2/namevalue192.168.36.170:9870/value/propertypropertynamedfs.namenode.shared.edits.dir/namevalueqjournal://192.168.36.167:8485;192.168.36.168:8485;192.168.36.169:8485/masters/value/propertypropertynamedfs.journalnode.edits.dir/namevalue/tmp/journaldata/value/propertypropertynamedfs.ha.automatic-failover.enabled/namevaluetrue/value/propertypropertynamedfs.client.failover.proxy.provider.masters/namevalueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value/propertypropertynamedfs.ha.fencing.methods/namevaluesshfenceshell(/bin/true)/value/propertypropertynamedfs.ha.fencing.ssh.private-key-files/namevalue/home/hadoop/.ssh/id_rsa/value/propertypropertynamedfs.ha.fencing.ssh.connect-timeout/namevalue30000/value/property/configuration修改hdfs-site.xml配置文件后确认配置内容正确。node234确认zookeeper集群启动在node2、node3、node4上确认ZooKeeper集群已正常启动。三个DN上依次启动在三个datanode节点上依次执行启动命令。查看查看datanode进程状态确认启动成功。server1上执行确保journalnode运行正常在server1上执行命令确保journalnode运行正常。Namenode数据默认存放在/tmp需要把数据拷贝到h2node5将namenode数据从/tmp目录拷贝到h2node5节点。格式化zookeeper只在h1上执行node1在h1node1上执行格式化ZooKeeper的命令。启动并查看各节点状态,node1,5作为高可用node234作为数据节点启动HDFS服务并查看各节点状态node1和node5作为高可用节点node2、node3、node4作为数据节点。查看namenode进程状态确认高可用节点正常运行。查看datanode进程状态确认数据节点正常运行。浏览器访问此时node1为activenode5为standby浏览器访问HDFS Web界面此时node1为active状态node5为standby状态。查看HDFS集群的datanode列表确认所有数据节点已正常注册。故障切换测试停止node1上namenode执行命令停止node1上的namenode进程进行故障切换测试。此时node5变为activenode1拒绝访问故障切换后node5自动变为active状态node1拒绝访问。再次手动启动node1的namenode此时node1变为standby手动重新启动node1的namenode后node1自动变为standby状态高可用机制正常工作。yarn高可用配置node15配置修改yarn-site.xmlconfigurationpropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME/value/propertypropertynameyarn.resourcemanager.ha.enabled/namevaluetrue/value/propertypropertynameyarn.resourcemanager.cluster-id/namevalueRM_CLUSTER/value/propertypropertynameyarn.resourcemanager.ha.rm-ids/namevaluerm1,rm2/value/propertypropertynameyarn.resourcemanager.hostname.rm1/namevalueserver1/value/propertypropertynameyarn.resourcemanager.hostname.rm2/namevalueserver5/value/propertypropertynameyarn.resourcemanager.recovery.enabled/namevaluetrue/value/propertypropertynameyarn.resourcemanager.store.class/namevalueorg.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore/value/propertypropertynameyarn.resourcemanager.zk-address/namevalue192.168.36.167:2181,192.168.36.168:2181,192.168.36.169:2181/value/property/configuration修改yarn-site.xml配置文件配置YARN高可用相关参数。修改mapred-site.xml修改mapred-site.xml配置文件设置MapReduce相关参数。启动YARN服务并查看进程状态确认服务正常启动浏览器访问8088端口查看YARN Web界面此时node1为active状态。node5为standby查看YARN Web界面确认node5为standby状态YARN高可用配置成功。