3 台机器全部 Anolis7.9当前这台是 master另外两台 node1、node2 用户xiaohuozi后面新建hadoop专门跑大数据所有节点全部执行【前置环境】集群规划表格节点主机名IP角色mastermaster192.168.3.57NameNode、ResourceManager、SecondaryNameNodenode1node1192.168.3.58DataNode、NodeManagernode2node2192.168.3.59DataNode、NodeManager把下面 IP 替换成你三台虚拟机真实 IP一、所有 3 台机器统一执行前置环境1. 修改主机名# master节点执行sudo hostnamectl set-hostname master# node1节点sudo hostnamectl set-hostname node1# node2节点sudo hostnamectl set-hostname node2改完重新登录终端主机名才刷新。2. 配置 hosts 文件三台内容一模一样sudo vim /etc/hosts写入IP 换成你自己三台的 IP192.168.3.57 master192.168.3.58 node1192.168.3.59 node2保存测试ping master、ping node1能通.3.所有节点创建 hadoop 运行用户# 设置密码三台设置同一个密码方便操作useradd hadooppasswd hadoopvim /etc/sudoers 给Hadoop用户增加sudo权限。4. 关闭防火墙、SELinux全部节点如不关闭需要配置防火墙策略# 防火墙(如关闭命令不对请注意版本的区别)sudo systemctl stop firewalldsudo systemctl disable firewalld# SELinuxsudo setenforce 0sudo sed -i s/SELINUXenforcing/SELINUXpermissive/g /etc/selinux/config# 关闭swapsudo swapoff -asudo sed -i /swap/s/^/#/ /etc/fstab5. 所有节点安装 JDK8最好不要使用系统自带的JDK后续运行jps会出问题已踩坑sudo yum install java-1.8.0-openjdk-devel -yreadlink -f /usr/bin/java/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.an7.x86_64/jre/bin/java#在/etc/profile下配置JAVA_HOME设置全局变量如担心影响全局可忽略此处export JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.an7.x86_64export PATH$JAVA_HOME/bin:$PATH#HADOOPexport HADOOP_HOME/home/hadoop/hadoop-3.3.6export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbinsource /etc/profilejava -versionecho $JAVA_HOME#配置单独的用户环境如下。切换 hadoop 用户配置 JAVA_HOMEsu - hadoopvim ~/.bashrc# 在末尾添加export JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.an7.x86_64/jre/export PATH$JAVA_HOME/bin:$PATHsource ~/.bashrcjava -versionecho $JAVA_HOME二、仅 master 节点操作配置 ssh 免密登录hadoop 用户下su - hadoopssh-keygen -t rsa# 一路回车不要设置密码# 分发公钥到三台ssh-copy-id hadoopmasterssh-copy-id hadoopnode1ssh-copy-id hadoopnode2#可能出现的问题Permission deniedhadoop 用户密码输错第一次分发 master 本机公钥时报的Host key verification failed之前 SSH 连过 node1/node2旧主机指纹和现在不匹配ssh 安全校验拦截ssh-copy-id直接失败解决步骤hadoop 用户master 节点操作第一步清理旧主机指纹解决 Host key verification failed# 清除三台旧记录ssh-keygen -R masterssh-keygen -R node1ssh-keygen -R node2第二步手动 ssh 登录一次三台机器缓存新指纹必须做输入yes确认然后输入正确的 hadoop 用户密码ssh hadoopmasterssh hadoopnode1ssh hadoopnode2登录成功后直接exit退出回来。第三步重新分发公钥ssh-copy-id hadoopmasterssh-copy-id hadoopnode1ssh-copy-id hadoopnode2⚠️ 这里一定要输入三台机器 hadoop 用户的正确密码密码三台建议保持一致第四步验证免密是否成功不用输密码才算 okssh hadoopmasterssh hadoopnode1ssh hadoopnode2任意一条如果不需要输入密码直接登录代表免密成功。✅ 测试ssh hadoopnode1不用输密码直接登录才算成功三、master 节点下载解压 Hadoophadoop 用户#新建一个app包用来存放安装包mkdir /home/hadoop/appcd /home/hadoop/app# 阿里云镜像下载(太慢wget https://mirrors.aliyun.com/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz#这个快apache全球CDNwget -c https://dlcdn.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gztar -zxvf hadoop-3.3.6.tar.gz -C /home/hadoop/#配置 hadoop 用户的环境变量 vim ~/.bashrc追加完整export JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.an7.x86_64/jre/export HADOOP_HOME/home/hadoop/hadoop-3.3.6export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoopexport PATH$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATHsource ~/.bashrchadoop version四、master 修改 Hadoop 配置文件 $HADOOP_HOME/etc/hadoop1. hadoop-env.shvim $HADOOP_HOME/etc/hadoop/hadoop-env.sh末尾增加export JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.an7.x86_64export HDFS_NAMENODE_USERhadoopexport HDFS_DATANODE_USERhadoopexport HDFS_SECONDARYNAMENODE_USERhadoopexport YARN_RESOURCEMANAGER_USERhadoopexport YARN_NODEMANAGER_USERhadoop2. core-site.xmlvim $HADOOP_HOME/etc/hadoop/core-site.xmlconfigurationpropertynamefs.defaultFS/namevaluehdfs://master:9000/value/propertypropertynamehadoop.tmp.dir/namevalue/home/hadoop/hadoopdata/value/property/configuration3. hdfs-site.xmlvim $HADOOP_HOME/etc/hadoop/hdfs-site.xmlconfigurationpropertynamedfs.replication/namevalue2/value/propertypropertynamedfs.namenode.secondary.http-address/namevaluemaster:9868/value/property/configuration4. mapred-site.xmlvim $HADOOP_HOME/etc/hadoop/mapred-site.xmlconfigurationpropertynamemapreduce.framework.name/namevalueyarn/value/property/configuration5. yarn-site.xmlvim $HADOOP_HOME/etc/hadoop/yarn-site.xmlconfigurationpropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.resourcemanager.hostname/namevaluemaster/value/property/configuration6. workersdatanode 节点列表vim $HADOOP_HOME/etc/hadoop/workers删掉里面默认localhost写入node1node2五、master 分发整个 hadoop 目录到 node1 node2# -rl 递归 保留软链接rsync -rl /home/hadoop/hadoop-3.3.6 hadoopnode1:/home/hadoop/rsync -rl /home/hadoop/hadoop-3.3.6 hadoopnode2:/home/hadoop/分发完node1、node2 的 hadoop 配置自动同步不用重复写配置六、初始化并启动集群仅 masterhadoop 用户# 格式化namenode【只执行一次重复格式化集群直接报废】hdfs namenode -format# 启动HDFSstart-dfs.sh# 启动YARNstart-yarn.sh验证进程每台机器执行 jpsmasterNameNode、SecondaryNameNode、ResourceManagernode1/node2DataNode、NodeManagerWeb 访问 HDFShttp://192.168.3.57:9870 YARNhttp://192.168.3.57:8088集群停止命令stop-yarn.shstop-dfs.shrm -rf /tmp/hadoop-hadoop-*.pid# 查看进程jps七、新增node或者某台node挂掉之后重新接入如有新增请按照之前的配置先完成基础配置完成基础配置之后下面操作和挂掉重启的node操作相同。下列命令在挂掉重启的机器上运行hdfs --daemon start datanodeyarn --daemon start nodemanagerjps完成之后将主节点的配置重新同步至挂掉的nodescp $HADOOP_HOME/etc/hadoop/{core-site.xml,hdfs-site.xml,yarn-site.xml,mapred-site.xml} hadoopnode2:$HADOOP_HOME/etc/hadoop/输入密码
