文章目录
- Fully-Distributed Operation(完全分布模式)== 重点
- 机器环境同步
- 集群规划
- 配置文件修改以及同步
- 步骤0:下面其他步骤涉及修改配置以这里为准(要不然部署使用过程会有很多问题)
- 通用配置(三台节点机器)
- 个性化配置-192.168.19.107
- 个性化配置-192.168.19.108
- 个性化配置-192.168.19.109
- 个性化配置-到时访问dfs、yarn页面的个人电脑
- 步骤1:配置修改【192.168.19.107】
- 步骤2:配置同步
- 步骤3:格式化集群中的dfs主节点
- 步骤4:手动启动(步骤4、5选择其中一个)
- 步骤4.1:192.168.19.107的DFS的相关节点启动
- 步骤4.2:192.168.19.108的DFS的相关节点启动
- 步骤4.3:192.168.19.109的DFS的相关节点启动
- 步骤5:一键启动(无需一个一个节点启动)== (步骤4、5选择其中一个)
- 步骤6:一键启动yarn集群
- 步骤7:启动mapred计算历史过程服务程序historyserver
- 步骤8:访问页面查看集群是否生效
- dfs页面
- yarn页面
Fully-Distributed Operation(完全分布模式)== 重点
参考: https://hadoop.apache.org/docs/r3.3.6/hadoop-project-dist/hadoop-common/ClusterSetup.html
切记、切记、切记: hadoop所有配置文件涉及主机相关,都使用域名进行配置,别使用具体主机的IP,否则会遇到奇奇怪怪的效果,整个搭建,请以步骤0的标准为参考搭建,其他步骤中涉及IP的配置请忽略。
机器环境同步
自行使用我写的多机文件同步脚本,同步一台机器的jdk、hadoop、脚本等文件: https://blog.csdn.net/weixin_39651356/article/details/136593905
集群规划
192.168.19.107 | 192.168.19.108 | 192.168.19.109 | |
---|---|---|---|
HDFS | NameNode、DataNode | DataNode | SecondaryNameNode、DataNode |
YARN | NodeManager | ResourceManager、NodeManager | NodeManager |
配置文件修改以及同步
配置一键同步脚本: https://blog.csdn.net/weixin_39651356/article/details/136593905
步骤0:下面其他步骤涉及修改配置以这里为准(要不然部署使用过程会有很多问题)
通用配置(三台节点机器)
静态DNS文件:/etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.19.107 zookeeper107 hadoop107
192.168.19.108 zookeeper108 hadoop108
192.168.19.109 zookeeper109 hadoop109
集群机器信息:/opt/module/hadoop-3.3.6/etc/hadoop/workers
hadoop107
hadoop108
hadoop109
hadoop组件公共配置:/opt/module/hadoop-3.3.6/etc/hadoop/core-site.xml
<configuration>
<!-- 默认是本地文件协议 file: == NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop107:9000</value>
</property>
<!-- 临时目录 默认/tmp/hadoop-${user.name} == 或者也可以是数据目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-3.3.6/tmp</value>
</property>
<!-- HDFS网页登录时使用的用户,使其具有可以在网页删除文件的权限,默认【dr.who】 -->
<property>
<name>hadoop.http.staticuser.user</name>
<value>root</value>
</property>
</configuration>
hadoop-dfs分布式文件组件配置:/opt/module/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
<configuration>
<!-- 集群设置为1, 默认3 -->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- dfs页面查看文件内容 -->
<!--
<property>
<name>dfs.datanode.http.address</name>
<value>192.168.19.107:9864</value>
</property>
-->
<!-- dfs页面访问地址 -->
<property>
<name>dfs.namenode.http-address</name>
<value>hadoop107:9870</value>
</property>
<!-- namenode的备份节点 == RPC协议改成http协议 -->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop109:9868</value>
</property>
</configuration>
hadoop-yarn节点、资源管理组件配置:/opt/module/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
<configuration>
<!-- Site specific YARN configuration properties == https://hadoop.apache.org/docs/r3.3.6/hadoop-yarn/hadoop-yarn-common/yarn-defa
ult.xml -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!--资源管理节点的IP地址-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop108</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_H
OME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
</property>
<!--
<property>
<name>yarn.timeline-service.hostname</name>
<value>192.168.19.107</value>
</property>
-->
<!-- 日志聚集启动 -->
<!--
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
-->
<!-- 日志聚集的日志保留的时间,单位秒 -->
<!--
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>2592000</value>
</property>
-->
</configuration>
hadoop-mapred节点、资源管理组件配置: /opt/module/hadoop-3.3.6/etc/hadoop/mapred-site.xml
<configuration>
<!-- The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
<!-- mr运行日志采集系统配置 -->
<!--
<property>
<name>mapreduce.jobhistory.address</name>
<value>192.168.19.107:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>192.168.19.107:19888</value>
</property>
-->
</configuration>
添加JDK环境变量: /opt/module/hadoop-3.3.6/etc/hadoop/hadoop-env.sh、/opt/module/hadoop-3.3.6/etc/hadoop/yarn-env.sh、/opt/module/hadoop-3.3.6/etc/hadoop/mapred-env.sh
export JAVA_HOME=/www/server/jdk8/jdk1.8.0_202
系统环境变量配置:/etc/profile
# 文件末尾添加如下配置 == 修改此文件后,需使用 source /etc/profile,使其加载入系统生效
JAVA_HOME=/www/server/jdk8/jdk1.8.0_202
CLASSPATH=$JAVA_HOME/lib/
PATH=$PATH:$JAVA_HOME/bin/:/www/server/liquibase
export PATH JAVA_HOME CLASSPATH
## Hadoop
export HADOOP_HOME=/opt/module/hadoop-3.3.6
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin:/opt/module/script
# 如果hadoop使用root用户启动,必须配这玩意
HDFS_NAMENODE_USER=root
HDFS_DATANODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
YARN_RESOURCEMANAGER_USER=root
YARN_NODEMANAGER_USER=root
export HDFS_NAMENODE_USER HDFS_DATANODE_USER HDFS_SECONDARYNAMENODE_USER YARN_RESOURCEMANAGER_USER YARN_NODEMANAGER_USER
个性化配置-192.168.19.107
需重启电脑生效
机器域名:/etc/hostname
hadoop107
个性化配置-192.168.19.108
需重启电脑生效
机器域名:/etc/hostname
hadoop108
一键同步配置到另外两台机器:同步脚本自行参考我的另一篇博客 https://blog.csdn.net/weixin_39651356/article/details/136593905?spm=1001.2014.3001.5502
xsyncByFile /opt/module/hadoop-3.3.6/etc/hadoop
个性化配置-192.168.19.109
需重启电脑生效
机器域名:/etc/hostname
hadoop109
个性化配置-到时访问dfs、yarn页面的个人电脑
静态DNS:C:\Windows\System32\drivers\etc\hosts
192.168.19.107 hadoop107
192.168.19.107 zookeeper107
192.168.19.108 hadoop108
192.168.19.108 zookeeper108
192.168.19.109 hadoop109
192.168.19.109 zookeeper109
步骤1:配置修改【192.168.19.107】
/opt/module/hadoop-3.3.6/etc/hadoop/hadoop-env.sh
/opt/module/hadoop-3.3.6/etc/hadoop/core-site.xml
<configuration>
<!-- 默认是本地文件协议 file: -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.19.107:9000</value>
</property>
<!-- 临时目录 默认/tmp/hadoop-${user.name} -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-3.3.6/tmp</value>
</property>
</configuration>
/opt/module/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
<configuration>
<!-- 集群设置为1, 默认3 -->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- namenode的备份节点 == RPC协议改成http协议 -->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>192.168.19.109:9868</value>
</property>
</configuration>
/opt/module/hadoop-3.3.6/etc/hadoop/yarn-env.sh
/opt/module/hadoop-3.3.6/etc/hadoop/yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties == https://hadoop.apache.org/docs/r3.3.6/hadoop-yarn/hadoop-yarn-common/yarn-default.xml -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!--资源管理节点的IP地址-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>192.168.19.108</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
</property>
<property>
<name>yarn.timeline-service.hostname</name>
<value>192.168.19.107</value>
</property>
<!-- 日志聚集启动 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<!-- 日志聚集的日志保留的时间,单位秒 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>2592000</value>
</property>
</configuration>
/opt/module/hadoop-3.3.6/etc/hadoop/mapred-env.sh
/opt/module/hadoop-3.3.6/etc/hadoop/mapred-site.xml
<configuration>
<!-- The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
<!-- mr运行日志采集系统配置 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>192.168.19.107:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>192.168.19.107:19888</value>
</property>
</configuration>
步骤2:配置同步
步骤3:格式化集群中的dfs主节点
192.168.19.107
# 格式化主dfs中的数据,确保运行此命令时dfs进程已停止
# 日志目录删除
rm -rf /opt/module/hadoop-3.3.6/logs && rm -rf /opt/module/hadoop-3.3.6/tmp
# 开始格式化
hdfs namenode -format
192.168.19.108、192.168.19.109
# 删除从节点这两个目录即可,无需运行dfs格式化命令
# 日志目录删除
rm -rf /opt/module/hadoop-3.3.6/logs && rm -rf /opt/module/hadoop-3.3.6/tmp
步骤4:手动启动(步骤4、5选择其中一个)
步骤4.1:192.168.19.107的DFS的相关节点启动
cd /opt/module/hadoop-3.3.6/sbin
bash hadoop-daemon.sh start namenode
bash hadoop-daemon.sh start datanode
步骤4.2:192.168.19.108的DFS的相关节点启动
cd /opt/module/hadoop-3.3.6/sbin
bash hadoop-daemon.sh start datanode
步骤4.3:192.168.19.109的DFS的相关节点启动
cd /opt/module/hadoop-3.3.6/sbin
# 可使用命令替代:hdfs --daemon stop namenode secondarynamenode
bash hadoop-daemon.sh start datanode
bash hadoop-daemon.sh start secondarynamenode
步骤5:一键启动(无需一个一个节点启动)== (步骤4、5选择其中一个)
more /opt/module/hadoop-3.3.6/etc/hadoop/workers == 集群服务器的信息进行配置
/opt/module/hadoop-3.3.6/sbin/start-dfs.sh == 脚本一键启动集群
步骤6:一键启动yarn集群
注意: ResourceManager节点在哪台机器,则有哪台机器运行此脚本
start-yarn.sh
步骤7:启动mapred计算历史过程服务程序historyserver
mapred --daemon start historyserver
步骤8:访问页面查看集群是否生效
dfs页面
浏览器访问: http://192.168.19.107:9870/
yarn页面
浏览器访问: http://192.168.19.108:8088/cluster
yarn调用dfs里面的文件使用mapreduce计算开始
hadoop jar /opt/module/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /wordCountData /wordCountDataOutput