hadoop
- 安装hadoop
- 更改文件配置
- 配置core-site.xml
- 配置hadoop-env.sh
- 配置hdfs-site.xml
- 配置mapred-site.xml
- 配置yarn-site.xml
- 配置环境
hadoop安装
安装hadoop
紧接上文，解压过的hadoop內部文件为
再进入etc内部的hadoop
修改hadoop313的权限
在/opt目录下，将hadoop目录下的所有权限更改
chown -R root:root ./hadoop313/
更改文件配置
在此路径下操作
在此目录下新建一个文件夹data,作为namenode上本地的hadoop临时文件夹
配置core-site.xml
vim ./core-site.xml
19 <configuration>
20 <property>
21 <name>fs.defaultFS</name>
22 <value>hdfs://cp145:9000</value>
23 </property>
24 <property>
25 <name>hadoop.tmp.dir</name>
26 <value>/opt/soft/hadoop313/data</value>
27 <description>namenode上本地的hadoop临时文件夹</description>
28 </property>
29 <property>
30 <name>hadoop.http.staticuser.user</name>
31 <value>root</value>
32 </property>
33 <property>
34 <name>io.file.buffer.size</name>
35 <value>131072</value>
36 <description>读写队列缓存：128k</description>
37 </property>
38 <property>
39 <name>hadoop.proxyuser.root.hosts</name>
40 <value>*</value>
41 </property>
42 <property>
43 <name>hadoop.proxyuser.root.groups</name>
44 <value>*</value>
45 </property>
46 </configuration>
配置hadoop-env.sh
vim hadoop-env.sh
54 export JAVA_HOME=/opt/soft/jdk180
配置hdfs-site.xml
vim hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>hadoop中每一个block文件的备份数量</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/soft/hadoop313/data/dfs/name</value>
<description>namenode上存储hdfsq名字空间元数据的目录</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/soft/hadoop313/data/dfs/data</value>
<description>datanode上数据块的物理存储位置目录</description>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
<description>关闭权限验证</description>
</property>
</configuration>
配置mapred-site.xml
vim mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>job执行框架：local,classic or yarn</description>
<final>true</final>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>cp145:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>cp145:19888</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1024</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>1024</value>
</property>
</configuration>
配置yarn-site.xml
vim yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>20000</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<name>yarn.nodemanager.localizer.address</name>
<value>cp145:8040</value>
</property>
<property>
<name>yarn.nodemanager.address</name>
<value>cp145:8050</value>
</property>
<property>
<name>yarn.nodemanager.webapp.address</name>
<value>cp145:8042</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/opt/soft/hadoop313/yarndata/yarn</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/opt/soft/hadoop313/yarndata/log</value>
</property>
</configuration>
配置环境
vim /etc/profile
#HADOOP_HOME
export HADOOP_HOME=/opt/soft/hadoop313
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_JOURNALNODE_USER=root
export HDFS_ZKFC_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
配置完
[root@cp145 hadoop]# source /etc/profile
[root@cp145 hadoop]# echo $HADOOP_HOME
/opt/soft/hadoop313
[root@cp145 hadoop]#
执行即可初始化
hdfs namenode -format
start-all.sh
输入jps，得出一下结果则启动成功
输入 ip:9870成功访问网站
