Hadoop伪分布式搭建

news2024/12/28 5:23:58

1 配置SSH免密登录

1.生成密钥

# ssh-keygen -t rsa

注意：需要经过4次回车

查看密钥及公钥

# cd /root/.ssh

拷贝公钥

# cp id_rsa.pub authorized_keys

2 测试本地免密登录

2 下载Hadoop安装包

使用wget命令从华为云上下载Hadoop安装文件

# wget -P /opt https://mirrors.huaweicloud.com/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

3 安装Hadoop

1.在opt目录下创建software目录

# mkdir /opt/software

2.把hadoop2.7.7文件解压到/opt/software目录下

# tar -zxvf /opt/hadoop-2.7.7.tar.gz -C /opt/software

3.把hadoop-2.7.7目录修改为hadoop

# mv /opt/software/hadoop-2.7.7 /opt/software/hadoop

4.配置Hadoop环境变量

# vim /etc/profile

环境变量立即生效

# source /etc/profile

5.对Hadoop进行配置

修改hadoop-env.sh文件，添加对jdk的支持

# vim /opt/software/hadoop/etc/hadoop/hadoop-env.sh

修改core-site.xml

# vim /opt/software/hadoop/etc/hadoop/core-site.xml

<configuration>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/opt/software/hadoop/tmp</value>
                <description>配置临时文件存放地址</description>
        </property>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://master:8020</value>
                <description>配置HDFS服务地址及端口</description>
        </property>
</configuration>

修改hdfs-site.xml

# vim /opt/software/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
</configuration>

配置mapred-site.xml

复制模板文件

# cp /opt/software/hadoop/etc/hadoop/mapred-site.xml.template /opt/software/hadoop/etc/hadoop/ mapred-site.xml

配置文件内容

# vim /opt/software/hadoop/etc/hadoop/mapred-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
</configuration>

配置yarn-site.xml

# vim /opt/software/hadoop/etc/hadoop/yarn-site.xml

<configuration>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:8032</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>3072</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

6.格式化hdfs