mac-hadoop3.3.6 源码构建以及踩坑记录

news2024/11/28 10:41:22

1. 为什么需要构建源码

因为hadoop的可执行文件是在专门的机器上编译的其中native库不一定能适用于每个机器导致在启动hadoop过程中出现烦人的警告 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

如何说为了解决警告的话可以直接使用已经编译好的库 GitHub地址： https://github.com/silent-night-no-trace/mac-native-hadoop-library
另外使用说明也在文档中可以找到

特殊说明：解决这个警告问题可以直接参考文档： https://blog.csdn.net/a15835774652/article/details/135649394

2. 构建准备

下载源码 https://github.com/apache/hadoop/releases/tag/rel%2Frelease-3.3.6
直接下载链接地址：https://github.com/apache/hadoop/archive/refs/tags/rel/release-3.3.6.zip
导入到idea 或者其他的工具

3. 环境准备

解压后的源码导入idea 在根目录下有个BUILDING.txt
我们直接参考这个编译

Unix System
JDK 1.8 （最好是）
Maven 3.3 or later
Protocol Buffers 3.7.1 (if compiling native code)
CMake 3.1 or newer (if compiling native code)
Zlib devel (if compiling native code)
Cyrus SASL devel (if compiling native code)
One of the compilers that support thread_local storage: GCC 4.8.1 or later, Visual Studio,
Clang (community version), Clang (version for iOS 9 and later) (if compiling native code)
openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
Doxygen ( if compiling libhdfspp and generating the documents )
Internet connection for first build (to fetch all Maven and Hadoop dependencies)
python (for releasedocs)
bats (for shell code testing)
Node.js / bower / Ember-cli (for YARN UI v2 building)

因为我们要编译native源码所以标记 if compiling native code 这个都需要安装
可以直接使用homebrew来安装但是需要注意强调的版本

Building on macOS (without Docker)

Installing required dependencies for clean install of macOS 10.14:

Install Xcode Command Line Tools
$ xcode-select --install
Install Homebrew
$ /usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”
Install OpenJDK 8
$ brew tap AdoptOpenJDK/openjdk
$ brew cask install adoptopenjdk8
Install maven and tools
$ brew install maven autoconf automake cmake wget
Install native libraries, only openssl is required to compile native code,
you may optionally install zlib, lz4, etc.
$ brew install openssl
Protocol Buffers 3.7.1 (required to compile native code)
$ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz
$ mkdir -p protobuf-3.7 && tar zxvf protobuf-java-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7
$ cd protobuf-3.7
$ ./configure
$ make
$ make check
$ make install
$ protoc --version

Note that building Hadoop 3.1.1/3.1.2/3.2.0 native code from source is broken
on macOS. For 3.1.1/3.1.2, you need to manually backport YARN-8622. For 3.2.0,
you need to backport both YARN-8622 and YARN-9487 in order to build native code.

4. 源码修改

修改主pom 仓库修改为阿里云仓库
项目根目录下的 pom.xml 在 repositories 第一个位置新增阿里云仓库

<repository>
   <id>maven-ali</id>
   <url>https://maven.aliyun.com/repository/public</url>
   <releases>
     <enabled>true</enabled>
   </releases>
   <snapshots>
     <enabled>true</enabled>
     <updatePolicy>always</updatePolicy>
     <checksumPolicy>fail</checksumPolicy>
   </snapshots>
 </repository>

修改 hadoop-common-project/hadoop-common/src/main/native/src/exception.c

# 删除下面代码
|| defined(__GLIBC_PREREQ) && __GLIBC_PREREQ(2, 32)

修改 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c

# 在头部添加一行
#include <sys/ioctl.h>

修改 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MapOutputCollector.cc
需要在PRIu64宏和字符串文字之间添加空格（注意每个PRIu64地方都需要按照如下进行改动）示例如下
```
# 修改前
LOG("%s-spill: { id: %d, collect: %"PRIu64" ms, "
# 修改后 
LOG("%s-spill: { id: %d, collect: %" PRIu64 " ms, "
```
修改 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/StringUtil.cc
需要在PRIu64宏和字符串文字之间添加空格（注意每个PRIu64地方都需要按照如下进行改动）示例如下
```
# 修改后
snprintf(tmp, 32, "%"PRIu64, v);
# 修改后 
snprintf(tmp, 32, "%" PRIu64, v);
```

5. 执行构建

# brew安装的openssl 快捷查看安装位置
brew --prefix openssl

# 官方推荐的执行命令/usr/local/opt/openssl 修改为自己的openssl路径
mvn package -Pdist,native -DskipTests -Dmaven.javadoc.skip \
    -Dopenssl.prefix=/usr/local/opt/openssl
# 遇到编译错误的话，修改后 可以直接从 指定项目开始编译 
#  直接编译 hadoop-yarn-server-nodemanager的native库
mvn clean package -Dmaven.javadoc.skip=true -Pdist,native -DskipTests -Dtar -rf :hadoop-yarn-server-nodemanager
# 直接编译 map reduce的native库
mvn clean package -Dmaven.javadoc.skip=true -Pnative -DskipTests -Dtar -rf :hadoop-mapreduce-client-nativetask

# 增加 编译的详细日志 方便排查问题  
mvn clean package -e -X -Pdist,native -DskipTests -Dmaven.javadoc.skip -Dopenssl.prefix=/usr/local/bin/openssl

然后就等待编译大概10来分钟
最后这里
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc.c
会报错提示 linux/loop.h 不存在这肯定是不存在的 linux特有的库
然后注释到在运行可能就会提示 openssl/evp.h 不存在可能是个bug 这个路径已经正确配置了。也是找不到

虽然说编译没有完成但是我们要hadoop-common 和 hadoop-hdfs的 native文件已经编译好了

另外 mapreduce的native 编译可以直接编译 hadoop-mapreduce-client-nativetask 这个项目
执行如下命令

mvn clean package -Dmaven.javadoc.skip=true -Pnative -DskipTests -Dtar -rf :hadoop-mapreduce-client-nativetask

当编译到如下图的时候即可停止
在这里插入图片描述
完成编译图如下还是推荐提前就停止可以看到完整的编译这部分 mapreduce native 耗时还是比较久的

当mapreduce-examples 执行完后说明编译的native库已经通过测试可以使用了

6. 使用

到这里我们完成了 3个主要项目的native库分别是hadoop-common hadoop-hdfs hadoop-mapreduce 其中 yarn项目因为涉及到修改源码并未完成编译

建议将原有的native库 $HADOOP_HOME/lib/native/ 备份下

# 拷贝 hadoop-common 的native文件 到hadoop 目录
cp hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/*  $HADOOP_HOME/lib/native/

# 拷贝hadoop hdfs native到 hadoop的lib/native目录
cp hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/target/usr/local/lib/*   $HADOOP_HOME/lib/native/

# 拷贝 mapreduce的native库
cp hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/target/usr/local/lib/* $HADOOP_HOME/lib/native/

然后在进行检查本地库支持信息

# 查看 Hadoop 支持的本地库信息：
$HADOOP_HOM/Ebin/hadoop checknative -a

效果图如下发现已经hadoop本地的库已经被加载到了以及 zlib zstd
在这里插入图片描述

7. 遗留的编译问题

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc.c 中的 linux/loop.h 文件不存在以及相应的代码修改
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc.c 中的 openssl/evp.h 文件不存在，但是本机上 openssl3已经正确配置了。