【Hadoop实战】Windows环境下编译Hadoop2（2.10.2-R0）

Windows环境下编译Hadoop2（2.10.2-R0）IDE

前提

根据Hadoop源码包解压之后编译帮助文件BUILDING.txt中关于windows的要求来准备环境

----------------------------------------------------------------------------------

Building on Windows

----------------------------------------------------------------------------------
Requirements:

* Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
  tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')

Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK 7.1 is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

The Windows SDK 8.1 is available to download at:

http://msdn.microsoft.com/en-us/windows/bg162891.aspx

Cygwin is neither required nor supported.

本机环境

JDK1.8
Windows 10 64位专业版
maven 3.9.2
git 2.41.0
ProtocolBuffer 2.5.0 这个要求要满足不然有报错（Github地址）
cmake 2.36.4
Visual Studio 2022 Professional
cygwin 安装包
IDEA 2022.2.5

编译方式

使用VS2022的x86 Native Tools Command Prompt for VS 2022命令窗口进行编译，窗口位置C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Visual Studio 2022\Visual Studio Tools\VC

mvn clean package -Pdist -DskipTests

问题小结

安装Visual Studio之后，所有问题几乎全部解决。

问题1

hadoop-annotations: Compilation failure: Compilation failure:
[ERROR] /F:/open-source-code/hadoop/hadoop-release-2.10.2-RC0/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:[20,23] ▒▒▒▒▒com.sun.javadoc▒▒▒▒▒▒

确定好JDK的版本

问题2

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (compile-ms-winutils) on project hadoop-common: Command execution failed.: Cannot run program "msbuild" (in directory "F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-common"): CreateProcess error=2, ϵͳ▒Ҳ▒▒▒ָ▒▒▒▒▒ļ▒▒▒ -> [Help 1]

msbuild是微软的一个编译工具，这里提示找不到msbuild，因此可能是没有安装msbuild或没有配置到环境变量中。下面这样是不行的，还会报其他的错，正确的方式是需要下载Visual Studio，并将msbuild的路径（如，C:\Program Files (x86)\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin）加入到系统环境变量Path中。

# 因为这个目录下有msbuild.exe文件
C:\Windows\Microsoft.NET\Framework\v4.0.30319

问题3

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (compile-ms-winutils) on project hadoop-common: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

方式一

通过VS2022打开，并重定向F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln和F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-common\src\main\native\native.sln

方式二

通过注释的内容来跳过，在hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-common项目下的pom.xml中

          <!--<plugin>
             <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <executions>
              <execution>
                <id>compile-ms-winutils</id>
                <phase>compile</phase>
                <goals>
                  <goal>exec</goal>
                </goals>
                <configuration>
                  <executable>msbuild</executable>
                  <arguments>
                    <argument>${basedir}/src/main/winutils/winutils.sln</argument>
                    <argument>/nologo</argument>
                    <argument>/p:Configuration=Release</argument>
                    <argument>/p:OutDir=${project.build.directory}/bin/</argument>
                    <argument>/p:IntermediateOutputPath=${project.build.directory}/winutils/</argument>
                    <argument>/p:WsceConfigDir=${wsce.config.dir}</argument>
                    <argument>/p:WsceConfigFile=${wsce.config.file}</argument>
                  </arguments>
                </configuration>
              </execution>
              <execution>
                <id>compile-ms-native-dll</id>
                <phase>compile</phase>
                <goals>
                  <goal>exec</goal>
                </goals>
                <configuration>
                  <executable>msbuild</executable>
                  <arguments>
                    <argument>${basedir}/src/main/native/native.sln</argument>
                    <argument>/nologo</argument>
                    <argument>/p:Configuration=Release</argument>
                    <argument>/p:OutDir=${project.build.directory}/bin/</argument>
                    <argument>/p:CustomSnappyPrefix=${snappy.prefix}</argument>
                    <argument>/p:CustomSnappyLib=${snappy.lib}</argument>
                    <argument>/p:CustomSnappyInclude=${snappy.include}</argument>
                    <argument>/p:RequireSnappy=${require.snappy}</argument>
                    <argument>/p:CustomZstdPrefix=${zstd.prefix}</argument>
                    <argument>/p:CustomZstdLib=${zstd.lib}</argument>
                    <argument>/p:CustomZstdInclude=${zstd.include}</argument>
                    <argument>/p:RequireZstd=${require.zstd}</argument>
                    <argument>/p:CustomOpensslPrefix=${openssl.prefix}</argument>
                    <argument>/p:CustomOpensslLib=${openssl.lib}</argument>
                    <argument>/p:CustomOpensslInclude=${openssl.include}</argument>
                    <argument>/p:RequireOpenssl=${require.openssl}</argument>
                  </arguments>
                </configuration>
              </execution>
            </executions>
          </plugin> -->

问题4

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project hadoop-kms: An Ant BuildException has occured: java.net.ConnectException: Connection timed out: connect
[ERROR] around Ant part ...<get skipexisting="true" src="https://archive.apache.org/dist/tomcat/tomcat-8/v8.5.75/bin/apache-tomcat-8.5.75.tar.gz" dest="downloads/apache-tomcat-8.5.75.tar.gz" verbose="true"/>... @ 5:183 in F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-kms\target\antrun\build-main.xml
[ERROR] -> [Help 1]

（有点挫，不知道为啥下不下来，单独从浏览器是可以下载的）

手动下载，放到对应的目录上，然后在注释的内容，在hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-kms项目下的pom.xml中

 <!-- <get
      src="${tomcat.download.url}"
      dest="downloads/apache-tomcat-${tomcat.version}.tar.gz"
      verbose="true" skipexisting="true"/> -->

Apache Hadoop HttpFS模块同样的问题，同样的处理。

问题5

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="cmake">... @ 8:166 in F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR] -> [Help 1]

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo does not exist.
[ERROR] around Ant part ...<copy todir="F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">... @ 14:127 in F:\open-source-code\hadoop\hadoop-release-2.10.2-RC0\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR] -> [Help 1]

应该还是跟native的环境要求有关，这里选择了跳过，在hadoop-release-2.10.2-RC0\hadoop-common-project\hadoop-kms项目下的pom.xml中，failonerror的值改为false，增加新建文件夹RelWithDebInfo的方法。

<configuration>
                  <target>
                    <mkdir dir="native/bin/RelWithDebInfo"/>
                    <condition property="generator" value="Visual Studio 10" else="Visual Studio 10 Win64">
                      <equals arg1="Win32" arg2="${env.PLATFORM}" />
                    </condition>
                    <mkdir dir="${project.build.directory}/native"/>
                    <exec executable="cmake" dir="${project.build.directory}/native"
                          failonerror="false">
                      <arg line="${basedir}/src/ -DGENERATED_JAVAH=${project.build.directory}/native/javah -DJVM_ARCH_DATA_MODEL=${sun.arch.data.model} -DREQUIRE_FUSE=${require.fuse} -G '${generator}'"/>
                    </exec>
                    <exec executable="msbuild" dir="${project.build.directory}/native"
                          failonerror="false">
                      <arg line="ALL_BUILD.vcxproj /nologo /p:Configuration=RelWithDebInfo /p:LinkIncremental=false"/>
                    </exec>
                    <!-- Copy for inclusion in distribution. -->
                    <copy todir="${project.build.directory}/bin">
                      <fileset dir="${project.build.directory}/native/bin/RelWithDebInfo"/>
                    </copy>
                  </target>
                </configuration>

问题6

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (pre-dist) on project hadoop-hdfs-native-client: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]

以下是互联网上的一种解法，但是在我未安装VS2022时并没有解决。然后是通过安装VS2022之后，使用x86 Native Tools Command Prompt for VS 2022命令窗口执行编译命令成功的。

下载安装Cygwin

配置cygwin64\bin到PATH

F:\software\cygwin64\bin

然后打开cygwin64的terminal窗口，执行命令

 mvn package -Pdist -DskipTests -Dtar -e -X -rf :hadoop-hdfs-native-client

导入IDEA

编译成功后使用IEDA打开源码即可，在打开的时候，会下载一些maven插件，可使用阿里云的maven仓库。