Docker搭建jenkins

DevOps概念
Docker部署Jenkins
- 制作Jenkins镜像
- - Dockerfile及所依赖的脚本
  - build镜像
- 利用docker-compose部署jenkins
配置Jenkins
- 管理员密码
- 插件安装
- 系统配置
- 全局工具配置
- - MAVEN 配置
  - JDK 配置
  - GIT 配置
  - MAVEN 配置
Jenkins + Maven + Git 自动化编译
- 找到token生成界面
- 生成git的token
- 进行jenkins配置
配置 spark 编译任务
- 新建任务
- 配置任务
- - 源码管理
  - 构建

DevOps概念

在这里插入图片描述
DevOps 一词的来自于 Development 和 Operations 的组合，突出重视软件开发人员和运维人员的沟通合作，通过自动化流程来使得软件构建、测试、发布更加快捷、频繁和可靠。DevOps 其实包含了三个部分：开发、测试和运维。换句话 DevOps 希望做到的是软件产品交付过程中IT工具链的打通，使得各个团队减少时间损耗，更加高效地协同工作。

DevOps 强调的是高效组织团队之间如何通过自动化的工具协作和沟通来完成软件的生命周期管理，从而更快、更频繁地交付更稳定的软件

如果从字面上来理解，DevOps 只是Dev（开发人员）+Ops（运维人员），实际上，它是一组过程、方法与系统的统称，其概念从2009 年首次提出发展到现在，内容非常丰富，有理论也有实践，包括组织文化、自动化、精益、反馈和分享等不同方面。

Docker部署Jenkins

Jenkins 项目产生两个发行线, 长期支持版本 (LTS) 和每周更新版本

两个版本都以 .war 文件, 原生包, 安装程序, 和 Docker 容器的形式分发

最新下载地址：https://jenkins.io/zh/download/

制作Jenkins镜像

Dockerfile及所依赖的脚本

Dockerfile

FROM centos:centos7.9.2009

MAINTAINER 笑起来真好看

ARG TARGETARCH

RUN yum update -y && yum install wget vim fontconfig fontconfig-devel fontconfig.i686 git gcc gcc-c++ \
    autoconf automake libtool curl lzo-devel zlib-devel openssl openssl-devel ncurses-devel python3 python3-pip rpm-build \
    snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop libXtst zlib zlib-devel lz4 lz4-devel yasm help2man epel-release -y && \
    mkdir -p /opt/software && mkdir -p /opt/compile && \
    wget -O /opt/software/cmake.tar.gz https://cmake.org/files/v3.15/cmake-3.15.0.tar.gz --no-check-certificate && \
    wget -O /opt/software/protobuf.tar.gz https://hub.nuaa.cf/protocolbuffers/protobuf/archive/refs/tags/v3.7.1.tar.gz --no-check-certificate && \
    wget -O /opt/software/snappy.tar.gz https://hub.nuaa.cf/google/snappy/archive/refs/tags/1.1.3.tar.gz --no-check-certificate && \
    wget -O /opt/software/zstd.tar.gz https://codeload.github.com/facebook/zstd/tar.gz/refs/tags/v1.4.10 --no-check-certificate && \
    wget -O /opt/software/nasm.tar.gz https://www.nasm.us/pub/nasm/releasebuilds/2.14.02/nasm-2.14.02.tar.gz --no-check-certificate && \
    wget -O /opt/software/isa-l.tar.gz https://codeload.github.com/intel/isa-l/tar.gz/refs/tags/v2.30.0 --no-check-certificate && \
    wget -O jenkins.war https://get.jenkins.io/war-stable/2.346.1/jenkins.war --no-check-certificate && \
    wget -O /opt/software/scala.tgz https://downloads.lightbend.com/scala/2.12.17/scala-2.12.17.tgz && \
    mkdir -p /usr/share/jenkins/ && mv jenkins.war /usr/share/jenkins/ && \
    yum clean all && rm -rf /var/cache/yum/*

# 双架构的 jdk 版本
ADD jdk-8u371-linux-${TARGETARCH}.tar.gz /opt/software/
ADD nodejs.tar.gz /opt/software/
ADD https://github.com/krallin/tini/releases/download/v0.19.0/tini-${TARGETARCH} /usr/bin/tini

COPY install_soft.sh /opt/software/
COPY check_soft.sh /opt/software/
COPY build_hadoop.sh /opt/compile/
COPY jenkins.sh /usr/local/bin/
COPY jenkins-support /usr/local/bin/

RUN /bin/bash -c /opt/software/install_soft.sh && yum clean all && rm -rf /var/cache/yum/* && chmod +x /usr/bin/tini && mkdir -p /usr/share/jenkins/ref

ENV LANG=C.UTF-8
ENV JENKINS_HOME=/var/jenkins_home
ENV JENKINS_SLAVE_AGENT_PORT=50000
ENV REF=/usr/share/jenkins/ref
ENV JENKINS_VERSION=2.405
ENV JENKINS_UC=https://updates.jenkins.io
ENV JENKINS_UC_EXPERIMENTAL=https://updates.jenkins.io/experimental
ENV JENKINS_INCREMENTALS_REPO_MIRROR=https://repo.jenkins-ci.org/incrementals
ENV COPY_REFERENCE_FILE_LOG=/var/jenkins_home/copy_reference_file.log
ENV NODE_HOME=/opt/software/nodejs
ENV JAVA_HOME=/opt/software/jdk1.8.0_371
ENV SCALA_HOME=/opt/software/scala
ENV MAVEN_HOME=/opt/software/maven
ENV PATH=$PATH:$MAVEN_HOME/bin:$JAVA_HOME/bin:$NODE_HOME/bin:$SCALA_HOME/bin

Entrypoint ["/usr/bin/tini", "--", "/usr/local/bin/jenkins.sh"]

其中 install_soft.sh 是自动化安装hadoop编译所需要的依赖软件包.需要可以私信我

jenkins.sh 是镜像启动的入口脚本

#! /bin/bash -e

: "${JENKINS_WAR:="/usr/share/jenkins/jenkins.war"}"
: "${JENKINS_HOME:="/var/jenkins_home"}"
: "${COPY_REFERENCE_FILE_LOG:="${JENKINS_HOME}/copy_reference_file.log"}"
: "${REF:="/usr/share/jenkins/ref"}"
touch "${COPY_REFERENCE_FILE_LOG}" || { echo "Can not write to ${COPY_REFERENCE_FILE_LOG}. Wrong volume permissions?"; exit 1; }
echo "--- Copying files at $(date)" >> "$COPY_REFERENCE_FILE_LOG"
find "${REF}" \( -type f -o -type l \) -exec bash -c '. /usr/local/bin/jenkins-support; for arg; do copy_reference_file "$arg"; done' _ {} +

# if `docker run` first argument start with `--` the user is passing jenkins launcher arguments
if [[ $# -lt 1 ]] || [[ "$1" == "--"* ]]; then

  # shellcheck disable=SC2001
  effective_java_opts=$(sed -e 's/^ $//' <<<"$JAVA_OPTS $JENKINS_JAVA_OPTS")

  # read JAVA_OPTS and JENKINS_OPTS into arrays to avoid need for eval (and associated vulnerabilities)
  java_opts_array=()
  while IFS= read -r -d '' item; do
    java_opts_array+=( "$item" )
  done < <([[ $effective_java_opts ]] && xargs printf '%s\0' <<<"$effective_java_opts")

  readonly agent_port_property='jenkins.model.Jenkins.slaveAgentPort'
  if [ -n "${JENKINS_SLAVE_AGENT_PORT:-}" ] && [[ "${effective_java_opts:-}" != *"${agent_port_property}"* ]]; then
    java_opts_array+=( "-D${agent_port_property}=${JENKINS_SLAVE_AGENT_PORT}" )
  fi

  readonly lifecycle_property='hudson.lifecycle'
  if [[ "${JAVA_OPTS:-}" != *"${lifecycle_property}"* ]]; then
    java_opts_array+=( "-D${lifecycle_property}=hudson.lifecycle.ExitLifecycle" )
  fi

  if [[ "$DEBUG" ]] ; then
    java_opts_array+=( \
      '-Xdebug' \
      '-Xrunjdwp:server=y,transport=dt_socket,address=*:5005,suspend=y' \
    )
  fi

  jenkins_opts_array=( )
  while IFS= read -r -d '' item; do
    jenkins_opts_array+=( "$item" )
  done < <([[ $JENKINS_OPTS ]] && xargs printf '%s\0' <<<"$JENKINS_OPTS")

  exec java -Duser.home="$JENKINS_HOME" "${java_opts_array[@]}" -jar "${JENKINS_WAR}" "${jenkins_opts_array[@]}" "$@"
fi

# As argument is not jenkins, assume user wants to run a different process, for example a `bash` shell to explore this image
exec "$@"

jenkins-support 内容如下：

#!/bin/bash -eu

: "${REF:="/usr/share/jenkins/ref"}"

# compare if version1 < version2
versionLT() {
    local v1; v1=$(echo "$1" | cut -d '-' -f 1 )
    local q1; q1=$(echo "$1" | cut -s -d '-' -f 2- )
    local v2; v2=$(echo "$2" | cut -d '-' -f 1 )
    local q2; q2=$(echo "$2" | cut -s -d '-' -f 2- )
    if [ "$v1" = "$v2" ]; then
        if [ "$q1" = "$q2" ]; then
            return 1
        else
            if [ -z "$q1" ]; then
                return 1
            else
                if [ -z "$q2" ]; then
                    return 0
                else
                    [  "$q1" = "$(echo -e "$q1\n$q2" | sort -V | head -n1)" ]
                fi
            fi
        fi
    else
        [  "$v1" = "$(echo -e "$v1\n$v2" | sort -V | head -n1)" ]
    fi
}

# returns a plugin version from a plugin archive
get_plugin_version() {
    local archive; archive=$1
    local version; version=$(unzip -p "$archive" META-INF/MANIFEST.MF | grep "^Plugin-Version: " | sed -e 's#^Plugin-Version: ##')
    version=${version%%[[:space:]]}
    echo "$version"
}

# Copy files from /usr/share/jenkins/ref into $JENKINS_HOME
# So the initial JENKINS-HOME is set with expected content.
# Don't override, as this is just a reference setup, and use from UI
# can then change this, upgrade plugins, etc.
copy_reference_file() {
    f="${1%/}"
    b="${f%.override}"
    rel="${b#"$REF/"}"
    version_marker="${rel}.version_from_image"
    dir=$(dirname "${rel}")
    local action;
    local reason;
    local container_version;
    local image_version;
    local marker_version;
    local log; log=false
    if [[ ${rel} == plugins/*.jpi ]]; then
        container_version=$(get_plugin_version "$JENKINS_HOME/${rel}")
        image_version=$(get_plugin_version "${f}")
        if [[ -e $JENKINS_HOME/${version_marker} ]]; then
            marker_version=$(cat "$JENKINS_HOME/${version_marker}")
            if versionLT "$marker_version" "$container_version"; then
                if ( versionLT "$container_version" "$image_version" && [[ -n $PLUGINS_FORCE_UPGRADE ]]); then
                    action="UPGRADED"
                    reason="Manually upgraded version ($container_version) is older than image version $image_version"
                    log=true
                else
                    action="SKIPPED"
                    reason="Installed version ($container_version) has been manually upgraded from initial version ($marker_version)"
                    log=true
                fi
            else
                if [[ "$image_version" == "$container_version" ]]; then
                    action="SKIPPED"
                    reason="Version from image is the same as the installed version $image_version"
                else
                    if versionLT "$image_version" "$container_version"; then
                        action="SKIPPED"
                        log=true
                        reason="Image version ($image_version) is older than installed version ($container_version)"
                    else
                        action="UPGRADED"
                        log=true
                        reason="Image version ($image_version) is newer than installed version ($container_version)"
                    fi
                fi
            fi
        else
            if [[ -n "$TRY_UPGRADE_IF_NO_MARKER" ]]; then
                if [[ "$image_version" == "$container_version" ]]; then
                    action="SKIPPED"
                    reason="Version from image is the same as the installed version $image_version (no marker found)"
                    # Add marker for next time
                    echo "$image_version" > "$JENKINS_HOME/${version_marker}"
                else
                    if versionLT "$image_version" "$container_version"; then
                        action="SKIPPED"
                        log=true
                        reason="Image version ($image_version) is older than installed version ($container_version) (no marker found)"
                    else
                        action="UPGRADED"
                        log=true
                        reason="Image version ($image_version) is newer than installed version ($container_version) (no marker found)"
                    fi
                fi
            fi
        fi
        if [[ ! -e $JENKINS_HOME/${rel} || "$action" == "UPGRADED" || $f = *.override ]]; then
            action=${action:-"INSTALLED"}
            log=true
            mkdir -p "$JENKINS_HOME/${dir}"
            cp -pr "${f}" "$JENKINS_HOME/${rel}";
            # pin plugins on initial copy
            touch "$JENKINS_HOME/${rel}.pinned"
            echo "$image_version" > "$JENKINS_HOME/${version_marker}"
            reason=${reason:-$image_version}
        else
            action=${action:-"SKIPPED"}
        fi
    else
        if [[ ! -e $JENKINS_HOME/${rel} || $f = *.override ]]
        then
            action="INSTALLED"
            log=true
            mkdir -p "$JENKINS_HOME/${dir}"
            cp -pr "$(realpath "${f}")" "$JENKINS_HOME/${rel}";
        else
            action="SKIPPED"
        fi
    fi
    if [[ -n "$VERBOSE" || "$log" == "true" ]]; then
        if [ -z "$reason" ]; then
            echo "$action $rel" >> "$COPY_REFERENCE_FILE_LOG"
        else
            echo "$action $rel : $reason" >> "$COPY_REFERENCE_FILE_LOG"
        fi
    fi
}

# Retries a command a configurable number of times with backoff.
#
# The retry count is given by ATTEMPTS (default 60), the initial backoff
# timeout is given by TIMEOUT in seconds (default 1.)
#
function retry_command() {
  local max_attempts=${ATTEMPTS-3}
  local timeout=${TIMEOUT-1}
  local success_timeout=${SUCCESS_TIMEOUT-1}
  local max_success_attempt=${SUCCESS_ATTEMPTS-1}
  local attempt=0
  local success_attempt=0
  local exitCode=0

  while (( attempt < max_attempts ))
  do
    set +e
    "$@"
    exitCode=$?
    set -e

    if [[ $exitCode == 0 ]]
    then
      success_attempt=$(( success_attempt + 1 ))
      if (( success_attempt >= max_success_attempt))
      then
        break
      else
        sleep "$success_timeout"
        continue
      fi
    fi

    echo "$(date -u '+%T') Failure ($exitCode) Retrying in $timeout seconds..." 1>&2
    sleep "$timeout"
    success_attempt=0
    attempt=$(( attempt + 1 ))
    timeout=$(( timeout ))
  done

  if [[ $exitCode != 0 ]]
  then
    echo "$(date -u '+%T') Failed in the last attempt ($*)" 1>&2
  fi

  return $exitCode
}

build镜像

docker build --platform=linux/arm64 -t centos-jenkins:v1 --no-cache .

利用docker-compose部署jenkins

version: '3'
services:
  jenkins:
    image: centos-jenkins:v1
    container_name: jenkins
    ports:
      - "8080:8080"
      - "50000:50000"
    networks:
      - back
    volumes:
      - ./jenkins:/var/jenkins_home
      - ./maven:/opt/software/maven
    privileged: true
    environment:
      - TZ="Asia/Shanghai"
    restart: always

networks:
  back:
    driver: bridge

使用 docker-compose up -d 就可以启动了

启动成功后就可以本地访问 jenkins访问地址

在这里插入图片描述

配置Jenkins

管理员密码

在这里插入图片描述
输入我们在控制台中看到的密钥，如果找不到可以到/root/.jenkins/secrets/initialAdminPassword中查看密钥

插件安装

网络好的情况下，直接选择安装推荐的插件。
在这里插入图片描述
必要的插件：

汉化插件 Chinese
Git
Maven Integration

安装完插件后，重启jenkins

系统配置

在这里插入图片描述

全局工具配置

MAVEN 配置

配置自定义的 settings 文件
在这里插入图片描述

JDK 配置

设置自定义的 JAVA_HOME 目录
在这里插入图片描述

GIT 配置

设置 git 执行文件路径
在这里插入图片描述

MAVEN 配置

设置全局 maven 的home路径
在这里插入图片描述

Jenkins + Maven + Git 自动化编译

在2021年8月13日git移除了通过用户名和密码在外部访问源码仓库的方式。

问题描述如下：

Caused by: hudson.plugins.git.GitException: Command “/usr/bin/git fetch --tags --progress https://github.com/xxxxx/hadoop.git +refs/heads/:refs/remotes/origin/” returned status code 128:

目前想要在 jenkins 中拉取 github 中的源代码，需要在 github 网页中生成 TOKEN 信息

找到token生成界面

点击个人头像，点击 Settings 设置界面
侧边栏点击 Developer settings 进入到开发者设置界面
进入到 Personal access tokens (classic) 界面
根据配置项产生新的 token

生成git的token

根据提示，填写 token 的名称，过期时间和作用域，具体根据个人实际需求进行选择

token 仅显示一次，一定要记录下来

在这里插入图片描述

进行jenkins配置

创建 Username with Password 的类型的凭证

填写用户名
填写 token 信息

配置 spark 编译任务

新建任务

选择自由风格的软件项目

在这里插入图片描述

配置任务

源码管理

在这里插入图片描述

构建

在这里插入图片描述
其中构建脚本如下：

#!/bin/sh 
export BUILD_ID=dontKillMe 
source ~/.bash_profile sh
${WORKSPACE}/dev/make-distribution.sh --pip --tgz -Phive -Phive-2.3 -Phive-thriftserver -Pscala-2.12 -Phadoop-3.2 -Pmesos -Pyarn -Pkubernetes

希望对正在查看文章的您有所帮助，记得关注、评论、收藏，谢谢您