Kubernetes实现CI与CD配置
一、基本介绍
基于 Kubernetes 实现 CI/CD 配置,其实和往常那些 CI/CD 配置并没有太大区别。都是通过 提交代码,拉取代码,构建代码,发布代码来实现的。 只不过要是通过 K8s 来实现的话,则是需要将构建好的代码打包成镜像,通过镜像的方式来运行。
CI/CD 流程图:
开发将代码提交代码仓库后,我们便可以通过在 Jenkins 上配置脚本或是 Pipline 的方式来实现代码发布,其中发布有两种方式,一种是通过手动发布,另外一种可以通过 WebHook 插件来实现提交代码便自动发布(生产环境不建议自动发布)
脚本内容一般分为:克隆代码、编译代码、将编译好的代码打包成镜像、运行镜像几个步骤。
二、基于 Kubernetes 实现 CI/CD 配置
1.配置 GitLab
1)安装 Docker-Compose
[root@k8s-master01 ~]# wget "https://github.com/docker/compose/releases/download/v2.3.2/docker-compose-$(uname -s)-$(uname -m)" -O /usr/local/bin/docker-compose
[root@k8s-master01 ~]# chmod +x /usr/local/bin/docker-compose
[root@k8s-master01 ~]# docker-compose --version
2)安装 GitLab
[root@k8s-master01 ~]# vim docker-compose.yml
version: '3'
services:
web:
image: 'gitlab/gitlab-ce:14.8.5-ce.0'
restart: always
hostname: 192.168.59.130
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://192.168.59.130'
ports:
- '1080:80'
- '1443:443'
- '1022:22'
volumes:
- '/app/gitlab/config:/etc/gitlab'
- '/app/gitlab/logs:/var/log/gitlab'
- '/app/gitlab/data:/var/opt/gitlab'
[root@k8s-master01 ~]# docker-compose up -d
2.配置 Jenkins
1)安装 NFS 存储,并配置共享目录
[root@k8s-master01 ~]# yum -y install nfs-utils rpcbind
[root@k8s-master01 ~]# echo "/app/jenkins *(rw,sync,no_root_squash)" > /etc/exports
[root@k8s-master01 ~]# mkdir /app/jenkins
[root@k8s-master01 ~]# systemctl start rpcbind nfs
2)创建 PV 和 PVC
[root@k8s-master01 ~]# vim jenkins-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
nfs:
server: 192.168.59.130
path: /app/jenkins
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pvc
spec:
resources:
requests:
storage: 10Gi
accessModes:
- ReadWriteMany
[root@k8s-master01 ~]# kubectl create -f jenkins-pv.yaml
3)创建 RBAC 授权
[root@k8s-master01 ~]# vim jenkins-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins-sa
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: jenkins-cr
rules:
- apiGroups: ["extensions","apps"]
resources: ["deployments"]
verbs: ["create","delete","get","list","watch","patch","update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create","delete","get","list","watch","patch","update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","update"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: jenkins-crb
roleRef:
kind: ClusterRole
name: jenkins-cr
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: jenkins-sa
namespace: default
[root@k8s-master01 ~]# kubectl create -f jenkins-sa.yaml
4)创建 StatefulSet
[root@k8s-master01 ~]# vim jenkins-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: jenkins
spec:
serviceName: jenkins
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
name: "jenkins"
labels:
app: jenkins
spec:
serviceAccountName: jenkins-sa
containers:
- name: jenkins
image: jenkins/jenkins:lts
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
- containerPort: 50000
volumeMounts:
- name: jenkins
mountPath: /var/jenkins_home
volumes:
- name: jenkins
persistentVolumeClaim:
claimName: jenkins-pvc
[root@k8s-master01 ~]# chown -R 1000 /app/jenkins
[root@k8s-master01 ~]# kubectl create -f jenkins-statefulset.yaml
5)创建 Service
[root@k8s-master01 ~]# vim jenkins-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: jenkins
spec:
type: NodePort
ports:
- name: http
port: 8080
targetPort: 8080
nodePort: 30080
- name: agent
port: 50000
targetPort: 50000
nodePort: 30090
selector:
app: jenkins
[root@k8s-master01 ~]# kubectl create -f jenkins-svc.yaml
6)配置 Jenkins
[root@k8s-master01 ~]# cat /app/jenkins/secrets/initialAdminPassword
a303d66e915e4ee5b26648a64fdff4be
http://192.168.59.131:30080/
我们这里安装推荐的插件即可,后面有需求可以再进行安装
3.实现 CI/CD 配置
1)在 Jenkins 宿主机上创建 SSH 密钥
[root@k8s-master01 ~]# ssh-keygen -t rsa
[root@k8s-master01 ~]# cat ~/.ssh/id_rsa.pub
将生成的公钥 私钥 移动到 nfs的jenkins的挂载目录下
或者 直接进入jenkins容器内生成ssh密钥
最终都是保存在/app/jenkins/.ssh文件夹 下面
2)将公钥上传到 Gitee上
3)将仓库克隆到本地
[root@k8s-master01 ~]# git clone git@gitee.com:JavaBigDataStudy/go-devops.git
4)编写 Go 代码
[root@k8s-master01 ~]# cd go-devops
[root@k8s-master01 test]# vim main.go
package main
import (
"fmt"
"net/http"
)
func HelloHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello World")
}
func main() {
http.HandleFunc("/", HelloHandler)
http.ListenAndServe(":8080", nil)
}
5)编写 Dockerfile
[root@k8s-master01 go-devops]# vim Dockerfile
FROM golang:1.16 as builder
ENV GO111MODULE=on \
GOPROXY=https://goproxy.cn,direct
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o main main.go
FROM busybox:1.28.4
WORKDIR /app
COPY --from=builder /app/ .
EXPOSE 8080
CMD ["./main"]
[root@k8s-master01 go-devops]# docker build -t test-web-server:devops-$(date +%Y-%m-%d-%H-%M-%S) .
6)提交代码
[root@k8s-master01 go-devops]# git add . # 提交到暂存区
[root@k8s-master01 go-devops]# git config --global user.email "dukng-jker@163.com" # 配置用户邮箱
[root@k8s-master01 go-devops]# git commit -m "This is test CI/CD" # 提交到本地仓库
[root@k8s-master01 go-devops]# git push # 推送到远程仓库
7)创建 Deployment 和 Service
[root@k8s-master01 ~]# vim test-web-server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-web-server
spec:
replicas: 1
selector:
matchLabels:
app: test-web-server
template:
metadata:
labels:
app: test-web-server
spec:
containers:
- name: test-web-server
image: test-web-server:devops-2023-05-20-15-18-13
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: test-web-server
spec:
type: NodePort
ports:
- name: test-web-server
port: 8080
targetPort: 8080
nodePort: 30188
selector:
app: test-web-server
[root@k8s-master01 ~]# kubectl create -f test-web-server.yaml
8)编写 Jenkins 发版脚本
[root@k8s-master01 ~]# vim jenkins-test.sh
#!/bin/bash
# 固定时间格式
Second=$(date +%Y-%m-%d-%H-%M-%S)
# 备份旧的镜像
Image=$(kubectl -s https://192.168.59.131:6443 describe pod | grep Image: | awk '{print $2}' | grep test)
echo $Image > /opt/test-image-$Second
# 克隆代码
cd /home/shell
if [ -d go-devops ];
then
mv go-devops /opt/test-devops-$Second
git clone git@gitee.com:JavaBigDataStudy/go-devops.git
else
git clone git@gitee.com:JavaBigDataStudy/go-devops.git
fi
# 发布新的镜像
cd /home/shell/go-devops && docker build -t test-web-server:devops-$Second .
# 上传到镜像仓库
if [ $? -eq 0 ];
then
docker tag test-web-server:devops-$Second 192.168.59.130:5000/test-web-server:devops-$Second
docker push 192.168.59.130:5000/test-web-server:devops-$Second
else
exit 1 # 退出 (防止运行下面命令)
fi
# 替换镜像
sed -i 's/image:.*/image: 192.168.59.130:5000\/test-web-server:devops-'$Second'/g' /home/shell/test-web-server.yaml
# 重启应用
kubectl delete -f /home/shell/test-web-server.yaml
kubectl create -f /home/shell/test-web-server.yaml
[root@k8s-master01 ~]# chmod +x jenkins-test.sh
上面这个脚本有两步需要注意:
上传到镜像仓库: 如果你们没有自己的镜像仓库,可以选择调整脚本, 提前安装registry镜像仓库
替换镜像: 我们上面配置的脚本是针对单个模块的,多个模块可以根据 for 循环来实现。
4.验证
1)在 Jenkins 上安装 SSH 插件
安装 SSH 插件的原因是因为,我们这个 Jenkins 是容器安装的,而脚本是在宿主机写的,所以通过远程到宿主机来运行脚本。
2)配置远程主机的用户名和密码
3)创建 Jenkins 私钥凭证(类型选择:SSH Username with private key)
4)配置 Jenkins 流水线
5)修改代码
6)在 Jenkins 上发布
访问应用服务
总结:
通过上面的操作,实现了从git提交代码到发布K8S应用服务的整个CICD流程