k8s上搭建devops环境

一、gitlab

1.安装gitlab

# 下载安装包
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-15.9.1-ce.0.el7.x86_64.rpm

# 安装
rpm -i gitlab-ce-15.9.1-ce.0.el7.x86_64.rpm

# 编辑 vi /etc/gitlab/gitlab.rb 文件
# 修改 external_url 访问路径 http://192.168.50.76:28080/
# 其他配置修改如下
gitlab_rails['time_zone'] = 'Asia/Shanghai'
puma['worker_processes'] = 2
sidekiq['max_concurrency'] = 8
postgresql['shared_buffers'] = "128MB"
postgresql['max_worker_processes'] = 4
prometheus_monitoring['enable'] = false

# 更新配置并重启
gitlab-ctl reconfigure
gitlab-ctl restart

2.页面配置

# 查看默认密码
cat /etc/gitlab/initial_root_password

访问url http://192.168.50.76:28080/

root/TkT5HbRv/zkPl8mh1lFZMrDzlat9n3eVM7CYJQyHaNk=
# 登录后修改默认密码 > 右上角头像 > Perferences > Password

# 修改系统配置：点击左上角三横 > Admin
# Settings > General > Account and limit > 取消 Gravatar enabled > Save changes

# 关闭用户注册功能
# Settings > General > Sign-up restrictions > 取消 Sign-up enabled > Save changes

# 开启 webhook 外部访问
# Settings > Network > Outbound requests > Allow requests to the local network from web hooks and services 勾选

# 设置语言为中文（全局）
# Settings > Preferences > Localization > Default language > 选择简体中文 > Save changes

# 设置当前用户语言为中文
# 右上角用户头像 > Preferences > Localization > Language > 选择简体中文 > Save changes

3.配置Secret

# 创建 gitlab 默认用户名密码 secret
echo root > ./username
echo wolfcode > password

kubectl create secret generic git-user-pass --from-file=./username --from-file=./password -n kube-devops

4.为项目配置webhook

进入项目点击侧边栏设置 > Webhooks 进入配置即可

URL：在 jenkins 创建 pipeline 项目后
触发来源：
推送事件：表示收到新的推送代码就会触发
标签推送事件：新标签推送才会触发
评论：根据评论决定触发
合并请求事件：创建、更新或合并请求触发

添加成功后，可以在下方点击测试按钮查看 jenkins 是否成功触发构建操作

5.卸载

# 停止服务
gitlab-ctl stop

# 卸载 rpm 软件（注意安装的软件版本是 ce 还是 ee）
rpm -e gitlab-ce

# 查看进程
ps -ef|grep gitlab
# 干掉第一个 runsvdir -P /opt/gitlab/service log 进程

# 删除 gitlab 残余文件
find / -name *gitlab* | xargs rm -rf
find / -name gitlab | xargs rm -rf

二、harbor

0.安装docker-compose

注意docker跟docker-compose版本对应关系

我们这里docker是26.1.4

#下载

wget https://github.com/docker/compose/releases/download/v2.27.0/docker-compose-linux-x86_64

#移动到/usr/local/bin下

mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose

#赋予执行权限

chmod +x /usr/local/bin/docker-compose

测试是否安装成功

docker-compose --version

1.安装harbor

# 下载 harbor 安装包
wget https://github.com/goharbor/harbor/releases/download/v2.5.0/harbor-offline-installer-v2.5.0.tgz
# 解压

tar -zxvf harbor-v2.5.0.tar.gz

#修改配置

cp harbor.yml.tmpl harbor.yml

vi harbor.yml

执行 sh install.sh 就行

查看是否启动成功

docker-compose ls

docker ps -a | grep harbor

访问url: 192.168.50.76:8858 admin/wolfcode

2.配置Secret

# 创建 harbor 访问账号密码（需要将下访问的配置信息改成你自己的）
kubectl create secret docker-registry harbor-secret --docker-server=192.168.50.76:8858 --docker-username=admin --docker-password=root -n kube-devops

这里密码输入错了，应该是wolfcode，后面来改。

查看是否创建完成

kubectl get secret -n kube-devops

3.配置docker(集群中每个节点)

vi /etc/docker/daemon.json

加入"insecure-registries":["192.168.50.76:8858"]

systemctl daemon-reload

systemctl restart docker

4.验证是否生效

docker login -uadmin 192.168.50.76:8858

三、SonarQube

1.安装

# 进入 /opt/k8s/devops

pgsql.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: kube-devops
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "managed-nfs-storage"
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-sonar
  namespace: kube-devops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-sonar
  template:
    metadata:
      labels:
        app: postgres-sonar
    spec:
      containers:
      - name: postgres-sonar
        image: postgres:14.2
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_DB
          value: "sonarDB"
        - name: POSTGRES_USER
          value: "sonarUser"
        - name: POSTGRES_PASSWORD 
          value: "123456"
        volumeMounts:
          - name: data
            mountPath: /var/lib/postgresql/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: postgres-data
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-sonar
  namespace: kube-devops
  labels:
    app: postgres-sonar
spec:
  type: NodePort
  ports:
  - name: postgres-sonar
    port: 5432
    targetPort: 5432
    protocol: TCP
  selector:
    app: postgres-sonar

sonarqube.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sonarqube-data
  namespace: kube-devops
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "managed-nfs-storage"
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sonarqube
  namespace: kube-devops
  labels:
    app: sonarqube
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sonarqube
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox:1.28.4
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      containers:
      - name: sonarqube
        image: sonarqube
        ports:
        - containerPort: 9000
        env:
        - name: SONARQUBE_JDBC_USERNAME
          value: "sonarUser"
        - name: SONARQUBE_JDBC_PASSWORD
          value: "123456"
        - name: SONARQUBE_JDBC_URL
          value: "jdbc:postgresql://postgres-sonar:5432/sonarDB"
        livenessProbe:
          httpGet:
            path: /sessions/new
            port: 9000
          initialDelaySeconds: 60
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /sessions/new
            port: 9000
          initialDelaySeconds: 60
          periodSeconds: 30
          failureThreshold: 6
        volumeMounts:
        - mountPath: /opt/sonarqube/conf
          name: data
        - mountPath: /opt/sonarqube/data
          name: data
        - mountPath: /opt/sonarqube/extensions
          name: data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: sonarqube-data 
---
apiVersion: v1
kind: Service
metadata:
  name: sonarqube
  namespace: kube-devops
  labels:
    app: sonarqube
spec:
  type: NodePort
  ports:
  - name: sonarqube
    port: 9000
    targetPort: 9000
    protocol: TCP
  selector:
    app: sonarqube

部署

kubectl apply -f sonarqube/

查看

kubectl get pod -n kube-devops

kubectl logs -f sonarqube-58669db48f-kd6c9 -n kube-devops

sonaqube url :http://192.168.50.76:31217/ admin/admin123

2.生成服务token

# 登录到 sonarqube 后台，点击头像 > MyAccount > Security > Generate Tokens > generate 生成 token 并复制

name输入jenkins

点击生成可得到

sqa_bcf05eab4a43b9f21718f13e653f37d98b488a9e

3.创建webhook服务

# 点击顶部菜单栏的配置 > 配置（小三角） > 网络调用

Name：wolfcode-jenkins
URL：http://<sonar ip>:<sonar port>/sonarqube-webhook/

4.创建项目

# SonarQube 顶部菜单栏 Projects > Create new project > 配置基础信息并保存 > Provide a token > Generate 生成 token > Continue

# 分别选择 Java / Maven 后，按照脚本配置 Jenkinsfile 中的 sonar 配置信息
mvn sonar:sonar -Dsonar.projectKey=k8s-cicd-demo

四、Jenkins

1.构建带maven环境的jenkins镜像

dockerfile

FROM jenkins/jenkins:2.462.2-jdk11
ADD ./apache-maven-3.9.0-bin.tar.gz /usr/local/
ADD ./sonar-scanner-4.8.0.2856-linux /usr/local/sonar-scanner

ENV MAVEN_HOME=/usr/local/apache-maven-3.9.0
ENV PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

USER root

RUN echo "jenkins ALL=NOPASSWD: ALL" >> /etc/sudoers
USER jenkins

另外需要准备的是maven的安装包和解压后的sonar安装包

# 构建带 maven 环境的 jenkins 镜像
docker build -t 192.168.50.76:8858/wolfcode/jenkins-maven:v1 .

# 登录 harbor
docker login -uadmin 192.168.50.76:8858
密码是wolfcode

# 推送镜像到 harbor
docker push 192.168.50.76:8858/wolfcode/jenkins-maven:v1

2.安装jenkins

# 进入 jenkins 目录，安装 jenkins
kubectl apply -f manifests/

jenkins-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: kube-devops
spec:
  storageClassName: managed-nfs-storage
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

jenkins-configmap.yaml

apiVersion: v1 
kind: ConfigMap 
metadata: 
  name: mvn-settings
  namespace: kube-devops
  labels: 
    app: jenkins-server
data: 
  settings.xml: |- 
    <?xml version="1.0" encoding="UTF-8"?>
    <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
        <localRepository>/var/jenkins_home/repository</localRepository>
        <servers>
                <server>
                        <id>releases</id>
                        <username>admin</username>
                        <password>wolfcode</password>
                </server>
                <server>
                        <id>snapshots</id>
                        <username>admin</username>
                        <password>wolfcode</password>
                </server>
        </servers>

        <mirrors>
                <mirror>
                        <id>releases</id>
                        <name>nexus maven</name>
                        <mirrorOf>*</mirrorOf>
                        <url>http://192.168.50.76:8868/repository/maven-public/</url>
                </mirror>
        </mirrors>

        <pluginGroups>
                <pluginGroup>org.sonarsource.scanner.maven</pluginGroup>
        </pluginGroups>
        <profiles>
                <profile>
                        <id>releases</id>
                        <activation>
                                <activeByDefault>true</activeByDefault>
                                <jdk>1.8</jdk>
                        </activation>
                        <properties>
                                <sonar.host.url>http://sonarqube:9000</sonar.host.url>
                        </properties>

                        <repositories>
                                <repository>
                                        <id>repository</id>
                                        <name>Nexus Repository</name>
                                        <url>http://192.168.50.76:8868/repository/maven-public/</url>
                                        <releases>
                                                <enable>true</enable>
                                        </releases>
                                        <snapshots>
                                                <enable>true</enable>
                                        </snapshots>
                                </repository>
                        </repositories>
                </profile>
        </profiles>
    </settings>

jenkins-serviceAccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: kube-devops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: kube-devops

jenkins-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: kube-devops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server
  template:
    metadata:
      labels:
        app: jenkins-server
    spec:
      serviceAccountName: jenkins-admin
      imagePullSecrets: 
        - name: harbor-secret # harbor 访问 secret
      containers:
        - name: jenkins
          image: 192.168.50.76:8858/wolfcode/jenkins-maven:v1
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            runAsUser: 0 # 使用 root 用户运行容器
          resources:
            limits:
              memory: "2Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "500m"
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home
            - name: docker
              mountPath: /run/docker.sock
            - name: docker-home
              mountPath: /usr/bin/docker
            - name: mvn-setting
              mountPath: /usr/local/apache-maven-3.9.0/conf/settings.xml
              subPath: settings.xml
            - name: daemon
              mountPath: /etc/docker/daemon.json
              subPath: daemon.json
            - name: kubectl
              mountPath: /usr/bin/kubectl
      volumes:
        - name: kubectl
          hostPath:
            path: /usr/bin/kubectl
        - name: jenkins-data
          persistentVolumeClaim:
              claimName: jenkins-pvc
        - name: docker
          hostPath:
            path: /run/docker.sock # 将主机的 docker 映射到容器中
        - name: docker-home
          hostPath:
            path: /usr/bin/docker
        - name: mvn-setting
          configMap:
            name: mvn-settings
            items:
            - key: settings.xml
              path: settings.xml
        - name: daemon
          hostPath: 
            path: /etc/docker/

jenkins-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: kube-devops
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector:
    app: jenkins-server
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080

# 查看是否运行成功
kubectl get po -n kube-devops

# 查看 service 端口，通过浏览器访问
kubectl get svc -n kube-devops

# 查看容器日志，获取默认密码
kubectl logs -f pod名称 -n kube-devops

url 192.168.50.76:32351 admin/admin

查看密码 cat /var/jenkins_home/secrets/initialAdminPassword

第一次登陆进去会有安装插件推荐，其中github不用，其他默认推荐的都可以装一下

3.安装插件

如果有些插件搜不到，可以去官网下载 https://plugins.jenkins.io/

jenkins界面插件安装入口：dashboard---->manage jenkins---->插件管理

记得以下插件安装完成后点击重启

3.1 build authorization token root

构建授权 token

3.2 gitlab

gitlab 配置插件

3.3 sonarqube scanner

代码质量审查工具
安装好插件后重启jenkins然后配置

1.在 Dashboard > 系统管理 >凭据管理>system>全局凭据>添加凭据

类型：scret text

scret: sqa_bcf05eab4a43b9f21718f13e653f37d98b488a9e 这就是上面sonarqube界面里面生成的token

ID: sonarqube-token

2.在 Dashboard > 系统管理 > Configure System 下面配置 SonarQube servers

Name：sonarqube # 注意这个名字要在 Jenkinsfile 中用到
Server URL：http://sonarqube:9000
Server authentication token：创建 credentials 配置为从 sonarqube 中得到的 token

3.(这步没做)进入系统管理 > 全局工具配置 > SonarQube Scanner > Add SonarQube Scanner
Name：sonarqube-scanner
自动安装：取消勾选
SONAR_RUNNER_HOME：/usr/local/sonar-scanner-cli

3.4 node and label parameter

节点标签参数配置

3.5 kubernetes

jenkins + k8s 环境配置

进入 Dashboard > 系统管理 > 节点管理 > Configure Clouds 页面

配置 k8s 集群
名称：kubernetes
点击 Kubernetes Cloud details 继续配置
Kubernetes 地址：
如果 jenkins 是运行在 k8s 容器中，直接配置服务名即可
https://kubernetes.default
如果 jenkins 部署在外部，那么则不仅要配置外部访问 ip 以及 apiserver 的端口（6443），还需要配置服务证书
Jenkins 地址：
如果部署在 k8s 集群内部：http://jenkins-service.kube-devops
如果在外部：http://192.168.50.76:32479（换成你们自己的）

配置完成后保存即可