文章目录
- gitlab on k8s
- GitLab部署
- helm 安装 postgresql
- helm 安装 redis 集群
- gitlab manifest 部署
- gitlab如何上传项目？
- gitlab监控metrics
- gitlab runner helm 部署
- helm部署问题
- k8s中gitlab exector架构图
- what is Gitlab Runner Helper?
- gitlab如何连接上k8s？ KAS `Kubernetes agent server`
- gitlab CI
- gitlab CI template
- gitlab CI Demo
- gitlab 和gitlab-runner的关系
- runner和exector的关系
- gitlab环境变量
- 参考
gitlab on k8s
GitLab部署
helm 安装 postgresql
helmchar: github
-
安装日志
[root@master2 ~]#helm install gitlib-db -n gitlab /opt/helm/postgresql/ NAME: gitlib-db LAST DEPLOYED: Mon Apr 24 09:05:58 2023 NAMESPACE: gitlab STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: postgresql CHART VERSION: 12.1.9 APP VERSION: 15.1.0 ** Please be patient while the chart is being deployed ** PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster: gitlib-db-postgresql.gitlab.svc.cluster.local - Read/Write connection To get the password for "postgres" run: export POSTGRES_PASSWORD=$(kubectl get secret --namespace gitlab gitlib-db-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d) To connect to your database run the following command: kubectl run gitlib-db-postgresql-client --rm --tty -i --restart='Never' --namespace gitlab --image 10.50.10.185/postgresql/bitnami/postgresql:15.1.0-debian-11-r20 --env="PGPASSWORD=$POSTGRES_PASSWORD" \ --command -- psql --host gitlib-db-postgresql -U postgres -d postgres -p 5432 > NOTE: If you access the container using bash, make sure that you execute "/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash" in order to avoid the error "psql: local user with ID 1001} does not exist" To connect to your database from outside the cluster execute the following commands: kubectl port-forward --namespace gitlab svc/gitlib-db-postgresql 5432:5432 & PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d postgres -p 5432 -
创建gitlab初始化数据库
helm 安装 redis 集群
chart: github
-
安装日志
[root@master2 ~]#helm install gitlib-redis -n gitlab /opt/helm/redis/ NAME: gitlib-redis LAST DEPLOYED: Mon Apr 24 09:08:19 2023 NAMESPACE: gitlab STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: redis CHART VERSION: 17.4.3 APP VERSION: 7.0.8 ** Please be patient while the chart is being deployed ** Redis® can be accessed via port 6379 on the following DNS name from within your cluster: gitlib-redis.gitlab.svc.cluster.local for read only operations For read/write operations, first access the Redis® Sentinel cluster, which is available in port 26379 using the same domain name above. To connect to your Redis® server: 1. Run a Redis® pod that you can use as a client: kubectl run --namespace gitlab redis-client --restart='Never' --image 10.50.10.185/redis/bitnami/redis:7.0.8-debian-11-r0 --command -- sleep infinity Use the following command to attach to the pod: kubectl exec --tty -i redis-client \ --namespace gitlab -- bash 2. Connect using the Redis® CLI: redis-cli -h gitlib-redis -p 6379 # Read only operations redis-cli -h gitlib-redis -p 26379 # Sentinel access To connect to your database from outside the cluster execute the following commands: kubectl port-forward --namespace gitlab svc/gitlib-redis 6379:6379 & redis-cli -h 127.0.0.1 -p 6379
gitlab manifest 部署
参考
官方的chart超级复杂，组件超级多。Gitlab 主要涉及到3个应用：Redis、Postgresql、Gitlab 核心程序，实际上我们只要将这3个应用分别启动起来，然后加上对应的配置就可以很方便的安装 Gitlab 了，我们这里选择使用的镜像不是官方的，而是 Gitlab 容器化中使用非常多的一个第三方镜像：sameersbn/gitlab
-
登录
http://chot-gitlab.prod.com:32100/ 账密: root/xxx
-
使用外部redis 和pg
-
使用sameersbn的gitlab镜像
-
使用nginx ingress 向外暴露服务 kubernetes.io/ingress.class: nginx
-
gitlab-prod.yaml
- 注意需要将pg和redis分别开启30300和30302 nodePort
- 这一步可优化为域名连接避免开太多的nodeport
--- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: gitlab-data-pvc namespace: gitlab spec: accessModes: - ReadWriteOnce resources: requests: storage: 200Gi storageClassName: nfs-storage-179sc --- apiVersion: apps/v1 kind: Deployment metadata: name: gitlab namespace: gitlab labels: name: gitlab spec: replicas: 1 selector: matchLabels: name: gitlab template: metadata: name: gitlab labels: name: gitlab spec: #nodeName: 192.168.102.22 # * containers: - name: gitlab image: 10.50.10.185/gitlab/sameersbn/gitlab:15.8.0-1 imagePullPolicy: IfNotPresent env: - name: TZ value: Asia/Shanghai - name: GITLAB_TIMEZONE value: Beijing - name: GITLAB_SECRETS_DB_KEY_BASE value: long-and-random-alpha-numeric-string # * - name: GITLAB_SECRETS_SECRET_KEY_BASE value: long-and-random-alpha-numeric-string # * - name: GITLAB_SECRETS_OTP_KEY_BASE value: long-and-random-alpha-numeric-string # * - name: GITLAB_ROOT_PASSWORD value: admin123 # * - name: GITLAB_ROOT_EMAIL value: ninesun@126.com # * - name: GITLAB_HOST value: chot-gitlab.prod.com # * - name: GITLAB_PORT value: "30400" - name: GITLAB_SSH_HOST value: k8s-22.host.com # * - name: GITLAB_SSH_PORT value: "30401" - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS value: "true" - name: GITLAB_NOTIFY_PUSHER value: "false" - name: GITLAB_BACKUP_SCHEDULE value: daily - name: GITLAB_BACKUP_TIME value: 01:00 - name: DB_TYPE value: postgres - name: DB_HOST value: gitlib-db-postgresql-hl #headless svc name: gitlib-db-postgresql-hl - name: DB_PORT value: "5432" - name: DB_USER value: postgres - name: DB_PASS value: "postgres" # * - name: DB_NAME value: gitlab_production # 这一步如果需要重新创建数据库，就需要单独建立。否则就把gitlab 的数据存储在默认数据库postgres - name: REDIS_HOST value: gitlib-redis-headless # headless svc name: gitlib-redis-headless - name: REDIS_PORT value: "6379" # 默认端口是6379 ports: - name: http containerPort: 80 - name: ssh containerPort: 22 volumeMounts: - mountPath: /home/git/data name: data livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 180 timeoutSeconds: 5 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 25 timeoutSeconds: 1 volumes: - name: data persistentVolumeClaim: claimName: gitlab-data-pvc --- apiVersion: v1 kind: Service metadata: name: gitlab namespace: gitlab labels: name: gitlab spec: ports: - name: http port: 80 targetPort: http nodePort: 30400 - name: ssh port: 22 targetPort: ssh nodePort: 30401 type: NodePort selector: name: gitlab --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: gitlab namespace: gitlab annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: chot-gitlab.prod.com http: paths: - path: / pathType: Prefix backend: service: name: gitlab port: number: 80-
gitlab需要启动多久？
启动大约需要4mins，期间可能会导致readlines probe检查失败。可以将initialDelaySeconds 调大一些
gitlab如何上传项目？
-
第一步: 添加需要推送项目的服务器的公钥
key的内容就是id_rsa.pub的内容
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-NlPzyq3P-1683179705925)(https://s3-us-west-2.amazonaws.com/secure.notion-static.com/19224c7f-f2c7-4569-ace6-28b0a579233b/Untitled.png)]
-
第二步: git 上传项目
# 添加remote url git remote add origin1 http://chot-gitlab.prod.com:30400/gitlab-instance-f410c318/gitlab-ci-k8s-demo.git # 查看remote url git remote -v origin https://github.com/myysophia/gitlab-ci-k8s-demo.git (fetch) origin https://github.com/myysophia/gitlab-ci-k8s-demo.git (push) origin1 http://chot-gitlab.prod.com:30400/gitlab-instance-f410c318/gitlab-ci-k8s-demo.git (fetch) origin1 http://chot-gitlab.prod.com:30400/gitlab-instance-f410c318/gitlab-ci-k8s-demo.git (push) # 推送代码 git push -u origin1 --all
gitlab监控metrics
这部分可以接入外部的grafana监控面板进行监控
• http://chot-gitlab.prod.com:30400/-/metrics?token=zuqjYZFKMof22VkTRLek
gitlab runner helm 部署
-
helm 部署
chart github:
-
全局runner和项目runner
官方建议自建的实例使用指定的runner，也就是每个项目一个单独的runner
两者如何选择？
-
全局runner token
-
项目runner token
helm部署问题
域名解析问题
- 参考链接
https://todoit.tech/k8s/gitlab-runner/
https://blog.csdn.net/boling_cavalry/article/details/106991576
- 报错处理: Incorrect Usage: flag provided but not defined: -template-config
版本问题
GitLab 社区版 15.8.0 需要使用对应的runner镜像(**https://docs.gitlab.com/runner/)****
-
docker push 10.50.10.185/gitlab/registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v15.8.0
-
如何查看gitlab版本: http://chot-gitlab.prod.com:32100/help
-
config.toml 配置 如何覆盖config.template.toml
非root用户→ 容器中~/.gitlab-runner/config.yaml
concurrent = 10 check_interval = 30 log_level = "info" shutdown_timeout = 0 [session_server] session_timeout = 1800 [[runners]] name = "chot-gitlab-runner-minio-gitlab-runner-55564b6469-k2d6h" url = "http://chot-gitlab.prod.com:32100/" id = 8 token = "vbR7MMTPKSL7dyPALsUN" token_obtained_at = 2023-02-17T03:24:19Z token_expires_at = 0001-01-01T00:00:00Z executor = "kubernetes" [runners.custom_build_dir] [runners.cache] Type = "s3" Shared = true MaxUploadedArchiveSize = 0 [runners.cache.s3] ServerAddress = "chot-minio-web.prod.com:32100" AccessKey = "IwA5ttRQsZlKkkQV" SecretKey = "C07BrPYktE997bMcWUdcHyXQPVPr3mSJ" BucketName = "gitlab" [runners.cache.gcs] [runners.cache.azure] [runners.kubernetes] host = "" bearer_token_overwrite_allowed = false image = "10.50.10.185/gitlab/ubuntu:16.04" namespace = "gitlab" namespace_overwrite_allowed = "" pull_policy = ["if-not-present"] node_selector_overwrite_allowed = "" helper_image = "10.50.10.185/gitlab/ubuntu:16.04" pod_labels_overwrite_allowed = "" service_account_overwrite_allowed = "" pod_annotations_overwrite_allowed = "" [runners.kubernetes.affinity] [runners.kubernetes.pod_security_context] [runners.kubernetes.init_permissions_container_security_context] [runners.kubernetes.init_permissions_container_security_context.capabilities] [runners.kubernetes.build_container_security_context] [runners.kubernetes.build_container_security_context.capabilities] [runners.kubernetes.helper_container_security_context] [runners.kubernetes.helper_container_security_context.capabilities] [runners.kubernetes.service_container_security_context] [runners.kubernetes.service_container_security_context.capabilities] [runners.kubernetes.volumes] [runners.kubernetes.dns_config] [runners.kubernetes.container_lifecycle] -
跑通一个gitlab cicd
https://github.com/yangshun2005/gitlab-cicd
-
gitlab exector 报错处理
-
使用k8s exector 无法拉取镜像
Running with gitlab-runner 15.8.0 (12335144) on chot-gitlab-runner-minio-gitlab-runner-6fbf87f59b-j6bhq S7qPFnrs, system ID: r_vB5NUhtcRQ1R Preparing the "kubernetes" executor 00:00 Using Kubernetes namespace: gitlab Using Kubernetes executor with image ubuntu:16.04 ... Using attach strategy to execute scripts... Preparing environment 00:03 Waiting for pod gitlab/runner-s7qpfnrs-project-1-concurrent-0bc7zl to be running, status is Pending WARNING: Failed to pull image with policy "IfNotPresent": image pull failed: rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: no such host ERROR: Job failed: prepare environment: waiting for pod running: pulling image "ubuntu:16.04": image pull failed: rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: no such host. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information修改runner values的文件
image: registry: 10.50.10.185/gitlab image: ubuntu tag: 16.04 Waiting for pod gitlab/runner-gm-nhepv-project-1-concurrent-057xgd to be running, status is Pending ERROR: Job failed: prepare environment: waiting for pod running: image pull failed: Failed to apply default image tag "map[image:ubuntu registry:10.50.10.185/gitlab tag:16.04]": couldn't parse image reference "map[image:ubuntu registry:10.50.10.185/gitlab tag:16.04]": invalid reference format. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more informationexector不支持这种image的方式
修改为
image: 10.50.10.185/gitlab/ubuntu:16.04
镜像拉取成功后会启动两个容器一个helper 一个 build镜像
如果runner分配给exector的工作1小时没有完成，这些容器就退出了。
error log
- helper 和 build镜像启动后，容器启动报错
/bin/bash: line 1: gitlab-runner-build: command not foundgitlab 流水线报错如下:
OCI runtime exec failed: exec failed: unable to start container process: exec: "gitlab-runner-helper": executable file not found in $PATH: unknown # 从pod yaml文件中找出 其中helper 容器中执行这段脚本 if [ -x /usr/local/bin/bash ]; then exec /usr/local/bin/bash elif [ -x /usr/bin/bash ]; then exec /usr/bin/bash elif [ -x /bin/bash ]; then exec /bin/bash elif [ -x /usr/local/bin/sh ]; then exec /usr/local/bin/sh elif [ -x /usr/bin/sh ]; then exec /usr/bin/sh elif [ -x /bin/sh ]; then exec /bin/sh elif [ -x /busybox/sh ]; then exec /busybox/sh else echo shell not found exit 1 fi查阅官方文档后发现这个helper 镜像还是个专有镜像，下载gitlab runner对应版本的helper, 这个helper镜像就是那个具体干活的，gitlab 的 .gitlab-ci.yml 根据配置的时间间隔把活给gitlab runner。
gitlab runner则是让helper images 去处理。所以刚开始的时候helper的image 用ubuntu是不对的。
具体理解参考Override the helper image部分:
Advanced configuration | GitLab
bitnami/gitlab-runner-helper:15.8.0
Running with gitlab-runner 15.8.0 (12335144) on chot-gitlab-runner-minio-gitlab-runner-75f87cfdbf-d89z6 fhyNaFUz, system ID: r_MDgwGq2YmKGA Preparing the "kubernetes" executor 00:00 Using Kubernetes namespace: gitlab Using Kubernetes executor with image 10.50.10.185/gitlab/ubuntu:16.04 ... Using attach strategy to execute scripts... Preparing environment Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending ContainersNotInitialized: "containers with incomplete status: [init-permissions]" ContainersNotReady: "containers with unready status: [build helper]" ContainersNotReady: "containers with unready status: [build helper]" Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending ContainersNotInitialized: "containers with incomplete status: [init-permissions]" ContainersNotReady: "containers with unready status: [build helper]" ContainersNotReady: "containers with unready status: [build helper]" Waiting for pod gitlab/runner-fhynafuz-project-1-concurrent-0v9gq4 to be running, status is Pending ContainersNotInitialized: "containers with incomplete status: [init-permissions]" ContainersNotReady: "containers with unready status: [build helper]" ContainersNotReady: "containers with unready status: [build helper]" /bin/bash: line 1: gitlab-runner-build: command not found实在没办法，提个issue吧
‣
April 21, 2023
官方更新了gitlab-runner-helper镜像版本
-
k8s中gitlab exector架构图
一图胜千言。
The Kubernetes executor for GitLab Runner | GitLab
what is Gitlab Runner Helper?
helm部署gitlab runner的时候有一个help image。
Gitlab Runner Helper 是与 Gitlab Runner 一起使用的辅助容器。 Gitlab Runner 允许运行 CI/CD 作业并将结果发送回 Gitlab。
gitlab如何连接上k8s？ KAS Kubernetes agent server
gitlab CI
gitlab CI template
lib/gitlab/ci/templates · master · GitLab.org / GitLab · GitLab
gitlab CI Demo
上来不应该直接尝试on k8s，为啥不从最简单的开始呢？
使用 docker部署 gitlab-runner ，注册一个exector 为shell的runner
-
先来个简单的.gitlab-ci.yaml
before_script: - echo "Before script section122333" - echo "For example you might run an update here or install a build dependency" - echo "Or perhaps you might print out some debugging details" after_script: - echo "After script section" - echo "For example you might do some cleanup here" build1: stage: build script: - echo "Do your build heresd" test1: stage: test script: - echo "Do a test here" - echo "For example run a test suite" test2: stage: test script: - echo "Do another parallel test here" - echo "For example run a lint test" deploy1: stage: deploy script: - echo "Do your deploy here" environment: production
第一步: 1. 建立 Docker Volume
目前runner部署在10.50.10.36 理论上哪台有docker环境的都可以的.
$ docker volume create gitlab-runner-config
第二步: 使用创建的卷启动GitLab Runner容器：
-env TZ=CST
docker run -d --name gitlab-runner --restart always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v gitlab-runner-config:/etc/gitlab-runner \
--add-host=chot-gitlab.prod.com:10.50.10.33 \
10.50.10.185/gitlab/gitlab/gitlab-runner:v15.8.0
第三步: register 一个runner，exector 选择shell
root@27172e22cf90:/# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=54 revision=12335144 version=15.8.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
http://chot-gitlab.prod.com:30400/
Enter the registration token:
GR1348941BbkUVr8B1UumMfNx4LrL
Enter a description for the runner:
[27172e22cf90]:
Enter tags for the runner (comma-separated):
Enter optional maintenance note for the runner:
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872
Registering runner... succeeded runner=GR1348941BbkUVr8B
Enter an executor: docker-ssh, virtualbox, docker-ssh+machine, instance, ssh, docker+machine, kubernetes, custom, docker, parallels, shell:
docker
Enter the default Docker image (for example, ruby:2.7):
10.50.10.185/gitlab/ubuntu:16.04
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"
或者一步到位:
gitlab-runner register --url http://chot-gitlab.prod.com:30400/ --registration-tokenGR1348941BbkUVr8B1UumMfNx4LrL --executor shell
第四步 查看CI结果
-
不同exector的runner配置
gitlab-runner register时可以指定不同的exector. 可以打不同的tag，让不同的项目或编译条件使用不同的exector.
gitlab-runner list Runtime platform arch=amd64 os=linux pid=27160 revision=12335144 version=15.8.0 Listing configured runners ConfigFile=/etc/gitlab-runner/config.toml meta-162 Executor=docker Token=_mFCm2xiCnFd8rKaAKFg URL=http://chot-gitlab.prod.com:30400/ meta-162 Executor=shell Token=_yvc4o5ycSACmtyTvBVK URL=http://chot-gitlab.prod.com:30400/ -
CI使用minio作为cache
例如在java项目编译的时候会有很多依赖包需要下载，如果每次都从网络拉取不稳定，这时候如果把依赖包缓存起来，下次构建项目就很快.
[runners.cache] Type = "s3" Shared = true [runners.cache.s3] AccessKey = "IwA5ttRQsZlKkkQV" SecretKey = "C07BrPYktE997bMcWUdcHyXQPVPr3mSJ" BucketName = "gitlab" ServerAddress = "chot-minio-api.prod.com:32100"
gitlab 和gitlab-runner的关系
GitLab CI 之 Runner 的 Executor 該如何選擇？
runner和exector的关系
runner并不是实际干活的，runner 可以指定特定的exector干活，在gitlab中有不同的exector，目前有这几种exector:
- docker-ssh
- virtualbox
- docker-ssh+machine
- instance
- ssh
- docker+machine
- kubernetes
- custom
- docker
- parallels
- shell
可以结合自己的技术栈进行选择，目前我们常使用的是shell 和 docker、k8s
gitlab环境变量
这块都是gitlab预设定的一些环境变量，更多请打开CI_DEBUG_TRACE mode。
例如下面三个环境变量分别是当前gitlab实例名、绝对路径和相对路径。
$ echo "${CI_PROJECT_NAMESPACE}"
gitlab-instance-f410c318
$ echo "${CI_PROJECT_DIR}"
/home/gitlab-runner/builds/hs9MHCAM/0/gitlab-instance-f410c318/gitlab-ci-k8s-demo
$ echo "${CI_PROJECT_PATH}"
gitlab-instance-f410c318/gitlab-ci-k8s-demo
参考
用 GitLab CI 进行持续集成
