文章目录
- GitOps 介绍
- 如何将业务代码构建为容器镜像?
- 如何将容器镜像部署到K8s?
- K8s如何实现自动扩容和自愈?
- 1.传统的扩容和自愈
- 2.k8s自愈机制
- 3.k8s弹性扩容
- 如何借助GitOps实现应用秒级自动发布和回滚?
- 1.传统 K8s 应用发布流程
- 2.从零搭建 GitOps 发布工作流
- 1.安装fluxcd
- 2.本地创建fluxcd目录并创建deploy
- 3.为 FluxCD 创建仓库连接信息
- 4.检查 GitRepository 对象
- 5.为 FluxCD 创建部署策略
- 6.体验gitops自动发布
- 7.体验gitops自动回滚
GitOps 介绍
传送门: https://blog.csdn.net/zfw_666666/article/details/126158696
如何将业务代码构建为容器镜像?
前提:需要对Docker容器技术有一定的了解。
#业务代码示例:
root@node1:~# cat app.py
from flask import Flask
import os
app = Flask(__name__)
app.run(debug=True)
@app.route('/')
def hello_world():
return 'Hello, my v1 version docker images!! ' + os.getenv("HOSTNAME") + ''
一、编写dockerfile
root@node1:~# cat Dockerfile
# syntax=docker/dockerfile:1
FROM python:3.8-slim-buster
RUN apt-get update && apt-get install -y procps vim apache2-utils && rm -rf /var/lib/apt/lists/*
WORKDIR /app
RUN pip3 install Flask==2.3.0 -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
COPY app.py .
ENV FLASK_ENV=development
CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]
二、构建
$ docker build -t hello-world-flask .
三、验证
$ docker run -d -p 8000:5000 hello-world-flask:latest
四、推送至个人仓库
docker image tag ...
docker push ...
docker pull ccr.ccs.tencentyun.com/app-public/hello-world-flask-amd:latest
如何将容器镜像部署到K8s?
前提:需要本地有k8s集群环境及对k8s资源对象有一定的了解。
1.编写Manifest
root@node1:~# cat flask-dp.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: hello-world-flask
name: hello-world-flask
spec:
ports:
- port: 5000
protocol: TCP
targetPort: 5000
selector:
app: hello-world-flask
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: hello-world-flask
name: hello-world-flask
spec:
replicas: 2
selector:
matchLabels:
app: hello-world-flask
template:
metadata:
labels:
app: hello-world-flask
spec:
containers:
- image: ccr.ccs.tencentyun.com/app-public/hello-world-flask-amd:latest
imagePullPolicy: IfNotPresent
name: hello-world-flask
resources:
limits:
cpu: 100m
requests:
cpu: 100m
2.查看和访问 Pod
root@node1:~# kubectl get pods,svc |grep flask
pod/hello-world-flask-9d8967487-68kzq 1/1 Running 1 (2d2h ago) 2d3h
pod/hello-world-flask-9d8967487-mbwn9 1/1 Running 1 (2d2h ago) 2d3h
service/hello-world-flask ClusterIP 10.233.47.55 <none> 5000/TCP 2d6h
root@node1:~# curl http://10.233.47.55:5000
Hello, my v1 version docker images!! hello-world-flask-9d8967487-68kzq
K8s如何实现自动扩容和自愈?
1.传统的扩容和自愈
在 VM 时代,我们的业务以进程的方式运行在虚拟机上,并由虚拟机对外提供服务。随着业务规模的扩大,我们需要支撑更多的访问流量,这时业务扩容就成了首先要考虑的问题。
在公有云环境下,VM 架构最典型的一种扩容方式是 弹性伸缩组。 意思是通过对虚拟机内存、CPU 等监控指标配置伸缩阈值,实现动态地自动伸缩。此外,我们一般还会结合虚拟机镜像、负载均衡器等云产品一并使用,如下图所示。
在这个架构中,负载均衡器是集群的唯一入口,它在接受访问流量后,一般会将流量通过加权轮训的方式转发到后端集群。负载均衡器一般是直接使用云厂商的产品,有一些团队也会自建高可用的 Nginx 作为集群入口。为了保证伸缩组节点的业务一致性,弹性伸缩组的所有 VM 都使用同一个虚拟机镜像。
其次,要在 VM 粒度实现业务自愈,常见的方案是使用 Crontab 定时检查业务进程或者通过守护进程的方式来运行,例如 Node PM2。
但是,这种架构有一些显而易见的缺陷。最大的问题有两个:(1)扩容慢;(2)负载均衡无法感知业务健康情况。
2.k8s自愈机制
你希望自愈解决什么问题?
我想,你可能最希望自愈能够帮我们解决服务自动重启的问题。也就是说,当业务进程意外中断,或者节点产生故障时,系统可以快速识别,自动重启并恢复服务。其次,你可能还会希望自愈能够自动转移故障,也就是让业务不健康的节点不接收流量,保证用户体验。
听起来是不是很棒,而 K8s 的自动自愈功能都可以帮你解决上面的这些问题,运维童鞋终于能摆脱 7*24 小时 Oncall 了。
【自愈演示】
现阶段,只需要知道三件事:
1. Pod 会被 Deployment 控制器管理起来,例如创建和销毁等;
2. Service 相当于弹性伸缩组的负载均衡器,它能以加权轮训的方式将流量转发到多个 Pod 副本上;
3. Ingress 相当于集群的外网访问入口。此处博主使用的是Traefik ingressroute
用svc去不断请求
root@node1:~# while true; do sleep 2; curl http://10.233.47.55:5000; echo -e '\n'$(date);done
Hello, my v1 version docker images!! hello-world-flask-9d8967487-mbwn9
Thu Jul 13 17:06:26 CST 2023
Hello, my v1 version docker images!! hello-world-flask-9d8967487-68kzq
Thu Jul 13 17:06:28 CST 2023
Hello, my v1 version docker images!! hello-world-flask-9d8967487-mbwn9
Thu Jul 13 17:06:30 CST 2023
Hello, my v1 version docker images!! hello-world-flask-9d8967487-68kzq
Thu Jul 13 17:06:32 CST 2023
root@node1:~# kubectl exec -it hello-world-flask-9d8967487-mbwn9 -- bash -c "killall python3"
模拟其中的一个 Pod 宕机,观察返回内容。会发现所有的请求流量都被转发到了没有故障的 Pod, 也就是说,故障成功地被转移了!等待几秒钟后pod重启恢复后,重新加入到了负载均衡接收外部流量。
梳理一下全过程。首先, K8s 感知到了业务 Pod 故障,立刻进行了故障转移并隔离了有故障的 Pod,并将请求转发到了其他健康的 Pod 中。随后重启了有故障的 Pod,最后将重启后的 Pod 加入到了负载均衡并开始接收外部请求。这些过程都是自动化完成的。
3.k8s弹性扩容
前提:自动扩容依赖于 K8s Metric Server 提供的监控指标,首先确保本地k8s环境中已经安装了。
一、通过 kubectl autoscale 命令来为 Deployment 创建自动扩容策略
root@node1:~# cat flask-hpa.yaml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: hello-world-flask
spec:
maxReplicas: 10
minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: hello-world-flask
targetCPUUtilizationPercentage: 10 #CPU 使用率阈值
注:
要使自动扩容生效,还需要为刚才部署的 hello-world-flask Deployment 设置资源配额,可以通过patch或者yaml中配置
root@node1:~# kubectl patch deployment hello-world-flask --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/resources", "value": {"requests": {"memory": "100Mi", "cpu": "100m"}}}]'
二、模拟业务高峰期场景,使用 ab 命令来创建并发请求
root@node1:~# ab -c 10 -n 5000 http://10.233.47.55:5000/
root@node1:~# kubectl get pods |grep flask
hello-world-flask-9d8967487-68kzq 1/1 Running 1 (2d2h ago) 2d4h
hello-world-flask-9d8967487-9mkg8 1/1 Terminating 0 36s
hello-world-flask-9d8967487-dsk7m 1/1 Terminating 0 21s
hello-world-flask-9d8967487-jcnwz 1/1 Terminating 0 36s
hello-world-flask-9d8967487-mbwn9 1/1 Running 1 (2d2h ago) 2d4h
hello-world-flask-9d8967487-rj2wp 0/1 Terminating 0 6s
hello-world-flask-9d8967487-srrk6 0/1 Terminating 0 6s
hello-world-flask-9d8967487-zjrtw 1/1 Terminating 0 21s
可以通过参数 --watch 表示持续监听 Pod 状态变化。在 ab 压力测试的过程中,会不断创建新的 Pod 副本, 这说明 K8s 已经感知到了 Pod 的业务压力,并且正在自动进行横向扩容。
如何借助GitOps实现应用秒级自动发布和回滚?
1.传统 K8s 应用发布流程
- 使用 kubectl set image 命令;
- 修改本地的 Manifest,
kubectl apply -f xx.yaml
- 修改集群内 Manifest
kubectl edit deploy xxx -n xx
2.从零搭建 GitOps 发布工作流
通俗来说,GitOps 就是以 Git 版本控制为理念的 DevOps 实践。
我们会将 Manifest 存储在 Git 仓库中作为期望状态,一旦修改并提交了 Manifest ,那么 GitOps 工作流就会 自动比对 Git 仓库和集群内工作负载的实际差异,并进行部署。
官网:https://fluxcd.io/flux/get-started/
要实现 GitOps 工作流,首先我们需要一个能够帮助我们监听 Git 仓库变化,自动部署的工具。此处以 FluxCD 为例一步步构建出一个 GitOps 工作流。
1.安装fluxcd
root@node1:~# kubectl apply -f https://ghproxy.com/https://raw.githubusercontent.com/lyzhang1999/resource/main/fluxcd/fluxcd.yaml
root@node1:~# kubectl get all -n flux-system
NAME READY STATUS RESTARTS AGE
pod/helm-controller-65d97f86f-hct7z 1/1 Running 4 (2d4h ago) 2d7h
pod/image-automation-controller-75cc9d6964-v9vpk 1/1 Running 3 (2d4h ago) 2d7h
pod/image-reflector-controller-6c99d4c47b-m8blb 1/1 Running 1 (2d4h ago) 2d7h
pod/kustomize-controller-6595b7976c-zqgrz 1/1 Running 1 (2d4h ago) 2d7h
pod/notification-controller-56f8f7f86b-cc786 1/1 Running 5 (2d4h ago) 2d7h
pod/source-controller-846457b955-7hzfc 1/1 Running 1 (2d4h ago) 2d7h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/notification-controller ClusterIP 10.233.11.156 <none> 80/TCP 2d7h
service/source-controller ClusterIP 10.233.4.62 <none> 80/TCP 2d7h
service/webhook-receiver ClusterIP 10.233.47.213 <none> 80/TCP 2d7h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/helm-controller 1/1 1 1 2d7h
deployment.apps/image-automation-controller 1/1 1 1 2d7h
deployment.apps/image-reflector-controller 1/1 1 1 2d7h
deployment.apps/kustomize-controller 1/1 1 1 2d7h
deployment.apps/notification-controller 1/1 1 1 2d7h
deployment.apps/source-controller 1/1 1 1 2d7h
NAME DESIRED CURRENT READY AGE
replicaset.apps/helm-controller-65d97f86f 1 1 1 2d7h
replicaset.apps/image-automation-controller-75cc9d6964 1 1 1 2d7h
replicaset.apps/image-reflector-controller-6c99d4c47b 1 1 1 2d7h
replicaset.apps/kustomize-controller-6595b7976c 1 1 1 2d7h
replicaset.apps/notification-controller-56f8f7f86b 1 1 1 2d7h
replicaset.apps/source-controller-846457b955 1 1 1 2d7h
2.本地创建fluxcd目录并创建deploy
$ mkdir fluxcd-demo && cd fluxcd-demo
➜ fluxcd-demo (main) ✔ cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: hello-world-flask
name: hello-world-flask
spec:
replicas: 2
selector:
matchLabels:
app: hello-world-flask
template:
metadata:
labels:
app: hello-world-flask
spec:
containers:
- image: ccr.ccs.tencentyun.com/app-public/hello-world-flask-amd:v1.0
name: hello-world-flask
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 100m
requests:
cpu: 100m
最后,在 Github 或 Gitlab 中创建 fluxcd-demo 仓库。为了方便测试,需要将仓库设置为公开权限,主分支为 Main,并将我们创建的 Manifest 推送至远端仓库:
https://github.com/Hugh-yw/fluxcd-demo
3.为 FluxCD 创建仓库连接信息
root@node1:~# cat fluxcd-repo.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: hello-world-flask
spec:
interval: 5s
ref:
branch: main
url: https://github.com/Hugh-yw/fluxcd-demo
注:
要将 URL 字段修改为你实际仓库的地址并使用 HTTPS 协议,branch 字段设置 main 分支。这里的 interval 代表每 5 秒钟主动拉取一次仓库并把它作为制品存储。
4.检查 GitRepository 对象
root@node1:~# kubectl get gitrepository
NAME URL AGE READY STATUS
hello-world-flask https://github.com/Hugh-yw/fluxcd-demo 2d7h True stored artifact for revision 'main/2c93e18e9700373a4fd29eb533ecf447b4765b57'
5.为 FluxCD 创建部署策略
root@node1:~# cat fluxcd-kustomize.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: hello-world-flask
spec:
interval: 5s
path: ./
prune: true
sourceRef:
kind: GitRepository
name: hello-world-flask
targetNamespace: default
注:
interval 参数表示 FluxCD 会每 5 秒钟运行一次工作负载差异对比;
path 参数表示 deployment.yaml 位于仓库的根目录中。
FluxCD在对比期望状态和集群实际状态的时候,如果发现差异就会触发重新部署。
资源检查:
root@node1:~# kubectl get kustomization
NAME AGE READY STATUS
hello-world-flask 2d7h True Applied revision: main/2c93e18e9700373a4fd29eb533ecf447b4765b57
6.体验gitops自动发布
1.代码提交
$ git add -A && git commit -m "Update image tag to v1"
$ git push origin main
root@node1:~# kubectl get pods --show-labels|grep flask
hello-world-flask-7d78848d75-ht6fk 1/1 Running 0 36s app=hello-world-flask,pod-template-hash=7d78848d75
hello-world-flask-7d78848d75-mqgl9 1/1 Running 0 31s app=hello-world-flask,pod-template-hash=7d78848d75
hello-world-flask-9d8967487-68kzq 1/1 Terminating 1 (2d4h ago) 2d6h app=hello-world-flask,pod-template-hash=9d8967487
hello-world-flask-9d8967487-mbwn9 1/1 Terminating 1 (2d4h ago) 2d6h app=hello-world-flask,pod-template-hash=9d8967487
2.查看触发重新部署的事件
root@node1:~# kubectl describe kustomization hello-world-flask
Name: hello-world-flask
Namespace: default
Labels: <none>
Annotations: <none>
API Version: kustomize.toolkit.fluxcd.io/v1beta2
Kind: Kustomization
Metadata:
Creation Timestamp: 2023-07-11T04:07:02Z
Finalizers:
finalizers.fluxcd.io
Generation: 1
Managed Fields:
API Version: kustomize.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizers.fluxcd.io":
Manager: gotk-kustomize-controller
Operation: Update
Time: 2023-07-11T04:07:02Z
API Version: kustomize.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:inventory:
.:
f:entries:
f:lastAppliedRevision:
f:lastAttemptedRevision:
f:observedGeneration:
Manager: gotk-kustomize-controller
Operation: Update
Subresource: status
Time: 2023-07-11T04:07:02Z
API Version: kustomize.toolkit.fluxcd.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:force:
f:interval:
f:path:
f:prune:
f:sourceRef:
.:
f:kind:
f:name:
f:targetNamespace:
Manager: kubectl-create
Operation: Update
Time: 2023-07-11T04:07:02Z
Resource Version: 40288002
UID: d4c061f3-fd9a-4c2f-8791-1b71e739bc51
Spec:
Force: false
Interval: 5s
Path: ./
Prune: true
Source Ref:
Kind: GitRepository
Name: hello-world-flask
Target Namespace: default
Status:
Conditions:
Last Transition Time: 2023-07-13T11:34:17Z
Message: Applied revision: main/2f35a6700aa47aa6872b595dc75acc9dea4a6f04
Reason: ReconciliationSucceeded
Status: True
Type: Ready
Inventory:
Entries:
Id: default_hello-world-flask_apps_Deployment
V: v1
Last Applied Revision: main/2f35a6700aa47aa6872b595dc75acc9dea4a6f04 #最新commitid
Last Attempted Revision: main/2f35a6700aa47aa6872b595dc75acc9dea4a6f04
Observed Generation: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ReconciliationSucceeded 45s (x37589 over 2d4h) kustomize-controller (combined from similar events): Reconciliation finished in 53.523094ms, next run in 5s
3.验证是否已更新
root@node1:~# curl http://10.233.47.55:5000
Hello, my first docker images! hello-world-flask-7d78848d75-mqgl9
通过上面的配置,我们让 FluxCD 自动完成了监听修改、比较和重新部署三个过程。
7.体验gitops自动回滚
要回滚 fluxcd-demo 仓库,首先需要找到上一次的提交记录。我们可以使用 git log 来查看它:
commit 2f35a6700aa47aa6872b595dc75acc9dea4a6f04 (HEAD -> main, origin/main)
Author: ywcheng
Date: Thu Jul 13 19:33:56 2023 +0800
Update image tag to latest
commit 2c93e18e9700373a4fd29eb533ecf447b4765b57
Author: ywcheng
Date: Tue Jul 11 12:54:31 2023 +0800
Update image tag to v1.0
--------------------------------------------------------------------
➜ fluxcd-demo (main) ✔ git reset --hard 2c93e18e9700373a4fd29eb533ecf447b4765b57
HEAD is now at 2c93e18 Update image tag to v1.0
➜ fluxcd-demo (main) ✔ git push origin main -f
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:Hugh-yw/fluxcd-demo.git
+ 2f35a67...2c93e18 main -> main (forced update)
再次查看触发器的events:
root@node1:~# kubectl describe kustomization hello-world-flask
......
Inventory:
Entries:
Id: default_hello-world-flask_apps_Deployment
V: v1
Last Applied Revision: main/2c93e18e9700373a4fd29eb533ecf447b4765b57
Last Attempted Revision: main/2c93e18e9700373a4fd29eb533ecf447b4765b57
Observed Generation: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ReconciliationSucceeded 3m43s (x37649 over 2d4h) kustomize-controller (combined from similar events): Reconciliation finished in 40.521018ms, next run in 5s
root@node1:~# curl http://10.233.47.55:5000
Hello, my v1 version docker images!! hello-world-flask-9d8967487-7j5n8
到这里,我们就成功体验了GitOps 工作流基础环节的发布和回滚实现(
小部分
)。
参考资料:https://time.geekbang.org/column/intro/100312001?tab=catalog