云原生Kubernetes: K8S 1.26版本部署KubeSphere

一、实验

1.环境

2.K8S 1.26版本部署HELM

3.K8S 1.26版本部署KubeSphere

4.安装KubeSphere DevOps

二、问题

1.如何安装Zadig

2.扩展插件Zadig安装失败

3.calico 如何实现不同node通信

4.如何清除docker占用的磁盘空间

5.如何强制删除资源

6.namespace删除不了

7.job如何实现删除资源

8.containerd容器使用ctr命令如何实现镜像操作

一、实验

1.环境

（1）主机

表1 主机

主机	架构	版本	IP	备注
master1	K8S master节点	1.26.0	192.168.204.190
node1	K8S node节点	1.26.0	192.168.204.191

（2）Termius连接

（3）master节点查看集群

1）查看node
kubectl get node
 
2)查看node详细信息
kubectl get node -o wide

2.K8S 1.26版本部署HELM

（1）查阅

https://github.com/helm/helm/tags

HELM版本与K8S集群兼容

（2）策略

当前K8S 集群为1.26.0版本，HELM 3.11.x 版本可以兼容。

所以选择3.11.0版本。

（3）下载

https://get.helm.sh/helm-v3.11.0-linux-amd64.tar.gz

Termius使用SFTP传输

（4）解压

tar -zxvf helm-v3.11.0-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
helm version

（5）命令补全

source <(helm completion bash)

3.K8S 1.26版本部署KubeSphere

（1）查阅

https://docs.kubesphere.com.cn/v4.0/02-quickstart/01-install-ks-core/

v1.26.x支持安装

（2）安装

helm upgrade --install -n kubesphere-system --create-namespace ks-core https://charts.kubesphere.io/main/ks-core-0.4.0.tgz

完整安装过程：

[root@master1 opt]# helm upgrade --install -n kubesphere-system --create-namespace ks-core https://charts.kubesphere.io/main/ks-core-0.4.0.tgz
Release "ks-core" does not exist. Installing it now.
NAME: ks-core
LAST DEPLOYED: Wed May 22 11:57:23 2024
NAMESPACE: kubesphere-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Please wait for several seconds for KubeSphere deployment to complete.

1. Make sure KubeSphere components are running:

     kubectl get pods -n kubesphere-system

2. Then you should be able to visit the console NodePort:

     Console: http://192.168.204.190:30880

3. To login to your KubeSphere console:

     Account: admin
     Password: "P@88w0rd"
     NOTE: Please change the default password after login.

For more details, please visit https://kubesphere.io.

（3）查看pod

kubectl get pods -n kubesphere-system

(4) 访问

http://192.168.204.190:30880

（5）输入初始账户及密码

账户: admin
密码: P@88w0rd

（6）修改密码

（7）进入系统

(8) 集群管理

（9）扩展中心

（10）搜索市场

关键字“CI/CD”

4.安装KubeSphere DevOps

(1) 查阅

KubeSphere 扩展市场

https://kubesphere.com.cn/marketplace/extensions/devops/

其他方式安装

https://www.kubesphere.io/zh/docs/v3.4/quick-start/minimal-kubesphere-on-k8s/

（2）同步云账户

（3）安装

KubeSphere DevOps版本支持情况

Kubernetes 版本>=1.19.0-0

KubeSphere 版本>=4.0.0-0

（4）下一步

（5）开始安装

安装中

成功

（6）下一步

（7）集群选择

（8）确认

（9）安装成功

（10）查看集群

二、问题

1.如何安装Zadig

(1) 查阅

Kubesphere扩展市场

https://kubesphere.com.cn/marketplace/extensions/zadig/

Zadig版本支持情况

Kubernetes 版本>=1.16.0-1.26.0

KubeSphere 版本>=4.0.0-0

（2）Zadig主页

https://koderover.com/zadig

（3）脚本方式安装

https://docs.koderover.com/zadig/Zadig%20v2.0.0/install/helm-deploy/

（4）Kubesphere安装Zadig

官方脚本安装

curl -LO https://github.com/koderover/zadig/releases/download/v2.0.0/install_quickstart.sh
chmod +x ./install_quickstart.sh

（8）申明变量

export IP=192.168.204.190
export PORT=30090

（9）安装

./install_quickstart.sh

这里安装过程预计持续 10 分钟左右

2.扩展插件Zadig安装失败

（1）报错

（2）原因分析

查看日志

2024-05-22T12:18:54.721997147+08:00 client.go:482: [debug] Ignoring delete failure for "zadig-post-upgrade" batch/v1, Kind=Job: jobs.batch "zadig-post-upgrade" not found

（3）解决方法

查阅相关问题

https://github.com/koderover/zadig/issues/2417

先卸载

然后重新安装

下一步

开始安装

安装中

依然报错

2024-05-22T12:39:41.303302680+08:00 upgrade.go:442: [debug] warning: Upgrade "zadig" failed: post-upgrade hooks failed: 1 error occurred:
2024-05-22T12:39:41.303305360+08:00 	* job failed: BackoffLimitExceeded
2024-05-22T12:39:41.303306374+08:00 
2024-05-22T12:39:41.303307165+08:00 
2024-05-22T12:39:41.327312921+08:00 Error: UPGRADE FAILED: post-upgrade hooks failed: 1 error occurred:
2024-05-22T12:39:41.327345989+08:00 	* job failed: BackoffLimitExceeded
2024-05-22T12:39:41.327352567+08:00 
2024-05-22T12:39:41.327356487+08:00 
2024-05-22T12:39:41.327363940+08:00 helm.go:84: [debug] post-upgrade hooks failed: 1 error occurred:
2024-05-22T12:39:41.327368417+08:00 	* job failed: BackoffLimitExceeded
2024-05-22T12:39:41.327371965+08:00 
2024-05-22T12:39:41.327375169+08:00 
2024-05-22T12:39:41.327380602+08:00 UPGRADE FAILED

这时可以采用脚本安装。

查看脚本（HELM为3.6.1版本）与master1节点冲突

node1节点部署

如报错显示访问超时，需要网络好时再执行

Error: UPGRADE FAILED: post-upgrade hooks failed: timed out waiting for the condition

3.calico 如何实现不同node通信

（1）查看

calico网络插件

kubectl -n kube-system get po -owide | grep calico-node

可以看到有两个容器网段分别分配给了master1和node1

kubectl get ipamblocks

（2）查看具体信息

kubectl get ipamblocks 10-244-137-64-26 -oyaml
kubectl get ipamblocks 10-244-166-128-26 -oyaml

（3）查看路由表

查看master1路由表(目标地址为10.244.66.128/26的请求会被通过网卡tunl0转发到192.168.204.191，也就是node1上；master1节点本机上的POD IP，则会直接被路由到对应的calico网卡。)

route -n

查看node1路由表（node01上也可以看到类似的路由条目，目标地址为10.244.137.64/26的请求会被通过网卡tunl0转发到192.168.204.190，也就是master1上）

route -n

4.如何清除docker占用的磁盘空间

（1）查询

master1节点查看docker占用磁盘空间

docker system df

docker system df -v

node1节点查看docker占用磁盘空间

（2）清除

master1节点仅删除停止运行的容器

docker container prune

noder1节点仅删除停止运行的容器

5.如何强制删除资源

（1）删除pod

命令

kubectl delete pod <your-pod-name> -n <name-space> --force --grace-period=0

删除

kubectl delete --all pods -n zadig  --force --grace-period=0

kubectl delete --all pods -n argocd  --force --grace-period=0

kubectl delete --all pods -n extension-devops  --force --grace-period=0

kubectl delete --all pods -n extension-zadig  --force --grace-period=0

（2）删除pv、pvc

kubectl patch pv xxx -p '{"metadata":{"finalizers":null}}'
kubectl patch pvc xxx -p '{"metadata":{"finalizers":null}}'

6.namespace删除不了

（1）报错

kubesphere-devops-system 一直为Terminating

（2）原因分析

选择一个Terminating namespace，并查看namespace 中的finalizer。

kubectl get namespace kubesphere-devops-system -o yaml

显示如下：

spec:
  finalizers:
  - kubernetes

（3）解决方法

导出json格式到文件

kubectl get namespace kubesphere-devops-system  -o json >tmp.json

编辑tmp.josn，删除finalizers 字段的值

 25     "spec": {                    #从此行开始删除
 26         "finalizers": [
 27             "kubernetes"
 28         ]
 29     },                           # 删到此行

开启proxy（执行该命令后，当前终端会被卡住）

kubectl proxy

打开新的一个窗口，执行以下命令

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/kubesphere-devops-system/finalize

确认处于Terminating 状态的namespace已经被删除

7.job如何实现删除资源

（1）查看状态

目前为已完成

（2）查看YAML

kind: Job
apiVersion: batch/v1
metadata:
  name: devops-post-delete
  namespace: extension-devops
  labels:
    controller-uid: 0e83e553-482f-4755-834a-9c0f07d4c5b9
    job-name: devops-post-delete
  annotations:
    batch.kubernetes.io/job-tracking: ''
    helm.sh/hook: post-delete
    helm.sh/hook-delete-policy: 'before-hook-creation,hook-succeeded'
    helm.sh/hook-weight: '1'
    revisions: >-
      {"1":{"status":"completed","succeed":1,"desire":1,"uid":"0e83e553-482f-4755-834a-9c0f07d4c5b9","start-time":"2024-05-22T15:52:49+08:00","completion-time":"2024-05-22T16:22:50+08:00"}}
spec:
  parallelism: 1
  completions: 1
  backoffLimit: 6
  selector:
    matchLabels:
      controller-uid: 0e83e553-482f-4755-834a-9c0f07d4c5b9
  template:
    metadata:
      creationTimestamp: null
      labels:
        controller-uid: 0e83e553-482f-4755-834a-9c0f07d4c5b9
        job-name: devops-post-delete
    spec:
      containers:
        - name: post-delete-job
          image: 'kubesphere/kubectl:v1.27.4'
          command:
            - /bin/bash
            - '-c'
            - |
              if kubectl get ns argocd; then
                kubectl delete ns argocd
              fi
              if kubectl get ns kubesphere-devops-system; then
                kubectl delete ns kubesphere-devops-system
              fi
              if kubectl get ns kubesphere-devops-worker; then
                kubectl delete ns kubesphere-devops-worker
              fi
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Never
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  completionMode: NonIndexed
  suspend: false

8.containerd容器使用ctr命令如何实现镜像操作

（1）帮助命令

[root@node1 ~]# ctr images --help
NAME:
   ctr images - manage images

USAGE:
   ctr images command [command options] [arguments...]

COMMANDS:
   check                    check existing images to ensure all content is available locally
   export                   export images
   import                   import images
   list, ls                 list images known to containerd
   mount                    mount an image to a target path
   unmount                  unmount the image from the target
   pull                     pull an image from a remote
   push                     push an image to a remote
   delete, del, remove, rm  remove one or more images by reference
   tag                      tag an image
   label                    set and clear labels for an image
   convert                  convert an image

OPTIONS:
   --help, -h  show help

（2）拉取

ctr images pull ghcr.io/dexidp/dex:v2.30.2

(3) 查看

crictl images list
或
ctr images list
或
ctr i ls -q

（4）导出

ctr image export  dev.tar.gz  ghcr.io/dexidp/dex:v2.30.2

（5）删除

1）查询
ctr image list | grep  ghcr.io/dexidp/dex:v2.30.2

2）删除
ctr image delete  ghcr.io/dexidp/dex:v2.30.2

（6）导入

1)导入
ctr image import  dev.tar.gz

2）查询
ctr image list | grep  ghcr.io/dexidp/dex:v2.30.2