kubernetes集群编排

k8s 集群部署

集群环境初始化

所有节点安装kubeadm

拉取集群所需镜像

集群初始化

安装flannel网络插件

设置kubectl命令补齐

k8s 集群部署

实验环境

主机名	ip	角色
k8s1（上一章的docker1）	192.168.81.10	reg.westos.org，harbor仓库
k8s2	192.168.81.11	master，k8s集群控制节点
k8s3	192.168.81.12	node，k8s集群工作节点
k8s4	192.168.81.13	node，k8s集群工作节点

所有节点禁用selinux和防火墙

所有节点同步时间和解析

所有节点安装docker-ce

所有节点禁用swap，注意注释掉/etc/fstab文件中的定义

集群环境初始化

所有k8s集群节点执行以下步骤

禁用swap

[root@k8s2 ~]# swapoff -a
[root@k8s2 ~]# vim /etc/fstab
#/dev/mapper/rhel-swap   swap                    swap    defaults        0 0

修改内核参数

[root@k8s2 sysctl.d]# vim docker.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1

[root@k8s2 ~]#  sysctl --system

配置仓库

[root@k8s2 yum.repos.d]# vim docker.repo 
[docker]
name=docker-ce
baseurl=https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/7/x86_64/stable/gpgcheck=0

[centos]
name=extras
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/7/extras/x86_64/
gpgcheck=0

[root@k8s2 ~]# yum install -y docker-ce
[root@k8s2 ~]# systemctl enable --now docker

配置守护进程

[root@k8s2 ~]# vim /etc/docker/daemon.json
{
        "registry-mirrors": ["https://reg.westos.org"],
        "exec-opts": ["native.cgroupdriver=systemd"],
        "log-driver": "json-file",
        "log-opts": {
                "max-size": "100m"
        },
        "storage-driver": "overlay2"
}

[root@k8s2 ~]# systemctl  restart docker

所有节点同步docker配置，以及拷贝harbor仓库的证书

[root@k8s1 ~]# cd /etc/docker/
[root@k8s1 docker]# ls
certs.d
[root@k8s1 docker]# scp -r certs.d/ k8s2:/etc/docker/

确保所有k8s节点可以从私有仓库下载镜像

所有节点安装kubeadm

[root@k8s2 yum.repos.d]# vim k8s.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=0

[root@k8s2 ~]# yum install -y kubelet-1.23.17-0 kubeadm-1.23.17-0 kubectl-1.23.17-0

[root@k8s2 ~]# systemctl enable --now kubelet

拉取集群所需镜像

[root@k8s1 ~]# docker load -i k8s-v1.23.17.tar

登录仓库

[root@k8s1 ~]# docker login reg.westos.org
Username: admin
Password:

先在harbor仓库上新建一个项目

[root@k8s1 ~]# docker images |grep google_containers | awk '{print $1":"$2}' | awk -F/ '{system("docker tag "$0" reg.westos.org/k8s/"$3"")}'

[root@k8s1 ~]# docker images  |grep k8s | awk '{system("docker push "$1":"$2"")}'

集群初始化

[root@k8s2 ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 --image-repository reg.westos.org/k8s --kubernetes-version v1.23.17

设置环境变量

[root@k8s2 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf

写入环境变量，确保重启后依然生效

[root@k8s2 ~]# vim .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
export KUBECONFIG=/etc/kubernetes/admin.conf

查看集群状态

[root@k8s2 ~]# kubectl get node
[root@k8s2 ~]# kubectl get pod -A

当前节点还没有就绪，是因为没有安装网路插件，pod还没运行

安装flannel网络插件

下载flannel网络插件

[root@k8s1 ~]# wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

修改镜像位置

[root@k8s1 ~]# vim kube-flannel.yml
[root@k8s1 ~]# scp kube-flannel.yml k8s2:

新建项目仓库

下载镜像

[root@k8s1 docker]# docker pull docker.io/flannel/flannel:v0.21.2
[root@k8s1 docker]# docker pull docker.io/flannel/flannel-cni-plugin:v1.1.2

上传镜像

[root@k8s1 docker]# docker images |grep flannel | awk '{print $1":"$2}' | awk '{system("docker tag "$0" reg.westos.org/"$0"")}'

[root@k8s1 docker]# docker push reg.westos.org/flannel/flannel:v0.21.2
[root@k8s1 docker]# docker push reg.westos.org/flannel/flannel-cni-plugin:v1.1.2

部署网络插件

[root@k8s2 ~]# kubectl apply -f kube-flannel.yml
[root@k8s2 ~]# kubectl  -n kube-flannel get pod
[root@k8s2 ~]# kubectl get node
[root@k8s2 ~]# kubectl get pod -A

扩容节点

[root@k8s3 ~]# kubeadm join 192.168.81.11:6443 --token 02n0ronbcez6a06uu5ogs3
--discovery-token-ca-cert-hash sha256:83a0b7b4f2d5dda0c4105121ba6a3aa8d747eed5386bcf654ceaaf50c66be9ce
[root@k8s4 ~]# kubeadm join 192.168.81.11:6443 --token 02n0ronbcez6a06uu5ogs3
--discovery-token-ca-cert-hash sha256:83a0b7b4f2d5dda0c4105121ba6a3aa8d747eed5386bcf654ceaaf50c66be9ce

[root@k8s2 ~]# kubectl get node

设置kubectl命令补齐

[root@k8s2 ~]# yum install -y bash-completion
[root@k8s2 ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
[root@k8s2 ~]# source  ~/.bashrc

集群升级

部署cri-docker （所有集群节点）

k8s从1.24版本开始移除了dockershim，所以需要安装cri-docker插件才能使用docker

软件下载：GitHub - Mirantis/cri-dockerd: dockerd as a compliant Container Runtime Interface for Kubernetes

安装

[root@k8s2 ~]# rpm -ivh cri-dockerd-0.3.5-3.el7.x86_64.rpm

配置cri-docker

[root@k8s2 ~]# vim /usr/lib/systemd/system/cri-docker.service
[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=reg.westos.org/k8s/pause:3.7

[root@k8s2 ~]# systemctl daemon-reload
[root@k8s2 ~]# systemctl  enable --now cri-docker
[root@k8s2 ~]# ll /var/run/cri-dockerd.sock
srw-rw---- 1 root docker 0 Jan  9 17:51 /var/run/cri-dockerd.sock

升级master节点

首先上传镜像到harbor仓库，便于升级

reg.westos.org/k8s/kube-apiserver                                v1.24.17

reg.westos.org/k8s/kube-proxy                                      v1.24.17

reg.westos.org/k8s/kube-scheduler                               v1.24.17

reg.westos.org/k8s/kube-controller-manager                 v1.24.17

reg.westos.org/k8s/etcd                                            3.5.3-0

reg.westos.org/k8s/pause                                           3.7

reg.westos.org/k8s/coredns                                         v1.8.6

升级kubeadm

[root@k8s2 ~]# yum install -y kubeadm-1.24.0-0

执行升级

[root@k8s2 ~]# kubeadm upgrade plan

修改节点套接字

[root@k8s2 ~]# kubectl edit nodes k8s2
...
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/cri-dockerd.sock

[root@k8s2 ~]# kubeadm upgrade apply v1.24.0

腾空节点

[root@k8s2 ~]# kubectl drain k8s2 --ignore-daemonsets

升级kubelet

[root@k8s2 ~]# yum install -y kubelet-1.24.0-0 kubectl-1.24.0-0

配置kubelet使用cri-docker

[root@k8s2 ~]# vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--pod-infra-container-image=reg.westos.org/k8s/pause:3.7 --container-runtime=remote --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock"

重启kubelet

[root@k8s2 ~]# systemctl daemon-reload 
[root@k8s2 ~]# systemctl  restart kubelet

解除节点保护

[root@k8s2 ~]# kubectl uncordon k8s2

升级worker节点

升级kubeadm

[root@k8s3 ~]# yum install -y kubeadm-1.24.0-0

执行升级

[root@k8s3 ~]# kubeadm upgrade node

腾空节点

[root@k8s2 ~]# kubectl drain k8s3 --ignore-daemonsets    //需要在master节点执行

配置kubelet使用cri-docker

[root@k8s3 ~]# vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--pod-infra-container-image=reg.westos.org/k8s/pause:3.7 --container-runtime=remote --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock"

修改节点套接字

[root@k8s2 ~]# kubectl edit nodes k8s3		//需要在master节点执行
...
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/cri-dockerd.sock

重启kubelet

[root@k8s3 ~]# systemctl daemon-reload 
[root@k8s3 ~]# systemctl  restart kubelet

接触节点保护

[root@k8s2 ~]# kubectl uncordon k8s3		//需要在master节点执行

其它节点依此类推

不使用docker集群部署

k8s2、k8s3、k8s4在配置前需要重置节点，关闭docker

[root@k8s2 ~]# kubeadm reset
[root@k8s3 ~]# kubeadm reset  --cri-socket unix:///var/run/cri-dockerd.sock
[root@k8s4 ~]# kubeadm reset  --cri-socket unix:///var/run/cri-dockerd.sock

所有节点清楚iptables规则

[root@k8s2 ~]# iptables -F
[root@k8s2 ~]# iptables -F -t nat

禁用所有节点docker和cri-docker服务

[root@k8s2 ~]# systemctl  disable  docker
[root@k8s2 ~]# systemctl  disable  cri-docker

重置后所有节点重启

之前部署过docker，containerd默认已经安装

修改配置

[root@k8s2 ~]# containerd config default | tee /etc/containerd/config.toml

[root@k8s2 ~]# cd /etc/containerd/
[root@k8s2 containerd]# vim config.toml
...
sandbox_image = "reg.westos.org/k8s/pause:3.7"
...
SystemdCgroup = true

拷贝证书

[root@k8s2 containerd]# mkdir -p /etc/containerd/certs.d/reg.westos.org 
[root@k8s2 containerd]# cp /etc/docker/certs.d/reg.westos.org/ca.crt /etc/containerd/certs.d/reg.westos.org/

[root@k8s2 containerd]# systemctl  restart containerd

[root@k8s2 containerd]# scp -r certs.d/ config.toml k8s3:/etc/containerd/
[root@k8s2 containerd]# scp -r certs.d/ config.toml k8s4:/etc/containerd/

[root@k8s3 docker]# systemctl disable --now docker cri-docker
[root@k8s3 docker]# systemctl  enable --now containerd
[root@k8s3 docker]# crictl config runtime-endpoint unix:///run/containerd/containerd.sock

[root@k8s4 ~]# systemctl disable --now docker cri-docker
[root@k8s4 ~]#  systemctl  enable --now containerd
[root@k8s4 ~]# crictl config runtime-endpoint unix:///run/containerd/containerd.sock

启动containerd

[root@k8s2 containerd ]# systemctl  enable containerd
[root@k8s2 containerd ]# systemctl  restart containerd
[root@k8s2 ~]# crictl config runtime-endpoint unix:///run/containerd/containerd.sock
[root@k8s2 ~]# crictl img
[root@k8s2 ~]# crictl pull reg.westos.org/k8s/pause:3.7

集群初始化

 kubeadm init --pod-network-cidr=10.244.0.0/16 --image-repository reg.westos.org/k8s --kubernetes-version v1.24.17