二进制方式部署K8s高可用集群

news2024/11/17 7:32:08

1 二进制方式部署K8s高可用集群

1.1 kubeadm 和二进制安装 k8s 适用场景分析

  • kubeadm 是官方提供的开源工具，是一个开源项目，用于快速搭建 kubernetes 集群，目前是比较方便和推荐使用的。kubeadm init 以及 kubeadm join 这两个命令可以快速创建 kubernetes 集群。Kubeadm初始化 k8s，所有的组件都是以 pod 形式运行的，具备故障自恢复能力。

  • kubeadm 是工具，可以快速搭建集群，也就是相当于用程序脚本帮我们装好了集群，属于自动部署，简化部署操作，自动部署屏蔽了很多细节，使得对各个模块感知很少，如果对 k8s 架构组件理解不深的话，遇到问题比较难排查。

  • kubeadm 适合需要经常部署 k8s，或者对自动化要求比较高的场景下使用。

  • 二进制：在官网下载相关组件的二进制包，如果手动安装，对 kubernetes 理解也会更全面。

  • Kubeadm 和二进制都适合生产环境，在生产环境运行都很稳定，具体如何选择，可以根据实际项目进行评估。

1.2 机器规划

  • 部署版本Kubernetes（k8s）v1.30.1

主机IP地址操作系统配置
k8s-master-01192.168.110.21CentOS Linux release 7.9.20094颗CPU 8G内存 100G硬盘
k8s-master-02192.168.110.22CentOS Linux release 7.9.20094颗CPU 8G内存 100G硬盘
k8s-master-03192.168.110.23CentOS Linux release 7.9.20094颗CPU 8G内存 100G硬盘
k8s-node-01192.168.110.24CentOS Linux release 7.9.20094颗CPU 8G内存 100G硬盘
k8s-node-02192.168.110.25CentOS Linux release 7.9.20094颗CPU 8G内存 100G硬盘
k8s-node-03192.168.110.26CentOS Linux release 7.9.20094颗CPU 8G内存 100G硬盘

  • VIP：192.168.110.20/24

1.3 基础环境部署(所有节点执行)

  • 运行初始化脚本

[root@k8s-all ~]# vim /root/init.sh
#!/bin/bash
​
echo_red() { echo -e "\e[31m$1\e[0m"; }
echo_green() { echo -e "\e[32m$1\e[0m"; }
echo_yellow() { echo -e "\e[33m$1\e[0m"; }
​
interface=$(ip route | grep default | sed -e "s/^.*dev.//" -e "s/.proto.*//")
LocalIP=$(ip addr show $interface | awk '/inet / {print $2}' | cut -d/ -f1 | tail -1)
linux=$(awk -F "("  '{print $1}' /etc/redhat-release)
hostname=$(hostname)
​
echo_yellow "当前系统发行版为  $linux"
echo_yellow "当前系统网卡名为  $interface"
echo_yellow "本机IP地址为  $LocalIP"
echo_yellow "本机主机名为  $hostname"
​
sleep 2
echo_yellow '正在初始化请稍后...'
​
autoconnect=$(nmcli dev show $interface | grep 'GENERAL.AUTOCONNECT:' | awk '{print $2}')
if [[ "$autoconnect" != "yes" ]]; then
    echo_yellow "检查网卡是否为开机自动连接..."
    nmcli con mod $interface connection.autoconnect yes &>/dev/null
    nmcli con up $interface &>/dev/null
    sed -i '/^ONBOOT/ c ONBOOT=yes' /etc/sysconfig/network-scripts/"ifcfg-${interface}" &>/dev/null
    systemctl restart network &>/dev/null
    echo_green "网卡已设置为开机自动连接"
else
    echo_green "网卡已设置为开机自动连接"
fi
​
ip_mode=$(nmcli dev show $interface | grep 'IP4.ADDRESS[1]' | awk '{print $2}')
gateway_dns=$(ip addr show $interface | awk '/inet / {split($2,a,"."); print a[1]"."a[2]"."a[3]}')
ip_mode2=$(grep BOOTPROTO /etc/sysconfig/network-scripts/"ifcfg-$interface" | awk -F "=" '{print $2}')
​
if [[ -z "$ip_mode" ]] || [ "$ip_mode2" != static ] ; then
    echo_green "正在检查网卡是否为静态IP地址..."
    nmcli con mod $interface ipv4.addresses "${LocalIP}/24" &>/dev/null
    nmcli con mod $interface ipv4.gateway "${gateway_dns}.2" &>/dev/null
    nmcli con mod $interface ipv4.dns "${gateway_dns}.2" &>/dev/null
    nmcli con mod $interface ipv4.method manual &>/dev/null
    nmcli con up $interface &>/dev/null
    sed -i '/^BOOTPROTO/ c BOOTPROTO=static' /etc/sysconfig/network-scripts/"ifcfg-${interface}" &>/dev/null
    systemctl restart network &>/dev/null
    echo_green "网卡已设置静态IP"
else
    echo_green "网卡已设置静态IP"
fi
​
​
for i in {1..3}; do
    if ping -c 4 -i 0.2 223.5.5.5 &>/dev/null; then
        echo_green "网络连通性正常"
        break
    else
        echo_red "网络错误请检查网络配置"
        echo_yellow "正在尝试重启网络（尝试次数：$i）"
        systemctl restart network &>/dev/null
        nmcli con up $interface &>/dev/null
        sleep 5
    fi
done
​
manage_selinux_firewall() {
    SELINUXSTATUS=$(getenforce)
​
    if [[ "$SELINUXSTATUS" == "Disabled" ]] || [[ "$SELINUXSTATUS" == "Permissive" ]]; then
        echo_green "SELinux已成功关闭或当前处于宽容模式"
    else
        echo_red "SELinux未能关闭，正在尝试手动关闭..."
        sed -i '/^SELINUX=/ c\SELINUX=disabled' /etc/selinux/config &>/dev/null
        setenforce 0
        if [[ "$(getenforce)" == "Disabled" ]] || [[ "$(getenforce)" == "Permissive" ]]; then
            echo_green "SELinux已成功关闭"
        else
            echo_red "SELinux未能关闭，请手动解决"
        fi
    fi
}
​
manage_selinux_firewall
​
FIREWALLSTATUS=$(systemctl is-active firewalld)
​
if [[ "$FIREWALLSTATUS" == "active" ]]; then
    echo_yellow "防火墙状态为开启，正在关闭防火墙..."
    systemctl disable --now firewalld &>/dev/null
    echo_green "防火墙已关闭"
else
    echo_green "防火墙无需操作"
fi
​
PACKAGES="lrzsz ntpdate sysstat net-tools wget vim bash-completion dos2unix tree psmisc chrony rsync lsof gcc"
echo_yellow "正在安装常用软件..."
yum -y install $PACKAGES &>/dev/null
​
if [ $? -eq 0 ]; then
    echo_green "安装成功"
else
    echo_red "安装失败，请检测yum镜像仓库"
fi
​
echo_yellow "准备开启 $linux 系统的体验吧！！！"
​
# Prompt for shutdown, reboot, or exit
echo_yellow "请选择一个操作："
echo_yellow "          关机 : 1"
echo_yellow "          重启 : 2"
echo_yellow "          退出 : 3"
​
read -p "请输入您的选择: " choice
​
delete_script() {
    echo_yellow "脚本执行完毕，正在删除自身..."
    rm -f "$0"
    echo_green "脚本已删除。"
}
​
case $choice in
    1)
        echo_yellow "正在执行关机操作..."
        delete_script
        init 0
        # 关机前删除脚本自身
        ;;
    2)
        echo_green "正在执行重启操作..."
        delete_script
        reboot
        # 重启前删除脚本自身
        ;;
    3)
        echo_yellow "请继续操作，脚本将退出。"
        # 退出前删除脚本自身
        delete_script
        exit 0
        ;;
    *)
        echo_red "无效的输入"
        # 退出前删除脚本自身
        delete_script
        exit 1
        ;;
esac
​
[root@k8s-all ~]# bash /root/init.sh

注意：

  • 若虚拟机是进行克隆的那么网卡的UUID会重复

  • 若UUID重复需要重新生成新的UUID

  • UUID重复无法获取到IPV6地址

  • 克隆出来的虚拟机 CentOS系统需要删除DUID

    rm -rf /etc/machine-id systemd-machine-id-setup reboot

  • 查看当前的网卡列表和 UUID：

    nmcli con show

  • 删除要更改 UUID 的网络连接：

    nmcli con delete uuid <原 UUID>

  • 重新生成 UUID：

    nmcli con add type ethernet ifname <接口名称> con-name <新名称>

  • 重新启用网络连接：

    nmcli con up <新名称>

  • 所有节点配置Hosts解析

[root@K8s-all ~]# cat >> /etc/hosts << EOF
192.168.110.21 k8s-master-01
192.168.110.22 k8s-master-02
192.168.110.23 k8s-master-03
192.168.110.24 k8s-node-01
192.168.110.25 k8s-node-02
192.168.110.26 k8s-node-03
EOF

  • k8s-master-01生成密钥，其他节点可以免密钥访问

[root@k8s-master-01 ~]# ssh-keygen -f ~/.ssh/id_rsa -N '' -q
[root@k8s-master-01 ~]# ssh-copy-id k8s-master-02
[root@k8s-master-01 ~]# ssh-copy-id k8s-master-03
[root@k8s-master-01 ~]# ssh-copy-id k8s-node-01
[root@k8s-master-01 ~]# ssh-copy-id k8s-node-02
[root@k8s-master-01 ~]# ssh-copy-id k8s-node-03

  • 配置NTP时间同步

sed -i '3,6 s/^/# /' /etc/chrony.conf
sed -i '6 a server ntp.aliyun.com iburst' /etc/chrony.conf 
systemctl restart chronyd.service 
chronyc sources

  • 禁用Swap交换分区

[root@K8s-all ~]# swapoff -a   #临时关闭
[root@K8s-all ~]# sed -i 's/.*swap.*/# &/' /etc/fstab  #永久关闭

  • 升级操作系统内核

[root@k8s-all ~]# wget -c http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/kernel-ml-6.0.3-1.el7.elrepo.x86_64.rpm
[root@k8s-all ~]# wget -c http://mirrors.coreix.net/elrepo-archive-archive/kernel/el7/x86_64/RPMS/kernel-ml-devel-6.0.3-1.el7.elrepo.x86_64.rpm
[root@k8s-all ~]# rpm -ivh kernel-ml-6.0.3-1.el7.elrepo.x86_64.rpm
[root@k8s-all ~]# rpm -ivh kernel-ml-devel-6.0.3-1.el7.elrepo.x86_64.rpm
[root@k8s-all ~]# awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
CentOS Linux (6.0.3-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.119.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.71.1.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-35f6b014eff0419881bbf71f1d9d4943) 7 (Core)
[root@k8s-all ~]# grub2-set-default 0
[root@k8s-all ~]# reboot
[root@k8s-all ~]# uname -r
6.0.3-1.el7.elrepo.x86_64

  • 配置内核转发及网桥过滤

[root@K8s-all ~]# cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
​
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
​
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
EOF
​
sysctl --system

  • 开启IPVS

[root@K8s-all ~]# yum install ipset ipvsadm -y
[root@K8s-all ~]# vim /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
​
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_vip ip_vs_sed ip_vs_ftp nf_conntrack"
​
for kernel_module in $ipvs_modules; 
do
        /sbin/modinfo -F filename $kernel_module >/dev/null 2>&1
        if [ $? -eq 0 ]; then
                /sbin/modprobe $kernel_module
        fi
done
​
chmod 755 /etc/sysconfig/modules/ipvs.modules
​
[root@K8s-all ~]# bash /etc/sysconfig/modules/ipvs.modules

1.4 安装docker作为Runtime

  • 二进制包下载地址：Index of linux/static/stable/x86_64/

[root@K8s-all ~]# wget -c https://mirrors.ustc.edu.cn/docker-ce/linux/static/stable/x86_64/docker-25.0.3.tgz
#解压
[root@K8s-all ~]# tar xf docker-*.tgz 
#拷贝二进制文件
[root@K8s-all ~]# cp docker/* /usr/bin/

  • 创建containerd的service文件

[root@K8s-all ~]# cat >/etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

[root@k8s-all ~]# systemctl enable --now containerd.service
[root@k8s-all ~]# systemctl is-active containerd.service
active

  • 准备docker的service文件

[root@K8s-all ~]# cat > /etc/systemd/system/docker.service <<EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service cri-docker.service docker.socket containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service
​
[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
OOMScoreAdjust=-500
​
[Install]
WantedBy=multi-user.target
EOF

  • 准备docker的socket文件

[root@K8s-all ~]# cat > /etc/systemd/system/docker.socket <<EOF
[Unit]
Description=Docker Socket for the API
​
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
​
[Install]
WantedBy=sockets.target
EOF

  • 配置加速器

[root@K8s-all ~]# mkdir -p /etc/docker
[root@K8s-all ~]# tee /etc/docker/daemon.json <<-'EOF'
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": [
"https://dbckerproxy.com",
"https://hub-mirror.c.163.com",
"https://mirror.baidubce.com",
"https://ccr.ccs.tencentyun.com"
]
}
EOF

  • 启动Docker

[root@K8s-all ~]# groupadd docker
#创建docker组

[root@K8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。

[root@K8s-all ~]# systemctl enable --now docker.socket
# 启用并立即启动docker.socket单元。docker.socket是一个systemd的socket单元，用于接收来自网络的Docker API请求。

[root@K8s-all ~]# systemctl enable --now docker.service
# 启用并立即启动docker.service单元。docker.service是Docker守护进程的systemd服务单元。

[root@K8s-all ~]# docker info
#验证

  • 安装部署cri-docker

    • 注意：K8s从1.24版本后不支持docker了所以这里需要用cri-dockererd

    • 下载地址：https://github.com/Mirantis/cri-dockerd/releases/

[root@K8s-all ~]# wget -c https://mirrors.chenby.cn/https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.10/cri-dockerd-0.3.10.amd64.tgz

# 解压cri-docker
[root@K8s-all ~]# tar xf cri-dockerd-*.amd64.tgz 
[root@K8s-all ~]# cp -r cri-dockerd/  /usr/bin/
[root@K8s-all ~]# chmod +x /usr/bin/cri-dockerd/cri-dockerd

  • 写入启动cri-docker配置文件

[root@K8s-all ~]# cat >  /usr/lib/systemd/system/cri-docker.service <<EOF
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

  • 写入cri-docker的socket配置文件

[root@K8s-all ~]# cat > /usr/lib/systemd/system/cri-docker.socket <<EOF
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target
EOF

  • 启动cri-docker

[root@K8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。

[root@K8s-all ~]# systemctl enable --now cri-docker.service
# 启用并立即启动cri-docker.service单元。cri-docker.service是cri-docker守护进程的systemd服务单元。

[root@K8s-all ~]# systemctl restart cri-docker.service
# 重启cri-docker.service单元，即重新启动cri-docker守护进程。

[root@K8s-all ~]# systemctl status docker.service
# 显示docker.service单元的当前状态，包括运行状态、是否启用等信息。

1.5 k8s与etcd下载及安装

仅在k8s-master-01操作

  • 下载并解压k8s安装包

[root@k8s-master-01 ~]# wget -c https://mirrors.chenby.cn/https://github.com/etcd-io/etcd/releases/download/v3.5.12/etcd-v3.5.12-linux-amd64.tar.gz
[root@k8s-master-01 ~]# wget -c https://dl.k8s.io/v1.30.1/kubernetes-server-linux-amd64.tar.gz
[root@k8s-master-01 ~]# tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

# 命令的解释如下：
# - tar：用于处理tar压缩文件的命令。
# - -xf：表示解压操作。
# - kubernetes-server-linux-amd64.tar.gz：要解压的文件名。
# - --strip-components=3：表示解压时忽略压缩文件中的前3级目录结构，提取文件时直接放到目标目录中。
# - -C /usr/local/bin：指定提取文件的目标目录为/usr/local/bin。
# - kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}：要解压和提取的文件名模式，用花括号括起来表示模式中的多个可能的文件名。

# 解压etcd安装文件
[root@k8s-master-01 ~]# tar -xf etcd*.tar.gz && mv etcd-*/etcd /usr/local/bin/ && mv etcd-*/etcdctl /usr/local/bin/

  • 查看版本

[root@k8s-master-01 ~]# kubelet --version
Kubernetes v1.30.1
[root@k8s-master-01 ~]# etcdctl version
etcdctl version: 3.5.12
API version: 3.5

  • 将组件发送至其他k8s节点

# 定义变量
[root@k8s-master-01 ~]# Master='k8s-master-02 k8s-master-03'
[root@k8s-master-01 ~]# Work='k8s-node-01 k8s-node-02 k8s-node-03'

# 拷贝master组件
[root@k8s-master-01 ~]# for NODE in $Master; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done

# 拷贝Work节点
[root@k8s-master-01 ~]# for NODE in $Work; do echo $NODE; scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done

# 所有节点创建目录
[root@k8s-all ~]# mkdir -p /opt/cni/bin

1.6 相关证书生成

  • k8s-master-01节点下载证书生成工具

[root@k8s-master-01 ~]# wget -c "https://mirrors.chenby.cn/https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64" -O /usr/local/bin/cfssl
[root@k8s-master-01 ~]# wget -c "https://mirrors.chenby.cn/https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64" -O /usr/local/bin/cfssljson

# 添加执行权限
[root@k8s-master-01 ~]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
1.6.1 生成etcd证书

  • 所有master节点创建证书存放目录

[root@k8s-master-all ~]# mkdir /etc/etcd/ssl -p

  • master01节点生成etcd证书

[root@k8s-master-01 ~]# cd /etc/etcd/ssl/

# 写入生成证书所需的配置文件
[root@k8s-master-01 ssl]# cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

[root@k8s-master-01 ssl]# cat > etcd-ca-csr.json  << EOF 
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

[root@k8s-master-01 ssl]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

#cfssl是一个用于生成TLS/SSL证书的工具，它支持PKI、JSON格式配置文件以及与许多其他集成工具的配合使用。
# gencert参数表示生成证书的操作。-initca参数表示初始化一个CA（证书颁发机构）。CA是用于签发其他证书的根证书。etcd-ca-csr.json是一个JSON格式的配置文件，其中包含了CA的详细信息，如私钥、公钥、有效期等。这个文件提供了生成CA证书所需的信息。

# 使用cfssl工具根据配置文件ca-csr.json生成一个CA证书
[root@k8s-master-01 ssl]# cat > etcd-csr.json << EOF 
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

# 用cfssl生成etcd证书
[root@k8s-master-01 ssl]# cfssl gencert \
-ca=/etc/etcd/ssl/etcd-ca.pem \
-ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
-config=ca-config.json \
-hostname=127.0.0.1,k8s-master-01,k8s-master-02,k8s-master-03,192.168.110.21,192.168.110.22,192.168.110.23 \
-profile=kubernetes \
etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

  • 将证书复制到其他节点

[root@k8s-master-01 ssl]# Master='k8s-master-02 k8s-master-03'
[root@k8s-master-01 ssl]# for NODE in $Master; do ssh $NODE "mkdir -p /etc/etcd/ssl"; for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}; done; done
1.6.2 生成k8s相关证书

  • 所有k8s节点创建证书存放目录

[root@k8s-all ~]# mkdir -p /etc/kubernetes/pki

  • master01节点生成k8s证书

[root@k8s-master-01 ~]# cd /etc/kubernetes/pki/
[root@k8s-master-01 pki]# cat > ca-csr.json   << EOF 
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF
​
[root@k8s-master-01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca
​
[root@k8s-master-01 pki]# cat > apiserver-csr.json << EOF 
{
  "CN": "kube-apiserver",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF
​
[root@k8s-master-01 pki]# cfssl gencert   \
-ca=/etc/kubernetes/pki/ca.pem   \
-ca-key=/etc/kubernetes/pki/ca-key.pem   \
-config=/etc/etcd/ssl/ca-config.json   \
-hostname=10.96.0.1,192.168.110.20,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,x.oiox.cn,k.oiox.cn,l.oiox.cn,o.oiox.cn,192.168.110.21,192.168.110.22,192.168.110.23,192.168.110.24,192.168.110.25,192.168.110.26,192.168.110.20 \
-profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

  • 生成apiserver聚合证书

[root@k8s-master-01 pki]# cat > front-proxy-ca-csr.json  << EOF 
{
  "CN": "kubernetes",
  "key": {
     "algo": "rsa",
     "size": 2048
  },
  "ca": {
    "expiry": "876000h"
  }
}
EOF
​
[root@k8s-master-01 pki]# cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca  # 生成CA证书
​
[root@k8s-master-01 pki]# cat > front-proxy-client-csr.json  << EOF 
{
  "CN": "front-proxy-client",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}
EOF
​
[root@k8s-master-01 pki]# cfssl gencert  \
-ca=/etc/kubernetes/pki/front-proxy-ca.pem   \
-ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   \
-config=/etc/etcd/ssl/ca-config.json   \
-profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client
1.6.3 生成controller-manage的证书

  • 选择使用那种高可用方案

    • 若使用 haproxy 那么为 --server=https://192.168.110.20:9443

    • 若使用 nginx方案，那么为 --server=https://127.0.0.1:8443

[root@k8s-master-01 pki]# cat > manager-csr.json << EOF 
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

[root@k8s-master-01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=/etc/etcd/ssl/ca-config.json \
-profile=kubernetes \
manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager

# 设置一个集群项
# 若使用 haproxy、keepalived 那么为 `--server=https://192.168.110.20:9443`
# 若使用 nginx方案，那么为 `--server=https://127.0.0.1:8443`
[root@k8s-master-01 pki]# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://127.0.0.1:8443 \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 设置一个环境项，一个上下文
[root@k8s-master-01 pki]# kubectl config set-context system:kube-controller-manager@kubernetes \
--cluster=kubernetes \
--user=system:kube-controller-manager \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 设置一个用户项
[root@k8s-master-01 pki]# kubectl config set-credentials system:kube-controller-manager \
--client-certificate=/etc/kubernetes/pki/controller-manager.pem \
--client-key=/etc/kubernetes/pki/controller-manager-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 设置默认环境
[root@k8s-master-01 pki]# kubectl config use-context system:kube-controller-manager@kubernetes \
--kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
1.6.4 生成kube-scheduler的证书
[root@k8s-master-01 pki]# cat > scheduler-csr.json << EOF 
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

[root@k8s-master-01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=/etc/etcd/ssl/ca-config.json \
-profile=kubernetes \
scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

# 配置一个名为"kubernetes"的集群
[root@k8s-master-01 pki]# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/pki/ca.pem \
--embed-certs=true \
--server=https://127.0.0.1:8443 \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 设置 kube-scheduler 组件的身份验证凭据
[root@k8s-master-01 pki]# kubectl config set-credentials system:kube-scheduler \
--client-certificate=/etc/kubernetes/pki/scheduler.pem \
--client-key=/etc/kubernetes/pki/scheduler-key.pem \
--embed-certs=true \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 设置一个名为"system:kube-scheduler@kubernetes"的上下文
[root@k8s-master-01 pki]# kubectl config set-context system:kube-scheduler@kubernetes \
--cluster=kubernetes \
--user=system:kube-scheduler \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig

# 配置Kubernetes集群中的调度器组件
[root@k8s-master-01 pki]# kubectl config use-context system:kube-scheduler@kubernetes \
--kubeconfig=/etc/kubernetes/scheduler.kubeconfig
1.6.5 生成admin的证书配置
[root@k8s-master-01 pki]# cat > admin-csr.json << EOF 
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF
​
# 生成Kubernetes admin的证书
[root@k8s-master-01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=/etc/etcd/ssl/ca-config.json \
-profile=kubernetes \
admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin
​
# 配置一个名为"kubernetes"的集群
[root@k8s-master-01 pki]# kubectl config set-cluster kubernetes     \
--certificate-authority=/etc/kubernetes/pki/ca.pem     \
--embed-certs=true     \
--server=https://127.0.0.1:8443     \
--kubeconfig=/etc/kubernetes/admin.kubeconfig
​
# 设置 kubernetes-admin 组件的身份验证凭据
[root@k8s-master-01 pki]# kubectl config set-credentials kubernetes-admin  \
--client-certificate=/etc/kubernetes/pki/admin.pem     \
--client-key=/etc/kubernetes/pki/admin-key.pem     \
--embed-certs=true     \
--kubeconfig=/etc/kubernetes/admin.kubeconfig
​
# 设置一个名为"kubernetes-admin@kubernetes"的上下文
[root@k8s-master-01 pki]# kubectl config set-context kubernetes-admin@kubernetes    \
--cluster=kubernetes     \
--user=kubernetes-admin     \
--kubeconfig=/etc/kubernetes/admin.kubeconfig
​
# 配置Kubernetes集群中的调度器组件
[root@k8s-master-01 pki]# kubectl config use-context kubernetes-admin@kubernetes  --kubeconfig=/etc/kubernetes/admin.kubeconfig
1.6.6 创建kube-proxy证书
[root@k8s-master-01 pki]# cat > kube-proxy-csr.json  << EOF 
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:kube-proxy",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

# 生成Kubernetes admin的证书
[root@k8s-master-01 pki]# cfssl gencert \
-ca=/etc/kubernetes/pki/ca.pem \
-ca-key=/etc/kubernetes/pki/ca-key.pem \
-config=/etc/etcd/ssl/ca-config.json \
-profile=kubernetes \
kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy

# 配置一个名为"kubernetes"的集群
[root@k8s-master-01 pki]# kubectl config set-cluster kubernetes     \
--certificate-authority=/etc/kubernetes/pki/ca.pem     \
--embed-certs=true     \
--server=https://127.0.0.1:8443     \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

# 设置 kube-proxy 组件的身份验证凭据
[root@k8s-master-01 pki]# kubectl config set-credentials kube-proxy  \
--client-certificate=/etc/kubernetes/pki/kube-proxy.pem     \
--client-key=/etc/kubernetes/pki/kube-proxy-key.pem     \
--embed-certs=true     \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

# 设置一个名为"kube-proxy@kubernetes"的上下文
[root@k8s-master-01 pki]# kubectl config set-context kube-proxy@kubernetes    \
--cluster=kubernetes     \
--user=kube-proxy     \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

# 配置Kubernetes集群中的调度器组件
[root@k8s-master-01 pki]# kubectl config use-context kube-proxy@kubernetes  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
1.6.7 创建ServiceAccount Key — secret
[root@k8s-master-01 pki]# openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
[root@k8s-master-01 pki]# openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub
1.6.8 将证书发送到其他master节点
[root@k8s-master-01 pki]# for NODE in k8s-master-02 k8s-master-03; do  for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do  scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done;  for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do  scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done

1.7 K8s系统组件配置

1.7.1 etcd配置

  • k8s-master-01配置

[root@k8s-master-01 ~]# cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master-01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.110.21:2380'
listen-client-urls: 'https://192.168.110.21:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.110.21:2380'
advertise-client-urls: 'https://192.168.110.21:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master-01=https://192.168.110.21:2380,k8s-master-02=https://192.168.110.22:2380,k8s-master-03=https://192.168.110.23:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

  • k8s-master-02配置

[root@k8s-master-02 ~]# cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master-02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.110.22:2380'
listen-client-urls: 'https://192.168.110.22:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.110.22:2380'
advertise-client-urls: 'https://192.168.110.22:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master-01=https://192.168.110.21:2380,k8s-master-02=https://192.168.110.22:2380,k8s-master-03=https://192.168.110.23:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

  • k8s-master-03配置

[root@k8s-master-03 ~]# cat > /etc/etcd/etcd.config.yml << EOF 
name: 'k8s-master-03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.110.23:2380'
listen-client-urls: 'https://192.168.110.23:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.110.23:2380'
advertise-client-urls: 'https://192.168.110.23:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master-01=https://192.168.110.21:2380,k8s-master-02=https://192.168.110.22:2380,k8s-master-03=https://192.168.110.23:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF
1.7.2 创建service（所有master节点操作）

  • 创建etcd.service并启动

[root@k8s-master-all ~]# cat > /usr/lib/systemd/system/etcd.service << EOF
​
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
​
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
​
[Install]
WantedBy=multi-user.target
Alias=etcd3.service
​
EOF

  • 创建etcd证书目录

[root@k8s-master-all ~]# mkdir /etc/kubernetes/pki/etcd
[root@k8s-master-all ~]# ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/
[root@k8s-master-all ~]# systemctl daemon-reload
[root@k8s-master-all ~]# systemctl enable --now etcd.service

  • 查看etcd状态

[root@k8s-master-all ~]# export ETCDCTL_API=3
[root@k8s-master-all ~]# etcdctl --endpoints="192.168.110.21:2379,192.168.110.22:2379,192.168.110.23:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.110.21:2379 | 968dab376a943298 |  3.5.12 |   20 kB |      true |      false |         2 |          8 |                  8 |        |
| 192.168.110.22:2379 | 86c98fa15965d9ab |  3.5.12 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
| 192.168.110.23:2379 | 4a6158ec2cc2b579 |  3.5.12 |   20 kB |     false |      false |         2 |          8 |                  8 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

1.8 高可用配置

  • 在Master服务器上操作

1.8.1 NGINX高可用方案
# 安装编译环境
[root@k8s-master-01 ~]# yum install gcc -y
​
# 下载解压nginx二进制文件
[root@k8s-master-01 ~]# wget -c http://nginx.org/download/nginx-1.25.3.tar.gz
[root@k8s-master-01 ~]# tar xf nginx-*.tar.gz
[root@k8s-master-01 ~]# cd nginx-*
​
# 进行编译
[root@k8s-master-01 ~]# ./configure --with-stream --without-http --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
[root@k8s-master-01 ~]# make && make install 
​
# 拷贝编译好的nginx
[root@k8s-master-01 ~]# node='k8s-master-02 k8s-master-03 k8s-node-01 k8s-node-02 k8s-node-03'
[root@k8s-master-01 ~]# for NODE in $node; do scp -r /usr/local/nginx/ $NODE:/usr/local/nginx/; done

  • 写入启动配置

[root@k8s-all ~]# cat > /usr/local/nginx/conf/kube-nginx.conf <<EOF
worker_processes 1;
events {
    worker_connections  1024;
}
stream {
    upstream backend {
      least_conn;
        hash $remote_addr consistent;
        server 192.168.110.21:6443        max_fails=3 fail_timeout=30s;
        server 192.168.110.22:6443        max_fails=3 fail_timeout=30s;
        server 192.168.110.23:6443        max_fails=3 fail_timeout=30s;
    }
    server {
        listen 127.0.0.1:8443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF
​
# 写入启动配置文件
[root@k8s-all ~]# cat > /etc/systemd/system/kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
​
[Service]
Type=forking
ExecStartPre=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -t
ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx
ExecReload=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target
EOF
​
# 启动
[root@k8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。
​
[root@k8s-all ~]# systemctl enable --now kube-nginx.service
# 启用并立即启动kube-nginx.service单元。kube-nginx.service是kube-nginx守护进程的systemd服务单元。
1.8.2 配置Keepalived
[root@k8s-master-01 ~]# yum install keepalived -y
[root@k8s-master-01 ~]# cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
 router_id master1
 script_user root
 enable_script_security
}
​
vrrp_script check_nginx {
  script "/etc/keepalived/check_nginx.sh"
  interval 3
  fall 3
  rise 2
}
​
vrrp_instance Nginx {
 state MASTER
 interface ens33
 virtual_router_id 51
 priority 200
 advert_int 1
 authentication {
   auth_type PASS
   auth_pass XCZKXY
 }
 track_script {
   check_nginx
 }
 virtual_ipaddress {
   192.168.110.20/24
 }
}
EOF
​
# 创建健康检测脚本
[root@k8s-master-01 ~]# cat > /etc/keepalived/check_nginx.sh<<EOF
#!/bin/sh
# nginx down
pid=`ps -C nginx --no-header | wc -l`
if [ $pid -eq 0 ]
then
    systemctl start kube-nginx.service
  sleep 5
    if [ `ps -C nginx --no-header | wc -l` -eq 0 ]
    then
        systemctl stop kube-nginx.service
    else
      exit 0
    fi
fi
EOF
​
[root@k8s-master-01 ~]# chmod +x /etc/keepalived/check_nginx.sh
​
[root@k8s-master-01 ~]# rsync -avz /etc/keepalived/{keepalived.conf,check_nginx.sh} k8s-master-02:/etc/keepalived/
​
[root@k8s-master-01 ~]# rsync -avz /etc/keepalived/{keepalived.conf,check_nginx.sh} k8s-master-03:/etc/keepalived/

  • 其他两个节点修改Keepalivbed配置

[root@k8s-master-02 ~]# sed -i 's/MASTER/BACKUP/' /etc/keepalived/keepalived.conf 
[root@k8s-master-02 ~]# sed -i 's/200/150/' /etc/keepalived/keepalived.conf

[root@k8s-master-03 ~]# sed -i 's/MASTER/BACKUP/' /etc/keepalived/keepalived.conf 
[root@k8s-master-03 ~]# sed -i 's/200/100/' /etc/keepalived/keepalived.conf

# 启动服务
[root@k8s-master-all ~]# systemctl daemon-reload
[root@k8s-master-all ~]# systemctl enable --now keepalived.service

  • 高可用测试

[root@k8s-master-01 ~]# ip a | grep 192.168.110.20/24
    inet 192.168.110.20/24 scope global secondary ens33
    
# 模拟Keepalived宕机
​
[root@k8s-master-01 ~]# systemctl stop keepalived
​
[root@k8s-master-02 ~]# ip a | grep 192.168.110.20/24   # VIP漂移到master-02
    inet 192.168.110.20/24 scope global secondary ens33
[root@k8s-master-03 ~]# ip a | grep 192.168.110.20/24   # master-02宕机
    inet 192.168.110.20/24 scope global secondary ens33
    
[root@k8s-master-03 ~]# ip a | grep 192.168.110.20/24
    inet 192.168.110.20/24 scope global secondary ens33  # VIP漂移到master-03
    
[root@k8s-master-01 ~]# systemctl start keepalived.service  # 恢复后正常
[root@k8s-master-01 ~]# ip a | grep 192.168.110.20/24
    inet 192.168.110.20/24 scope global secondary ens33
     
[root@k8s-all ~]# ping -c 2 192.168.110.20   #确保集群内部可以通讯
PING 192.168.110.20 (192.168.110.20) 56(84) bytes of data.
64 bytes from 192.168.110.20: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 192.168.110.20: icmp_seq=2 ttl=64 time=2.22 ms
​
--- 192.168.110.20 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 1.034/1.627/2.220/0.593 ms

1.9 k8s组件配置

  • 所有k8s节点创建以下目录

[root@k8s-all ~]# mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes
1.9.1 创建apiserver

  • master-01节点配置

[root@k8s-master-01 ~]# cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
​
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
​
[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
      --v=2  \\
      --allow-privileged=true  \\
      --bind-address=0.0.0.0  \\
      --secure-port=6443  \\
      --advertise-address=192.168.110.21 \\
      --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\
      --service-node-port-range=30000-32767  \\
      --etcd-servers=https://192.168.110.21:2379,https://192.168.110.22:2379,https://192.168.110.23:2379 \\
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \\
      --enable-bootstrap-token-auth=true  \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
      --requestheader-allowed-names=aggregator  \\
      --requestheader-group-headers=X-Remote-Group  \\
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
      --requestheader-username-headers=X-Remote-User \\
      --enable-aggregator-routing=true
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
​
[Install]
WantedBy=multi-user.target
​
EOF

  • master-02节点配置

[root@k8s-master-02 ~]# cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
​
[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
      --v=2  \\
      --allow-privileged=true  \\
      --bind-address=0.0.0.0  \\
      --secure-port=6443  \\
      --advertise-address=192.168.110.22 \\
      --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\
      --service-node-port-range=30000-32767  \\
      --etcd-servers=https://192.168.110.21:2379,https://192.168.110.22:2379,https://192.168.110.23:2379 \\
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
      --authorization-mode=Node,RBAC  \\
      --enable-bootstrap-token-auth=true  \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
      --requestheader-allowed-names=aggregator  \\
      --requestheader-group-headers=X-Remote-Group  \\
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
      --requestheader-username-headers=X-Remote-User \\
      --enable-aggregator-routing=true
​
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
​
[Install]
WantedBy=multi-user.target
​
EOF

  • master-03节点配置

[root@k8s-master-03 ~]# cat > /usr/lib/systemd/system/kube-apiserver.service  << EOF
​
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
​
[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
      --v=2  \\
      --allow-privileged=true  \\
      --bind-address=0.0.0.0  \\
      --secure-port=6443  \\
      --advertise-address=192.168.110.23 \\
      --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112  \\
      --service-node-port-range=30000-32767  \\
      --etcd-servers=https://192.168.110.21:2379,https://192.168.110.22:2379,https://192.168.110.23:2379 \\
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \\
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \\
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \\
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \\
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \\
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \\
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \\
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \\
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \\
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \\
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \\
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \\
      --authorization-mode=Node,RBAC  \\
      --enable-bootstrap-token-auth=true  \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \\
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \\
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \\
      --requestheader-allowed-names=aggregator  \\
      --requestheader-group-headers=X-Remote-Group  \\
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \\
      --requestheader-username-headers=X-Remote-User \\
      --enable-aggregator-routing=true
​
Restart=on-failure
RestartSec=10s
LimitNOFILE=65535
​
[Install]
WantedBy=multi-user.target
​
EOF
1.9.2 启动apiserver
[root@k8s-master-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。
​
[root@k8s-master-all ~]# systemctl enable --now kube-apiserver.service
# 启用并立即启动kube-apiserver.service单元。kube-apiserver.service是kube-apiserver守护进程的systemd服务单元。
​
[root@k8s-master-all ~]# systemctl restart kube-apiserver.service
# 重启kube-apiserver.service单元，即重新启动etcd守护进程。

1.10 配置kube-controller-manager service

# 所有master节点配置，且配置相同
# 172.16.0.0/12为pod网段
[root@k8s-master-all ~]# cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
      --v=2 \\
      --bind-address=0.0.0.0 \\
      --root-ca-file=/etc/kubernetes/pki/ca.pem \\
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \\
      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \\
      --service-account-private-key-file=/etc/kubernetes/pki/sa.key \\
      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \\
      --leader-elect=true \\
      --use-service-account-credentials=true \\
      --node-monitor-grace-period=40s \\
      --node-monitor-period=5s \\
      --controllers=*,bootstrapsigner,tokencleaner \\
      --allocate-node-cidrs=true \\
      --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \\
      --cluster-cidr=172.16.0.0/12,fc00:2222::/112 \\
      --node-cidr-mask-size-ipv4=24 \\
      --node-cidr-mask-size-ipv6=120 \\
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

EOF

  • 启动kube-controller-manager

[root@k8s-master-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。

[root@k8s-master-all ~]# systemctl enable --now kube-controller-manager.service
# 启用并立即启动kube-controller-manager.service单元。kube-controller-manager.service是kube-controller-manager守护进程的systemd服务单元。

[root@k8s-master-all ~]# systemctl restart kube-controller-manager.service
# 重启kube-controller-manager.service单元，即重新启动etcd守护进程。

1.11 配置kube-scheduler service

[root@k8s-master-all ~]# cat > /usr/lib/systemd/system/kube-scheduler.service << EOF

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
      --v=2 \\
      --bind-address=0.0.0.0 \\
      --leader-elect=true \\
      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

EOF

  • 启动并查看服务状态

[root@k8s-master-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。

[root@k8s-master-all ~]# systemctl enable --now kube-scheduler.service
# 启用并立即启动kube-scheduler.service单元。kube-scheduler.service是kube-scheduler守护进程的systemd服务单元。

[root@k8s-master-all ~]# systemctl restart kube-scheduler.service
# 重启kube-scheduler.service单元，即重新启动etcd守护进程。

1.12 TLS Bootstrapping配置

1.12.1 在master01上配置
# 设置 Kubernetes 集群配置
[root@k8s-master-01 ~]# kubectl config set-cluster kubernetes     \
--certificate-authority=/etc/kubernetes/pki/ca.pem     \
--embed-certs=true     --server=https://127.0.0.1:8443     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

# 设置凭证信息
[root@k8s-master-01 ~]# kubectl config set-credentials tls-bootstrap-token-user     \
--token=c8ad9c.2e4d610cf3e7426e \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

# 设置上下文信息
[root@k8s-master-01 ~]# kubectl config set-context tls-bootstrap-token-user@kubernetes     \
--cluster=kubernetes     \
--user=tls-bootstrap-token-user     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

# 设置当前上下文
[root@k8s-master-01 ~]# kubectl config use-context tls-bootstrap-token-user@kubernetes     \
--kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

[root@k8s-master-01 ~]# mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config
1.12.2 查看集群状态
[root@k8s-master-01 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok        
scheduler            Healthy   ok        
etcd-0               Healthy   ok 

# 写入bootstrap-token
[root@k8s-master-01 ~]# cat > bootstrap.secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-c8ad9c
  namespace: kube-system
type: bootstrap.kubernetes.io/token
stringData:
  description: "The default bootstrap token generated by 'kubelet '."
  token-id: c8ad9c
  token-secret: 2e4d610cf3e7426e
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"
  auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-bootstrap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:default-node-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-autoapprove-certificate-rotation
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver
EOF

[root@k8s-master-01 ~]# kubectl apply -f bootstrap.secret.yaml

1.13 node节点配置

  • 在master01上将证书复制到node节点

[root@k8s-master-01 ~]# cd /etc/kubernetes/
 
[root@k8s-master-01 ~]# for NODE in k8s-master-02 k8s-master-03 k8s-node-01 k8s-node-02 k8s-node-03; do ssh $NODE mkdir -p /etc/kubernetes/pki; for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig kube-proxy.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done; done
1.13.1 当使用docker作为Runtime
# 所以节点操作
[root@k8s-all ~]# cat > /usr/lib/systemd/system/kubelet.service << EOF

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=network-online.target firewalld.service cri-docker.service docker.socket containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
    --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig  \\
    --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
    --config=/etc/kubernetes/kubelet-conf.yml \\
    --container-runtime-endpoint=unix:///run/cri-dockerd.sock  \\
    --node-labels=node.kubernetes.io/node= 


[Install]
WantedBy=multi-user.target
EOF
1.13.2 所有k8s节点创建kubelet的配置文件
[root@k8s-all ~]# cat > /etc/kubernetes/kubelet-conf.yml <<EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF
1.13.3 启动kubelet
[root@k8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。

[root@k8s-all ~]# systemctl enable --now kubelet.service
# 启用并立即启动kubelet.service单元。kubelet.service是kubelet守护进程的systemd服务单元。

[root@k8s-all ~]# systemctl restart kubelet.service
# 重启kubelet.service单元，即重新启动kubelet守护进程。
1.13.4 查看集群
[root@k8s-master-01 ~]# kubectl get nodes
NAME            STATUS     ROLES    AGE     VERSION
k8s-master-01   NotReady   <none>   13m     v1.30.1
k8s-master-02   NotReady   <none>   8m53s   v1.30.1
k8s-master-03   NotReady   <none>   8m52s   v1.30.1
k8s-node-01     NotReady   <none>   5m37s   v1.30.1
k8s-node-02     NotReady   <none>   5m38s   v1.30.1
k8s-node-03     NotReady   <none>   5m39s   v1.30.1
​
# 查看容器运行时
[root@k8s-master-01 ~]# kubectl describe node | grep Runtime
  Container Runtime Version:  docker://25.0.3
  Container Runtime Version:  docker://25.0.3
  Container Runtime Version:  docker://25.0.3
  Container Runtime Version:  docker://25.0.3
  Container Runtime Version:  docker://25.0.3
  Container Runtime Version:  docker://25.0.3

1.14 kube-proxy配置

1.14.1将kubeconfig发送至其他节点
# master-01执行
[root@k8s-master-01 ~]# for NODE in k8s-master-02 k8s-master-03 k8s-node-01 k8s-node-02 k8s-node-03; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig; done
1.14.2 所有k8s节点添加kube-proxy的service文件
[root@k8s-all ~]# cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
​
[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/etc/kubernetes/kube-proxy.yaml \\
  --cluster-cidr=172.16.0.0/12,fc00:2222::/112 \\
  --v=2
Restart=always
RestartSec=10s
​
[Install]
WantedBy=multi-user.target
​
EOF
1.14.3 所有k8s节点添加kube-proxy的配置
[root@k8s-all ~]# cat > /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/12,fc00:2222::/112
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms
EOF
1.14.4 启动kube-proxy
[root@k8s-all ~]# systemctl daemon-reload
# 用于重新加载systemd管理的单位文件。当你新增或修改了某个单位文件（如.service文件、.socket文件等），需要运行该命令来刷新systemd对该文件的配置。
​
[root@k8s-all ~]# systemctl enable --now kube-proxy.service
# 启用并立即启动kube-proxy.service单元。kube-proxy.service是kube-proxy守护进程的systemd服务单元。
​
[root@k8s-all ~]# systemctl restart kube-proxy.service
# 重启kube-proxy.service单元，即重新启动kube-proxy守护进程。

1.15 安装网络插件

  • CentOS 7 要升级libseccomp 不然 无法安装网络插件

# 升级runc
[root@k8s-master-01 ~]# wget -c https://mirrors.chenby.cn/https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64
​
[root@k8s-master-01 ~]# install -m 755 runc.amd64 /usr/local/sbin/runc
[root@k8s-master-01 ~]# cp -p /usr/local/sbin/runc  /usr/local/bin/runc
[root@k8s-master-01 ~]# cp -p /usr/local/sbin/runc  /usr/bin/runc
​
#查看当前版本
[root@k8s-master-01 ~]# rpm -qa | grep libseccomp
libseccomp-2.3.1-4.el7.x86_64

  • 安装Calico

[root@k8s-master-01 ~]# wget -c https://gitee.com/kong-xiangyuxcz/svn/releases/download/V3.25.0/calico.yaml
​
# 所以节点手动拉取镜像
docker pull docker.io/calico/cni:v3.25.0
docker pull docker.io/calico/node:v3.25.0
docker pull docker.io/calico/kube-controllers:v3.25.0
​
[root@k8s-master-01 ~]# kubectl apply -f calico.yaml 
​
[root@k8s-master-01 ~]# kubectl get pod -n kube-system
NAME                                       READY   STATUS     RESTARTS   AGE
calico-kube-controllers-5b9b456c66-n9lz6   1/1     Running    0          2m52s
calico-node-49v62                          1/1     Running    0          2m52s
calico-node-64blt                          1/1     Running    0          2m52s
calico-node-668qt                          1/1     Running    0          2m52s
calico-node-9ktxk                          1/1     Running    0          2m52s
calico-node-njgvp                          1/1     Running    0          2m52s
​
[root@k8s-master-01 ~]# kubectl get node
NAME            STATUS     ROLES    AGE   VERSION
k8s-master-01   Ready      <none>   52m   v1.30.1
k8s-master-02   Ready      <none>   47m   v1.30.1
k8s-master-03   Ready      <none>   47m   v1.30.1
k8s-node-01     Ready      <none>   44m   v1.30.1
k8s-node-02     Ready      <none>   44m   v1.30.1
k8s-node-03     Ready      <none>   44m   v1.30.1

1.16 安装命令行自动补全功能

yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/2116708.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

MySQL—死锁

MySQL—死锁

什么是死锁&#xff1f; 好比是两个事务都在等待对方释放锁&#xff0c;之后进行下一步操作&#xff0c;但是最后双方都没有释放资源&#xff0c;所以导致一直处于等待的状态。 但是服务器不会让死锁的状态一直持续&#xff0c;会关闭其中一个影响较小的事务&#xff08;右边的…
阅读更多...
紫光展锐完成Android 15同步升级，驱动技术创新与生态共赢

紫光展锐完成Android 15同步升级，驱动技术创新与生态共赢

近日&#xff0c;紫光展锐宣布&#xff0c;展锐5G移动平台T820、T770、T765、T760、T750以及4G平台T620、T619、T616、T615、T612、T606&#xff0c;完成Android 15同步升级。相较于过往Android发布&#xff0c;今年同步升级Android 15主要有三大提升&#xff1a; ■ 紫光展锐实…
阅读更多...
SaToken认证授权快速上手

SaToken认证授权快速上手

SA-TOKEN认证授权快速上手 SaToken简介认证授权实现1 创建5张测试基础表2 配置文件3 Sa-Token的全局拦截器4 自定义权限认证接口扩展5 SaToken用户控制层 测试验证用户2登录验证权限用户1登录验证权限项目代码结构 SaToken简介 SaToken 是一个轻量级 Java 权限认证框架&#x…
阅读更多...
【信号】信号的产生

【信号】信号的产生

信号的概念 什么是信号&#xff1f;我们生活中的红绿灯&#xff0c;闹钟&#xff0c;外面电话等等这些都是信号&#xff0c;我们是怎么认识这些信号的&#xff0c;我们认识这些信号&#xff0c;并且知道这些信号的处理方法&#xff0c;对于进程来说&#xff0c;也会认识相应的…
阅读更多...
容器编排平台Kubernetes简介

容器编排平台Kubernetes简介

目录 什么是K8s 为什么需要K8s 什么是容器(Contianer) K8s能做什么&#xff1f; K8s的架构原理 控制平面(Control plane) kube-apiserver etcd kube-scheduler kube-controller-manager cloud-controller-manager 小结 节点组件(Node) container runtime Pod kubelet ku…
阅读更多...
Chrome、Edge、360及Firefox浏览器加载多个ActiveX插件的介绍

Chrome、Edge、360及Firefox浏览器加载多个ActiveX插件的介绍

allWebPlugin简介 allWebPlugin中间件是一款为用户提供安全、可靠、便捷的浏览器插件服务的中间件产品&#xff0c;致力于将浏览器插件重新应用到所有浏览器。它将现有ActiveX控件直接嵌入浏览器&#xff0c;实现插件加载、界面显示、接口调用、事件回调等。支持Chrome、Firefo…
阅读更多...
vc-align源码分析 -- ant-design-vue系列

vc-align源码分析 -- ant-design-vue系列

vc-align源码分析 源码地址&#xff1a;https://github.com/vueComponent/ant-design-vue/tree/main/components/vc-align 1 基础代码 1.1 名词约定 需要对齐的节点叫source&#xff0c;对齐的目标叫target。 1.2 props 提供了两个参数&#xff1a; align&#xff1a;对…
阅读更多...
成为优秀程序员-代码篇

成为优秀程序员-代码篇

1. 序言 刚毕业参加工作时候&#xff0c;公司正在快速扩张&#xff0c;我入职的时候组内刚刚招了一大波人&#xff0c;当时leader提出集体cr代码来拉齐团队内的编码规范&#xff0c;每当有对于相对重要改动大的项目就会集体cr代码&#xff0c;老板想法初衷是好的&#xff0c;但…
阅读更多...
esp32 gpio 初始化不同类型的管脚，产生脉冲，发生中断

esp32 gpio 初始化不同类型的管脚，产生脉冲，发生中断

硬件&#xff1a;D4与D18 连接&#xff0c;二极管接D15与3.3v脚 图片 二极管同期性点亮&#xff0c;间隔1秒 参考esp32官网程序&#xff0c;从此程序可以看出&#xff0c;中断程序没有处理任何数据&#xff0c;只是把中断发生的事件存入队列。而用另一新线程来处理中断事务。…
阅读更多...
GNSS CTS GNSS Start and Location Flow of Android15

GNSS CTS GNSS Start and Location Flow of Android15

目录 1. 本文概述2.CTS 测试3.Gnss Flow3.1 Gnss Start Flow3.2 Gnss Location Output Flow 1. 本文概述 本来是为了做Android 14 Gnss CTS 的相关环境的搭建和测试&#xff0c;然后在测试中遇到了一些问题&#xff0c;去寻找CTS源码(/cts/tests/tests/location/src/android/l…
阅读更多...
Vue3-05_组件高级

Vue3-05_组件高级

背景 对组件的进一步了解,如组件之间通信等知识点&#xff0c;根据教程实现购物车功能&#xff0c;并修复原本的bug. watch 侦听器 用途 watch 侦听器允许开发者监视数据的变化&#xff0c;从而针对数据的变化做特定的操作。例如&#xff0c;监视用户名的变化并发起请求&am…
阅读更多...
大模型AI一体机对行业的帮助

大模型AI一体机对行业的帮助

大模型AI一体机&#xff0c;如AntSKPro AI离线知识库一体机&#xff0c;是专门为企业和机构设计的集成系统&#xff0c;旨在提供高效的人工智能服务。这类一体机通常包含预训练的大型机器学习模型&#xff0c;以及必要的硬件和软件资源&#xff0c;以支持复杂的数据处理和分析任…
阅读更多...
maven 编译构建可以执行的jar包

maven 编译构建可以执行的jar包

&#x1f49d;&#x1f49d;&#x1f49d;欢迎莅临我的博客&#xff0c;很高兴能够在这里和您见面&#xff01;希望您在这里可以感受到一份轻松愉快的氛围&#xff0c;不仅可以获得有趣的内容和知识&#xff0c;也可以畅所欲言、分享您的想法和见解。 推荐&#xff1a;「storm…
阅读更多...
网易云音乐歌单下载器

网易云音乐歌单下载器

最近要帮小朋友下载一些小学的诗词mp3&#xff0c;找了各种工具&#xff0c;还是这个好使 yun-playlist-downloader: 网易云音乐歌单下载器 特性 支持歌单 / 专辑 / 电台音质选择下载超时 / 重试再次下载默认跳过已下载部分, 使用 content-length 匹配自定义文件名下载进度显…
阅读更多...
【代码随想录训练营第42期 Day53打卡 - 图论Part4 - 卡码网 110. 字符串接龙 105. 有向图的完全可达性

【代码随想录训练营第42期 Day53打卡 - 图论Part4 - 卡码网 110. 字符串接龙 105. 有向图的完全可达性

目录 一、个人感受 二、题目与题解 题目一&#xff1a;卡码网 110. 字符串接龙 题目链接 题解&#xff1a;BFS哈希 题目二&#xff1a;卡码网 105. 有向图的完全可达性 题目链接 题解&#xff1a;DFS 三、小结 一、个人感受 对于两大基本搜索&#xff1a; 深度优先搜…
阅读更多...
JDBC：连接数据库

JDBC：连接数据库

文章目录 报错 报错 Exception in thread “main” java.sql.SQLException: Can not issue SELECT via executeUpdate(). 最后这里输出的还是地址&#xff0c;就是要重写toString()方法&#xff0c;但是我现在还不知道怎么写 修改完的代码&#xff0c;但是数据库显示&#…
阅读更多...
WebGL系列教程二（环境搭建及初始化Shader）

WebGL系列教程二（环境搭建及初始化Shader）

目录 1 前言2 新建html页面3 着色器介绍3.1 顶点着色器、片元着色器与光栅化的概念3.2 声明顶点着色器3.3 声明片元着色器 4 坐标系(右手系)介绍5 着色器初始化5.1 给一个画布canvas5.2 获取WebGL对象5.3 创建着色器对象5.4 获取着色器对象的源5.5 绑定着色器的源5.6 编译着色器…
阅读更多...
MiniGPT-3D, 首个高效的3D点云大语言模型，仅需一张RTX3090显卡，训练一天时间，已开源

MiniGPT-3D, 首个高效的3D点云大语言模型，仅需一张RTX3090显卡，训练一天时间，已开源

项目主页&#xff1a;https://tangyuan96.github.io/minigpt_3d_project_page/ 代码&#xff1a;https://github.com/TangYuan96/MiniGPT-3D 论文&#xff1a;https://arxiv.org/pdf/2405.01413 MiniGPT-3D在多个任务上取得了SoTA&#xff0c;被ACM MM2024接收&#xff0c;只拥…
阅读更多...
佰朔资本：9月首选行业为汽车、电子、医药生物等

佰朔资本：9月首选行业为汽车、电子、医药生物等

5—8月商场接连调整&#xff0c;9月开端进入成绩空窗期&#xff0c;流动性和政策改动从头成为商场中心驱动力&#xff0c;风格切换先行&#xff0c;对当时的商场能够豁达一些。价值和生长风格切换的拐点开始闪现&#xff0c;生长相对价值的成绩优势开端走扩&#xff0c;美联储降…
阅读更多...
Axure中继器介绍

Axure中继器介绍

中继器我们一般在处理重复性比较高的任务时&#xff0c;能让我们达到事半功倍的效果&#xff0c;中继器在整个axure中属于复杂程度比较高的功能&#xff0c;我们今天大致讲一下常用的方法即可。 一、声明一个中继器 默认展示为三行。 点击样式&#xff0c;这里我们可以添加删…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023