配置K8S主从集群前置准备操作

一：主节点操作 查看主机域名->编辑域名

[root@localhost ~]# hostname
localhost.localdomain
[root@localhost ~]# hostnamectl set-hostname nodemaster
[root@localhost ~]# hostname
nodemaster
[root@localhost ~]#

1.1 编辑HOST 从节点也做相应操作

[root@vbox-master-01-vbox-01 ~]# vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.101  master-k8s-vbox

1.2 从节点操作 查看从节点102域名->编辑域名

[root@localhost ~]# hostname
localhost.localdomain
[root@localhost ~]# hostnamectl set-hostname nodeslavetwo
[root@localhost ~]# hostname
nodeslavetwo

1.3 从节点操作 查看从节点103域名->编辑域名

[root@localhost /]# hostname
localhost.localdomain
[root@localhost /]# hostnamectl set-hostname nodeslavethree
[root@localhost /]# hostname
nodeslavethree

二：安装自动填充，虚拟机默认没有

[root@vbox-master-01-vbox-01 ~]# yum -y install bash-completion
已加载插件：fastestmirror, product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.

Determining fastest mirrors
 * base: ftp.sjtu.edu.cn
 * extras: mirrors.nju.edu.cn
 * updates: mirrors.aliyun.com

三：关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

四：关闭交换空间

free -h
swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab
free -h

五：禁用 Selinux

sed -i “s/^SELINUX=enforcing/SELINUX=disabled/g” /etc/sysconfig/selinux

[root@nodemaster /]# sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux

六： 允许 ip tables 检查桥接流量

iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT

七：设置K8S相关系统参数

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

7.1：配置Docker 阿里源

mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://hnkfbj7x.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

7.2：添加阿里云 k8s 镜像仓库

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

7.3：安装 k8s 1.14.0 所需依赖

7.3.1：设置所 sysctl 参数，重新启动后配置不变

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

7.3.2：应用 sysctl 参数而不重新启动

sudo sysctl --system

八：安装K8S – kubelet，kubeadm，kubectl核心组件

8.1：安装命令

yum install -y kubelet-1.25.5-0 kubeadm-1.25.5-0 kubectl-1.25.5-0 --disableexcludes=kubernetes --nogpgcheck

8.2：查看服务状态

systemctl status kubelet

8.3：发现异常 “command failed” err="failed to load kubelet config file(注意：此异常和主从节点配置有关，放在后面解决)

"command failed" err="failed to load kubelet config file, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory, path: /var/lib/kubelet/config.yaml"

8.3.1：查看具体报错

journalctl -u kubelet.service

8.3.2： 发现异常 Flag --cgroup-driver has been deprecated, This parameter should be set via the config file

Flag --cgroup-driver has been deprecated, This parameter should be set via the config file

 Unable to update cni config: no networks found i

8.4：针对上述异常操作如下

8.4.1： 修改docker cgroup驱动

cat > /etc/docker/daemon.json <<EOF
{"exec-opts": ["native.cgroupdriver=systemd"]}
EOF

[root@vbox-master-01-vbox-01 ~]# cat > /etc/docker/daemon.json <<EOF
> {"exec-opts": ["native.cgroupdriver=systemd"]}
> EOF
[root@vbox-master-01-vbox-01 ~]#

8.4.2：配置网络，选择 Calico

8.5： 下载相关操作

8.5.1：安装wget

yum -y install wget

8.5.2：下载网络

wget --no-check-certificate https://projectcalico.docs.tigera.io/archive/v3.25/manifests/calico.yaml

8.5.3：修改 calico.yaml

yum -y install vim-enhanced
vim calico.yaml

 # 在 - name: CLUSTER_TYPE 下方添加如下内容
- name: CLUSTER_TYPE
  value: "k8s,bgp"
  # 下方为新增内容
- name: IP_AUTODETECTION_METHOD
  value: "interface=网卡名称"


# INTERFACE_NAME=ens33
# sed -i '/k8s,bgp/a \            - name: IP_AUTODETECTION_METHOD\n              value: "interface=INTERFACE_NAME"' calico.yaml
# sed -i "s#INTERFACE_NAME#$INTERFACE_NAME#g" calico.yaml

8.5.4：配置K8S 网络

kubectl apply -f calico.yaml

九：再次查看 pods，nodes

kubectl get nodes -o wide

9.1：注意异常： The connection to the server localhost:8080 was refused(和主从节点有关，先kubeinit没问题)

[root@vbox-master-01-vbox-01 ~]# kubectl apply -f calico.yaml
The connection to the server localhost:8080 was refused - did you specify the right host or port?

9.2：解决方式

cd /etc/kubernetes/
echo “export KUBECONFIG=/etc/kubernetes/kubelet.conf” >> /etc/profile
source /etc/profile

[root@vbox-master-01-vbox-01 ~]# kubectl apply -f calico.yaml
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@vbox-master-01-vbox-01 ~]#
[root@vbox-master-01-vbox-01 ~]# cd /etc/kubernetes/
[root@vbox-master-01-vbox-01 kubernetes]# echo "export KUBECONFIG=/etc/kubernetes/kubelet.conf" >> /etc/profile
[root@vbox-master-01-vbox-01 kubernetes]#
[root@vbox-master-01-vbox-01 kubernetes]# source /etc/profile
[root@vbox-master-01-vbox-01 kubernetes]#

9.3：再次执行抛异常处的命令 kubeadm init

kubeadm init

9.4：注意异常 hostname和 CRI v1 runtime API

[WARNING Hostname]: hostname "vbox-master-01-vbox-01": lookup vbox-master-01-vbox-01 on 192.168.1.1:53: no such host

error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: time="2023-05-02T02:16:37+08:00" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1

9.5：解决 kubeadm init 相关异常

9.5.1：编辑vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.101  vbox-master-01-vbox-01

9.5.2：ps aux | grep docker | grep -v grep

9.5.3：cat /etc/containerd/config.toml

查看到disabled_plugins = [“cri”]

9.5.4：rm -fr /etc/containerd/config.toml

9.5.5：systemctl restart containerd

9.5.6：systemctl status containerd.service

[root@vbox-master-01-vbox-01 /]# systemctl status containerd.service
● containerd.service - containerd container runtime
   Loaded: loaded (/usr/lib/systemd/system/containerd.service; disabled; vendor preset: disabled)
   Active: active (running) since 二 2023-05-02 03:06:09 CST; 2min 35s ago
     Docs: https://containerd.io
  Process: 20008 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 20010 (containerd)
    Tasks: 45
   Memory: 117.1M
   CGroup: /system.slice/containerd.service
           ├─11977 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 82a64934d6d49e4a3b55883f2fa3c959a1d16f091133644c2f2ab75de46ec09e -address ...
           ├─12134 /usr/bin/containerd-shim-runc-v2 -namespace moby -id f0d1f44c6dd8bb6015af52737aaebcdec2a35f6df0ff7d538c399c19bc622a41 -address ...
           ├─13051 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 6de7f49404eb3aa5ea14e02405c973bc850217a519a8b3a85bf47bf8aae23e2d -address ...
           └─20010 /usr/bin/containerd

5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.691263973+08:00" level=error msg="failed to load cni du...onfig"
5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.691710874+08:00" level=info msg=serving... address=/run....ttrpc
5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.691797968+08:00" level=info msg=serving... address=/run...d.sock
5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.691949132+08:00" level=info msg="containerd successfull...1014s"
5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.705709657+08:00" level=info msg="Start subscribing cont...event"
5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.705803917+08:00" level=info msg="Start recovering state"
5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.705874098+08:00" level=info msg="Start event monitor"
5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.705891342+08:00" level=info msg="Start snapshots syncer"
5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.705900497+08:00" level=info msg="Start cni network conf...fault"
5月 02 03:06:09 vbox-master-01-vbox-01 containerd[20010]: time="2023-05-02T03:06:09.705907733+08:00" level=info msg="Start streaming server"
Hint: Some lines were ellipsized, use -l to show in full.

十：解决 & 再次执行kubeadm init 查看

10.1：执行kubeadm init

[root@vbox-master-01-vbox-01 kubernetes]# kubeadm init
I0502 02:25:08.004988   18148 version.go:256] remote version is much newer: v1.27.1; falling back to: stable-1.25
[init] Using Kubernetes version: v1.25.9
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'