Linux随记（十）

一、harborv2.6.4 --> harbor-offline-installer-v2.11.0-rc3 --> v2.9.4 – 随记

漏洞信息：

CVE-2023-20902
	timing condition in Harbor 2.6.x and below, Harbor 2.7.2 and below,  Harbor 2.8.2 and below, and Harbor 1.10.17 and below allows an attacker with network access to create jobs/stop job tasks and retrieve job task information.
---
CVE-2022-46463
	an access control issue in Harbor v1.X.X to v2.5.3 allows attackers to access public and private image repositories without authentication. NOTE: the vendor's position is that this "is clearly described in the documentation as a feature."

环境信息：

harbor1 /nfs_server   1.101 、 harbor2 1.102 、 keepalived 1.200

随记内容：

#1
原版本 v2.6
升级到v2.11 (这玩意有问题，只能登录主ip ，副IP登不上，而且push一直 retry)
然后放弃v2.11 ，去弄v2.9. （这时候主/副都能登录了。）
虽说主副都是挂载nfs的 /data/harbor。 但是他双方的内容是不一致的，


#2 这时候去注释yml默认的pg，使用其external database。
搭建了个主从pgsql。不懂为啥harbor-core这时候报错了（说什么脏数据）
后面用了个非主流方法
（主IP还是使用内部的pgsql ， 副IP使用外接pgsql，配置连接到主IP的 pgsql认他为主库）

主–部署pgsql的docker-compose.yml内容：

version: '2'

services:
  postgresql-master:
    image: bitnami/postgresql:11.14.0
    ports:
      - '5432:5432'
    volumes:
      - 'postgresql_master_data:/bitnami/postgresql'
    environment:
      - 'POSTGRESQL_USERNAME=harbor_db'
      - 'POSTGRESQL_PASSWORD=123'
      - 'POSTGRESQL_POSTGRES_PASSWORD=123'
      - 'POSTGRESQL_REPLICATION_MODE=master'
      - 'POSTGRESQL_REPLICATION_USER=repl_user'
      - 'POSTGRESQL_REPLICATION_PASSWORD=repl_password'
      - 'POSTGRESQL_SYNCHRONOUS_COMMIT_MODE=on'
      - 'POSTGRESQL_NUM_SYNCHRONOUS_REPLICAS=1'

volumes:
  postgresql_master_data:
    driver: local

从-部署pgsql的docker-compose.yml内容：

version: '2'

services:
  postgresql-slave:
    image: bitnami/postgresql:11.14.0
    ports:
      - '5432:5432'
    volumes:
      - 'postgresql_slave_data:/bitnami/postgresql'
    environment:
      - 'POSTGRESQL_USERNAME=harbor_db'
      - 'POSTGRESQL_PASSWORD=123'
      - 'POSTGRESQL_POSTGRES_PASSWORD=123'
      - 'POSTGRESQL_REPLICATION_MODE=slave'
      - 'POSTGRESQL_REPLICATION_USER=repl_user'
      - 'POSTGRESQL_REPLICATION_PASSWORD=repl_password'
      - 'POSTGRESQL_MASTER_HOST=10.xx.xx.xx'   #这里写master pg库的IP
      - 'POSTGRESQL_MASTER_PORT_NUMBER=5432'

volumes:
  postgresql_slave_data:
    driver: local

二、iptables处理nfs漏洞（目标主机showmount -e信息泄露(CVE-1999-0554)），并加固其他服务访问 – 随记。

环境信息

K8s 、docker 、nfs 、mysql、iptables
注意下面10.xx.xx.0/24 换成实际的环境IP段。
iptables -nL 看下结果

参考：
https://blog.csdn.net/sxf1061700625/article/details/126844812
https://blog.csdn.net/weixin_46516542/article/details/124279644
https://blog.csdn.net/baidu_35848778/article/details/132883593

执行的iptables语句
限制80 / 3306端口（ /符号后面的端口，在-dports后换成对应的端口号即可）

iptables -I INPUT -p tcp -m multiport --dports 80 -j DROP
iptables -I INPUT -s 127.0.0.1 -p tcp -m multiport --dports 80 -j ACCEPT
iptables -I INPUT -s 10.xx.xx.0/24 -p tcp -m multiport --dports 80 -j ACCEPT
iptables -I INPUT -s 10.244.0.0/16 -p tcp -m multiport --dports 80 -j ACCEPT
iptables -I INPUT -s 10.96.0.0/12 -p tcp -m multiport --dports 80 -j ACCEPT

iptables -I DOCKER-USER -p tcp -m multiport --dports 80 -j DROP
iptables -I DOCKER-USER -s 127.0.0.1 -p tcp -m multiport --dports 80 -j ACCEPT
iptables -I DOCKER-USER -s 10.xx.xx.0/24 -p tcp -m multiport --dports 80 -j ACCEPT
iptables -I DOCKER-USER -s 10.244.0.0/16 -p tcp -m multiport --dports 80 -j ACCEPT
iptables -I DOCKER-USER -s 10.96.0.0/12 -p tcp -m multiport --dports 80 -j ACCEPT

iptables -I DOCKER -p tcp -m multiport --dports 80 -j DROP
iptables -I DOCKER -s 127.0.0.1 -p tcp -m multiport --dports 80 -j ACCEPT
iptables -I DOCKER -s 10.xx.xx.0/24 -p tcp -m multiport --dports 80 -j ACCEPT
iptables -I DOCKER -s 10.244.0.0/16 -p tcp -m multiport --dports 80 -j ACCEPT
iptables -I DOCKER -s 10.96.0.0/12 -p tcp -m multiport --dports 80 -j ACCEPT

限制22端口

iptables -A INPUT -p tcp --dport 22 -s 172.xx.xx.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -s 10.xx.xx.129/29 -j ACCEPT

限制其他端口

#### 集群内
iptables -I INPUT -p tcp -m multiport --dports 10250,2379,2380,6443 -j DROP
iptables -I INPUT -s 127.0.0.1 -p tcp -m multiport --dports 10250,2379,2380,6443 -j ACCEPT
iptables -I INPUT -s 10.xx.xx.0/24 -p tcp -m multiport --dports 10250,2379,2380,6443 -j ACCEPT
iptables -I INPUT -s 10.244.0.0/16 -p tcp -m multiport --dports 10250,2379,2380,6443 -j ACCEPT
iptables -I INPUT -s 10.96.0.0/12 -p tcp -m multiport --dports 10250,2379,2380,6443 -j ACCEPT

查看k8s / docker的相关ip段信息（calico、 svc 、docker 等信息）

ip addr show  

docker ip范围  
docker network inspect bridge
		"Subnet": "172.17.0.0/16",

k8s的ip范围：
kubectl cluster-info dump | grep -i bind-address -A 6
	--cluster-cidr=10.244.0.0/16,2001:db8:42:0::/48   

kubectl get cm -n kube-system  kubeadm-config   -oyaml
 podSubnet: 10.244.0.0/16,2001:db8:42:0::/48
 serviceSubnet: 10.96.0.0/12,2001:db8:42:1::/108

三、bclinux 8.6的OpenSSL 1.1.1k升级到OpenSSL 3.1.6 （2024年6月版本）（在线和离线rpm包安装方式）

背景描述：

由于OpenSSL 安全漏洞(CVE-2024-2511)）, 而且openssl已经停止维护v1.1.1的。所以选择升级到v3.1.xx版本。
（生产环境是没外网没ISO镜像源的，只能在虚拟机先联网操作后，再去生产上操作。）

3.1 升级过程：

#0 当前openssl 版本
[root@localhost ~]# openssl version
OpenSSL 1.1.1k  FIPS 25 Mar 2021

PRETTY_NAME="BigCloud Enterprise Linux 8.6 (Core)"

#1 安装依赖
#yum groupinstall "Development Tools" -y
#yum install -y gcc perl-Digest-SHA perl-IPC-Cmd perl-Data-Dump zlib-devel

#实际yum的    rpm -Uvh *.rpm
yum install -y gcc make perl-IPC-Cmd  perl-Pod-Html

#2 下载openssl包
wget https://www.openssl.org/source/openssl-3.1.6.tar.gz
tar xf openssl-3.1.6.tar.gz

#3 编译安装
cd openssl-3.1.6
./config enable-fips --prefix=/usr/local --openssldir=/usr/local/openssl

make -j 16 && make install


#4 配置环境
#openssl文件位于/usr/local/bin下
#配置文件位于/usr/local/ssl下
#库文件位于/usr/local/lib64下

mv /usr/bin/openssl /usr/bin/openssl.bak

#5 创软连接 或复制（二选一）
ln -s /usr/local/bin/openssl /usr/bin/openssl
ln -s /usr/local/include/openssl /usr/include/openssl

#cp /usr/local/bin/openssl /usr/bin/openssl
#cp -rp /usr/local/include/openssl /usr/include/

#6 动态链接库（二选一）
#方法一
echo "/usr/local/lib64" > /etc/ld.so.conf.d/openssl.conf

#方法二  创软连或复制
#cp /usr/local/lib64/libssl.so.3  /usr/lib/libssl.so.3
#cp /usr/local/lib64/libcrypto.so.3 /usr/lib/libcrypto.so.3

#软连
#ln -s /usr/local/lib64/libssl.so.3 /usr/lib/libssl.so.3
#ln -s /usr/local/lib64/libcrypto.so.3 /usr/lib/libcrypto.so.3

#7 验证  END
ldconfig -v
]$  openssl version
OpenSSL 3.1.6 4 Jun 2024 (Library: OpenSSL 3.1.6 4 Jun 2024)

3.2 排障过程：（内容没排版，可略）

简单来说就是缺什么依赖/包，就去安装即可。
##### 报错1  you may need to install the IPC::Cmd module

所以
[root@localhost openssl-3.1.6]# yum install perl-IPC-Cmd
上次元数据过期检查：3:21:20 前，执行于 2024年06月14日 星期五 20时25分07秒。
依赖关系解决。
============================================================================================================================================================================
 软件包                                               架构                           版本                                           仓库                               大小
============================================================================================================================================================================
安装:
 perl-IPC-Cmd                                         noarch                         2:1.02-1.0.1.an8                               AppStream                          38 k
安装依赖关系:
 perl-ExtUtils-MM-Utils                               noarch                         1:7.34-1.0.1.an8                               AppStream                          16 k
 perl-Locale-Maketext                                 noarch                         1.28-396.0.1.an8                               AppStream                          94 k
 perl-Locale-Maketext-Simple                          noarch                         1:0.21-421.0.1.an8                             AppStream                          78 k
 perl-Module-CoreList                                 noarch                         1:5.20181130-1.0.1.an8                         AppStream                          81 k
 perl-Module-Load                                     noarch                         1:0.32-395.0.1.an8                             AppStream                          16 k
 perl-Module-Load-Conditional                         noarch                         0.68-395.0.1.an8                               AppStream                          21 k
 perl-Module-Metadata                                 noarch                         1.000033-395.0.1.an8                           AppStream                          39 k
 perl-Params-Check                                    noarch                         1:0.38-395.0.1.an8                             AppStream                          21 k
 perl-version                                         x86_64                         6:0.99.24-1.el8                                AppStream                          66 k

事务概要
============================================================================================================================================================================
安装  10 软件包

总下载：469 k
安装大小：1.3 M
确定吗？[y/N]： y
下载软件包：
(1/10): perl-ExtUtils-MM-Utils-7.34-1.0.1.an8.noarch.rpm                                                                                     60 kB/s |  16 kB     00:00    
(2/10): perl-IPC-Cmd-1.02-1.0.1.an8.noarch.rpm                                                                                              120 kB/s |  38 kB     00:00    
(3/10): perl-Locale-Maketext-Simple-0.21-421.0.1.an8.noarch.rpm                                                                             690 kB/s |  78 kB     00:00    
(4/10): perl-Locale-Maketext-1.28-396.0.1.an8.noarch.rpm                                                                                    256 kB/s |  94 kB     00:00    
(5/10): perl-Module-Load-0.32-395.0.1.an8.noarch.rpm                                                                                        260 kB/s |  16 kB     00:00    
(6/10): perl-Module-Load-Conditional-0.68-395.0.1.an8.noarch.rpm                                                                            357 kB/s |  21 kB     00:00    
(7/10): perl-Module-Metadata-1.000033-395.0.1.an8.noarch.rpm                                                                                742 kB/s |  39 kB     00:00    
(8/10): perl-Params-Check-0.38-395.0.1.an8.noarch.rpm                                                                                       353 kB/s |  21 kB     00:00    
(9/10): perl-version-0.99.24-1.el8.x86_64.rpm                                                                                               1.2 MB/s |  66 kB     00:00    
(10/10): perl-Module-CoreList-5.20181130-1.0.1.an8.noarch.rpm                                                                               309 kB/s |  81 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
总计                                                                                                                                        813 kB/s | 469 kB     00:00     
运行事务检查
事务检查成功。
运行事务测试
事务测试成功。
运行事务
  准备中  :                                                                                                                                                             1/1 
  安装    : perl-version-6:0.99.24-1.el8.x86_64                                                                                                                        1/10 
  安装    : perl-Module-CoreList-1:5.20181130-1.0.1.an8.noarch                                                                                                         2/10 
  安装    : perl-Module-Metadata-1.000033-395.0.1.an8.noarch                                                                                                           3/10 
  安装    : perl-Module-Load-1:0.32-395.0.1.an8.noarch                                                                                                                 4/10 
  安装    : perl-Locale-Maketext-1.28-396.0.1.an8.noarch                                                                                                               5/10 
  安装    : perl-Locale-Maketext-Simple-1:0.21-421.0.1.an8.noarch                                                                                                      6/10 
  安装    : perl-Params-Check-1:0.38-395.0.1.an8.noarch                                                                                                                7/10 
  安装    : perl-Module-Load-Conditional-0.68-395.0.1.an8.noarch                                                                                                       8/10 
  安装    : perl-ExtUtils-MM-Utils-1:7.34-1.0.1.an8.noarch                                                                                                             9/10 
  安装    : perl-IPC-Cmd-2:1.02-1.0.1.an8.noarch                                                                                                                      10/10 
  运行脚本: perl-IPC-Cmd-2:1.02-1.0.1.an8.noarch                                                                                                                      10/10 
  验证    : perl-ExtUtils-MM-Utils-1:7.34-1.0.1.an8.noarch                                                                                                             1/10 
  验证    : perl-IPC-Cmd-2:1.02-1.0.1.an8.noarch                                                                                                                       2/10 
  验证    : perl-Locale-Maketext-1.28-396.0.1.an8.noarch                                                                                                               3/10 
  验证    : perl-Locale-Maketext-Simple-1:0.21-421.0.1.an8.noarch                                                                                                      4/10 
  验证    : perl-Module-CoreList-1:5.20181130-1.0.1.an8.noarch                                                                                                         5/10 
  验证    : perl-Module-Load-1:0.32-395.0.1.an8.noarch                                                                                                                 6/10 
  验证    : perl-Module-Load-Conditional-0.68-395.0.1.an8.noarch                                                                                                       7/10 
  验证    : perl-Module-Metadata-1.000033-395.0.1.an8.noarch                                                                                                           8/10 
  验证    : perl-Params-Check-1:0.38-395.0.1.an8.noarch                                                                                                                9/10 
  验证    : perl-version-6:0.99.24-1.el8.x86_64                                                                                                                       10/10 

已安装:
  perl-ExtUtils-MM-Utils-1:7.34-1.0.1.an8.noarch               perl-IPC-Cmd-2:1.02-1.0.1.an8.noarch                      perl-Locale-Maketext-1.28-396.0.1.an8.noarch       
  perl-Locale-Maketext-Simple-1:0.21-421.0.1.an8.noarch        perl-Module-CoreList-1:5.20181130-1.0.1.an8.noarch        perl-Module-Load-1:0.32-395.0.1.an8.noarch         
  perl-Module-Load-Conditional-0.68-395.0.1.an8.noarch         perl-Module-Metadata-1.000033-395.0.1.an8.noarch          perl-Params-Check-1:0.38-395.0.1.an8.noarch        
  perl-version-6:0.99.24-1.el8.x86_64                         

完毕！

#####报错 2
No C compiler found, please specify one with the environment variable CC,
[root@localhost openssl-3.1.6]# yum install  gcc
上次元数据过期检查：3:23:34 前，执行于 2024年06月14日 星期五 20时25分07秒。
依赖关系解决。
============================================================================================================================================================================
 软件包                                     架构                              版本                                               仓库                                  大小
============================================================================================================================================================================
安装:
 gcc                                        x86_64                            8.5.0-10.1.0.1.an8_6                               AppStream                             22 M
安装依赖关系:
 annobin                                    x86_64                            10.29-3.0.1.an8                                    AppStream                            116 k
 cpp                                        x86_64                            8.5.0-10.1.0.1.an8_6                               AppStream                             10 M
 glibc-devel                                x86_64                            2.28-189.5.0.1.an8_6                               BaseOS                                79 k
 glibc-headers                              x86_64                            2.28-189.5.0.1.an8_6                               BaseOS                               483 k
 isl                                        x86_64                            0.16.1-6.el8                                       AppStream                            834 k
 libasan                                    x86_64                            8.5.0-10.1.0.1.an8_6                               BaseOS                               411 k
 libatomic                                  x86_64                            8.5.0-10.1.0.1.an8_6                               BaseOS                                23 k
 libmpc                                     x86_64                            1.1.0-9.1.an8                                      AppStream                             60 k
 libubsan                                   x86_64                            8.5.0-10.1.0.1.an8_6                               BaseOS                               150 k
 libxcrypt-devel                            x86_64                            4.1.1-6.an8                                        BaseOS                                24 k

事务概要
============================================================================================================================================================================
安装  11 软件包

总下载：35 M
安装大小：91 M
确定吗？[y/N]： y
下载软件包：
(1/11): annobin-10.29-3.0.1.an8.x86_64.rpm                                                                                                  227 kB/s | 116 kB     00:00    
(2/11): isl-0.16.1-6.el8.x86_64.rpm                                                                                                         1.0 MB/s | 834 kB     00:00    
(3/11): libmpc-1.1.0-9.1.an8.x86_64.rpm                                                                                                     381 kB/s |  60 kB     00:00    
(4/11): glibc-devel-2.28-189.5.0.1.an8_6.x86_64.rpm                                                                                         992 kB/s |  79 kB     00:00    
(5/11): glibc-headers-2.28-189.5.0.1.an8_6.x86_64.rpm                                                                                       880 kB/s | 483 kB     00:00    
(6/11): libasan-8.5.0-10.1.0.1.an8_6.x86_64.rpm                                                                                             1.4 MB/s | 411 kB     00:00    
(7/11): libatomic-8.5.0-10.1.0.1.an8_6.x86_64.rpm                                                                                           225 kB/s |  23 kB     00:00    
(8/11): libubsan-8.5.0-10.1.0.1.an8_6.x86_64.rpm                                                                                            1.3 MB/s | 150 kB     00:00    
(9/11): libxcrypt-devel-4.1.1-6.an8.x86_64.rpm                                                                                              287 kB/s |  24 kB     00:00    
(10/11): gcc-8.5.0-10.1.0.1.an8_6.x86_64.rpm                                                                                                3.9 MB/s |  22 MB     00:05    
(11/11): cpp-8.5.0-10.1.0.1.an8_6.x86_64.rpm                                                                                                1.3 MB/s |  10 MB     00:08    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
总计                                                                                                                                        4.3 MB/s |  35 MB     00:08     
运行事务检查
事务检查成功。
运行事务测试
事务测试成功。
运行事务
  准备中  :                                                                                                                                                             1/1 
  运行脚本: glibc-headers-2.28-189.5.0.1.an8_6.x86_64                                                                                                                  1/11 
  安装    : glibc-headers-2.28-189.5.0.1.an8_6.x86_64                                                                                                                  1/11 
  安装    : libxcrypt-devel-4.1.1-6.an8.x86_64                                                                                                                         2/11 
  安装    : glibc-devel-2.28-189.5.0.1.an8_6.x86_64                                                                                                                    3/11 
  运行脚本: glibc-devel-2.28-189.5.0.1.an8_6.x86_64                                                                                                                    3/11 
  安装    : libmpc-1.1.0-9.1.an8.x86_64                                                                                                                                4/11 
  安装    : cpp-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                            5/11 
  运行脚本: cpp-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                            5/11 
  安装    : libubsan-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                       6/11 
  运行脚本: libubsan-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                       6/11 
  安装    : libatomic-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                      7/11 
  运行脚本: libatomic-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                      7/11 
  安装    : libasan-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                        8/11 
  运行脚本: libasan-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                        8/11 
  安装    : isl-0.16.1-6.el8.x86_64                                                                                                                                    9/11 
  运行脚本: isl-0.16.1-6.el8.x86_64                                                                                                                                    9/11 
  安装    : gcc-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                           10/11 
  运行脚本: gcc-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                           10/11 
  安装    : annobin-10.29-3.0.1.an8.x86_64                                                                                                                            11/11 
  运行脚本: annobin-10.29-3.0.1.an8.x86_64                                                                                                                            11/11 
  验证    : annobin-10.29-3.0.1.an8.x86_64                                                                                                                             1/11 
  验证    : cpp-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                            2/11 
  验证    : gcc-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                            3/11 
  验证    : isl-0.16.1-6.el8.x86_64                                                                                                                                    4/11 
  验证    : libmpc-1.1.0-9.1.an8.x86_64                                                                                                                                5/11 
  验证    : glibc-devel-2.28-189.5.0.1.an8_6.x86_64                                                                                                                    6/11 
  验证    : glibc-headers-2.28-189.5.0.1.an8_6.x86_64                                                                                                                  7/11 
  验证    : libasan-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                        8/11 
  验证    : libatomic-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                      9/11 
  验证    : libubsan-8.5.0-10.1.0.1.an8_6.x86_64                                                                                                                      10/11 
  验证    : libxcrypt-devel-4.1.1-6.an8.x86_64                                                                                                                        11/11 

已安装:
  annobin-10.29-3.0.1.an8.x86_64                cpp-8.5.0-10.1.0.1.an8_6.x86_64          gcc-8.5.0-10.1.0.1.an8_6.x86_64         glibc-devel-2.28-189.5.0.1.an8_6.x86_64    
  glibc-headers-2.28-189.5.0.1.an8_6.x86_64     isl-0.16.1-6.el8.x86_64                  libasan-8.5.0-10.1.0.1.an8_6.x86_64     libatomic-8.5.0-10.1.0.1.an8_6.x86_64      
  libmpc-1.1.0-9.1.an8.x86_64                   libubsan-8.5.0-10.1.0.1.an8_6.x86_64     libxcrypt-devel-4.1.1-6.an8.x86_64     

完毕！


#### 3 编译的过程信息
[root@localhost openssl-3.1.6]# ./config enable-fips --prefix=/usr/local --openssldir=/usr/local/openssl
Configuring OpenSSL version 3.1.6 for target linux-x86_64
Using os-specific seed configuration
Created configdata.pm
Running configdata.pm
Created Makefile.in
Created Makefile
Created include/openssl/configuration.h

**********************************************************************
***                                                                ***
***   OpenSSL has been successfully configured                     ***
***                                                                ***
***   If you encounter a problem while building, please open an    ***
***   issue on GitHub <https://github.com/openssl/openssl/issues>  ***
***   and include the output from the following command:           ***
***                                                                ***
***       perl configdata.pm --dump                                ***
***                                                                ***
***   (If you are new to OpenSSL, you might want to consult the    ***
***   'Troubleshooting' section in the INSTALL.md file first)      ***
***                                                                ***
**********************************************************************
[root@localhost openssl-3.1.6]# echo $?
0

##### 4 
yum install make 

##### 5  make install后报错
Can‘t locate Pod/Html.pm in @INC (you may need to install the Pod::Html module) (@INC contains: . /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at ./util/mkpod2html.pl line 14.
BEGIN failed--compilation aborted at ./util/mkpod2html.pl line 14.
make: *** [Makefile:29308：doc/html/man1/CA.pl.html] 错误 2

所以：
[root@localhost openssl-3.1.6]# yum install perl-Pod-Html
上次元数据过期检查：0:01:31 前，执行于 2024年06月15日 星期六 00时01分02秒。
依赖关系解决。
============================================================================================================================================================================
 软件包                                    架构                               版本                                              仓库                                   大小
============================================================================================================================================================================
安装:
 perl-Pod-Html                             noarch                             1.22.02-421.0.1.an8                               AppStream                              87 k

事务概要
============================================================================================================================================================================
安装  1 软件包

总下载：87 k
安装大小：36 k
确定吗？[y/N]： y
确定吗？[y/N]： y
下载软件包：
perl-Pod-Html-1.22.02-421.0.1.an8.noarch.rpm                                                                                                362 kB/s |  87 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
总计                                                                                                                                        361 kB/s |  87 kB     00:00     
运行事务检查
事务检查成功。
运行事务测试
事务测试成功。
运行事务
  准备中  :                                                                                                                                                             1/1 
  安装    : perl-Pod-Html-1.22.02-421.0.1.an8.noarch                                                                                                                    1/1 
  运行脚本: perl-Pod-Html-1.22.02-421.0.1.an8.noarch                                                                                                                    1/1 
  验证    : perl-Pod-Html-1.22.02-421.0.1.an8.noarch                                                                                                                    1/1 

已安装:
  perl-Pod-Html-1.22.02-421.0.1.an8.noarch                                                                                                                                  

完毕！

离线rpm包：

### yum install 过程中下载的rpm包
#离线的系统环境，上传下面的rpm包。 然后 rpm -Uvh *.rpm 进行安装即可。
[root@localhost var]# find ./ -type f -name "*.rpm"
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-ExtUtils-MM-Utils-7.34-1.0.1.an8.noarch.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-IPC-Cmd-1.02-1.0.1.an8.noarch.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-Locale-Maketext-1.28-396.0.1.an8.noarch.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-Locale-Maketext-Simple-0.21-421.0.1.an8.noarch.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-Module-CoreList-5.20181130-1.0.1.an8.noarch.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-Module-Load-0.32-395.0.1.an8.noarch.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-Module-Load-Conditional-0.68-395.0.1.an8.noarch.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-Module-Metadata-1.000033-395.0.1.an8.noarch.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-Params-Check-0.38-395.0.1.an8.noarch.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-version-0.99.24-1.el8.x86_64.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/annobin-10.29-3.0.1.an8.x86_64.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/cpp-8.5.0-10.1.0.1.an8_6.x86_64.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/gcc-8.5.0-10.1.0.1.an8_6.x86_64.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/isl-0.16.1-6.el8.x86_64.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/libmpc-1.1.0-9.1.an8.x86_64.rpm
./cache/dnf/AppStream-ba7193d610770ffb/packages/perl-Pod-Html-1.22.02-421.0.1.an8.noarch.rpm
./cache/dnf/BaseOS-d7f3346380282c5a/packages/glibc-devel-2.28-189.5.0.1.an8_6.x86_64.rpm
./cache/dnf/BaseOS-d7f3346380282c5a/packages/glibc-headers-2.28-189.5.0.1.an8_6.x86_64.rpm
./cache/dnf/BaseOS-d7f3346380282c5a/packages/libasan-8.5.0-10.1.0.1.an8_6.x86_64.rpm
./cache/dnf/BaseOS-d7f3346380282c5a/packages/libatomic-8.5.0-10.1.0.1.an8_6.x86_64.rpm
./cache/dnf/BaseOS-d7f3346380282c5a/packages/libubsan-8.5.0-10.1.0.1.an8_6.x86_64.rpm
./cache/dnf/BaseOS-d7f3346380282c5a/packages/libxcrypt-devel-4.1.1-6.an8.x86_64.rpm
./cache/dnf/BaseOS-d7f3346380282c5a/packages/make-4.2.1-11.an8.x86_64.rpm



##### 错误
error: Failed dependencies:
        perl(Time::HiRes) is needed by perl-IPC-Cmd-2:1.02-1.0.1.an8.noarch

yum install perl\(Time::HiRes\)
yum install yum-utils
yumdownloader --resolve perl-Time-HiRes


[root@GX-harbor-59 /home/nwom/opensslV3.1/openssl-yilaibao]$  rpm -Uvh perl-Time-HiRes-1.9758-2.0.1.an8.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:perl-Time-HiRes-4:1.9758-2.0.1.an################################# [100%]
   
   
##### 错误
you may need to install the Digest::SHA module 

[root@GX-harbor-59 /home/nwom/opensslV3.1/openssl-yilaibao]$  
[root@GX-harbor-59 /home/nwom/opensslV3.1/openssl-yilaibao]$  rpm -Uvh perl-Digest-SHA-6.02-1.el8.x86_64.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:perl-Digest-SHA-1:6.02-1.el8     ################################# [100%]

参考：

bclinux官网镜像源：
https://mirrors.cmecloud.cn/bclinux/el8.6/BaseOS/x86_64/os/Packages/

https://www.bilibili.com/video/BV1cM411R7r8    《升级到 openssl  v3.1》
视频中博主的操作截图：

在这里插入图片描述

四、bclinux8.6 升级k8s集群 v1.23.17 --> v1.24.0 --> v1.24.17

环境信息：

#系统信息
[root@slave-57 ~]# uname -a
Linux slave-57 4.19.0-372.26.3.el8.bclinux.x86_64 #1 SMP Mon Dec 5 11:05:59 CST 2022 x86_64 x86_64 x86_64 GNU/Linux

BigCloud Enterprise Linux 8.6 (Core)

#原k8s集群信息：
[root@xx]$  kubectl get nodes
NAME           STATUS   ROLES                  AGE    VERSION
master-52   Ready    control-plane,master   142d   v1.23.17
master-54   Ready    control-plane,master   142d   v1.23.17
master-56   Ready    control-plane,master   142d   v1.23.17
slave-55    Ready    <none>                 142d   v1.23.17
slave-57    Ready    <none>                 142d   v1.23.17

由于漏洞信息（Kubernetes 输入验证错误漏洞(CVE-2023-3676)（CVE-2023-3955））、Kubernetes 安全漏洞(CVE-2023-2431)
受影响的产品和版本：Kubelet 1.27.0至1.27.1版本，1.26.0至1.26.4版本，1.25.0至1.25.9版本，1.24.13及之前版本。

4.0 前置操作

#所有节点操作
# 0、 导入所需要的镜像包：   （这里并不需要升级etcd、coredns）
]$  docker load -i k8s.tar 
Loaded image: registry.aliyuncs.com/google_containers/etcd:3.5.3-0
Loaded image: registry.aliyuncs.com/google_containers/coredns:v1.8.6
Loaded image: registry.aliyuncs.com/google_containers/etcd:3.5.6-0

###主要使用到 下面的包：
Loaded image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.0
Loaded image: registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.0
Loaded image: registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.0
Loaded image: registry.aliyuncs.com/google_containers/kube-proxy:v1.24.0

Loaded image: registry.aliyuncs.com/google_containers/pause:3.7
Loaded image: registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.17
Loaded image: registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.17
Loaded image: registry.aliyuncs.com/google_containers/kube-proxy:v1.24.17
Loaded image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.17


###所需的rpm包：   cd /xx/packages

f46e0356e279308a525195d1ae939268faaea772a119cb752480be2b998bec54-kubelet-1.24.17-0.x86_64.rpm
2b0a91cdc51521c09c61f15da12d533a0e478e5d09f0326f079414b90fd77b03-kubeadm-1.24.17-0.x86_64.rpm
c3dc5ffa817d2c69bdd77494b5b9240568c4eb0d06b7b1bf3546bdab971741f5-kubectl-1.24.17-0.x86_64.rpm
cri-dockerd-0.3.14-3.el8.x86_64.rpm
363f3fbfa8b89bb978e2d089e52ba59847f143834f8ea1b559afa864d8c5c011-kubelet-1.24.0-0.x86_64.rpm
dda11ee75bc7fcb01e32512cefb8f686dc6a7383516b8b0828adb33761fe602e-kubeadm-1.24.0-0.x86_64.rpm
0c7a02e05273d05ea82ca13546853b65fbc257dd159565ce6eb658a0bdf31c9f-kubectl-1.24.0-0.x86_64.rpm




# 1 、安装cri-docker   根据自己环境下载对应的包 这里是el8的。 软件下载：https://github.com/Mirantis/cri-dockerd
rpm -ivh cri-dockerd-0.3.14-3.el8.x86_64.rpm


vim /usr/lib/systemd/system/cri-docker.service 
#修改fd://后面的
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7

systemctl daemon-reload
systemctl  enable --now cri-docker
systemctl  status cri-docker

ll /var/run/cri-dockerd.sock
srw-rw---- 1 root docker 0 Jan  9 17:51 /var/run/cri-dockerd.sock

4.1 升级master节点

升级v1.24.0的操作（之后v1.24.17的操作重复一遍即可。）

#修改nodes节点信息
kubectl edit nodes master-52  #升v1.24.17时不做 
修改为：    
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/cri-dockerd.sock


#操作 kubeadm
yum install -y  dda11ee75bc7fcb01e32512cefb8f686dc6a7383516b8b0828adb33761fe602e-kubeadm-1.24.0-0.x86_64.rpm
#kubeadm upgrade plan
kubeadm upgrade apply v1.24.0    ##之后其他的master/worker节点就用kubeadm upgrade node
#驱逐
kubectl drain master-52 --ignore-daemonsets

#升级kubectl 、kubelet
yum install -y 0c7a02e05273d05ea82ca13546853b65fbc257dd159565ce6eb658a0bdf31c9f-kubectl-1.24.0-0.x86_64.rpm
yum install -y 363f3fbfa8b89bb978e2d089e52ba59847f143834f8ea1b559afa864d8c5c011-kubelet-1.24.0-0.x86_64.rpm

#修改env文件
vim /var/lib/kubelet/kubeadm-flags.env   #升v1.24.17时不做
修改为：
KUBELET_KUBEADM_ARGS="--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7  --container-runtime=remote --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock"

systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet

解除节点保护
[root@master-52 ~]# kubectl uncordon gx-master-52

#重启刚才没重启的pod  （看存活时间的来重启）
 kubectl get pod -A -owide | grep master-52
 kubectl delete pod calico-node-bx6h4 kube-proxy-svj7x   -n kube-system

此时验证:
#此时
[root@master/package]$  kubectl get nodes
NAME           STATUS   ROLES           AGE    VERSION
master-52   Ready    control-plane   142d   v1.24.0
master-54   Ready    control-plane   142d   v1.23.17
master-56   Ready    control-plane   142d   v1.23.17
slave-55    Ready    <none>          142d   v1.23.17
slave-57    Ready    <none>          142d   v1.23.17

4.2 升级worker节点

yum install -y dda11ee75bc7fcb01e32512cefb8f686dc6a7383516b8b0828adb33761fe602e-kubeadm-1.24.0-0.x86_64.rpm
#执行：
kubeadm upgrade node
	]$   kubeadm upgrade node
	[upgrade] Reading configuration from the cluster...
	[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
	[preflight] Running pre-flight checks
	[preflight] Skipping prepull. Not a control plane node.
	[upgrade] Skipping phase. Not a control plane node.
	[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
	[upgrade] The configuration for this node was successfully updated!
	[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
	
 #回master执行
 kubectl drain slave-57 --ignore-daemonsets

 #回到worker节点
  yum install -y 0c7a02e05273d05ea82ca13546853b65fbc257dd159565ce6eb658a0bdf31c9f-kubectl-1.24.0-0.x86_64.rpm
 yum install -y 363f3fbfa8b89bb978e2d089e52ba59847f143834f8ea1b559afa864d8c5c011-kubelet-1.24.0-0.x86_64.rpm

 vim /var/lib/kubelet/kubeadm-flags.env  #升v1.24.17不做
 修改为：
KUBELET_KUBEADM_ARGS="--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7  --container-runtime=remote --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock"

# 修改节点套接字
[root@server2 ~]# kubectl edit nodes slave-57 #需要在master节点执行   #升v1.24.17不做
    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/cri-dockerd.sock

#重启kubelet
[root@slave-55 ~]# systemctl daemon-reload
[root@slave-55 ~]# systemctl restart kubelet
systemctl status kubelet


#解除节点保护
[root@server2 ~]# kubectl uncordon slave-57 #需要在master节点执行

###重启一些pod
kubectl get pod -owide -A | grep slave-57
kubectl delete pod calico-node-58hsq kube-proxy-wvghz   -n kube-system

参考博客：

https://blog.csdn.net/Gong_yz/article/details/129367570  （主要参考）
https://blog.csdn.net/cronaldo91/article/details/135194533

4.3 报错处理

在master54执行kubeadm upgrade node报错

#执行-v=5也是差不多结果。
[root@ master-54 /home/$  kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
unable to fetch the kubeadm-config ConfigMap: failed to getAPIEndpoint: could not retrieve API endpoints for node " master-54" using pod annotations: timed out waiting for the condition
To see the stack trace of this error execute with --v=5 or higher

##执行过程中   /var/log/message 信息
Jun 26 14:36:04  master-54 kubelet[2678165]: I0626 14:36:04.201461 2678165 scope.go:110] "RemoveContainer" containerID="1a2a8847aa4c3e425dc2d32e67708f913d7096ccf33b3dfaf02f3e7906078d5e"
Jun 26 14:36:04  master-54 kubelet[2678165]: E0626 14:36:04.201921 2678165 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver- master-54_kube-system(6119e1900881a0ca4a96a88618277ebe)\"" pod="kube-system/kube-apiserver- master-54" podUID=6119e1900881a0ca4a96a88618277ebe
Jun 26 14:36:16  master-54 kubelet[2678165]: I0626 14:36:16.201568 2678165 scope.go:110] "RemoveContainer" containerID="1a2a8847aa4c3e425dc2d32e67708f913d7096ccf33b3dfaf02f3e7906078d5e"
Jun 26 14:36:16  master-54 kubelet[2678165]: E0626 14:36:16.202098 2678165 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver- master-54_kube-system(6119e1900881a0ca4a96a88618277ebe)\"" pod="kube-system/kube-apiserver- master-54" podUID=6119e1900881a0ca4a96a88618277ebe

#这里发现他说kube-apiserver 出现CrashLoopBackOff 了。
[root@ master-54 /var/log]$  kubectl get pod -A -owide | grep api
kube-system   kube-apiserver                                    1/1     Running            11 (20h ago)     4d19h   1X.xx.xx.54      master-54   <none>           <none>
kube-system   kube-apiserver- master-52                       1/1     Running            1 (4h24m ago)    4h23m   1X.xx.xx.52      master-52   <none>           <none>
kube-system   kube-apiserver- master-54                       0/1     CrashLoopBackOff   249 (30s ago)    20h     1X.xx.xx.54      master-54   <none>           <none>
kube-system   kube-apiserver- master-56                       1/1     Running            2 (3h25m ago)    3h25m   1X.xx.xx.56      master-56   <none>           <none>


###处理方法 ，直接删了 kube-apiserver （谨慎操作！） （这里可能跟之前修复CVE2016时候 出现问题冲突报错了。）
[root@ master-54 /var/log]$  cd /etc/kubernetes/manifests/
[root@ master-54 /etc/kubernetes/manifests]$  ls
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

[root@ master-54 /etc/kubernetes/manifests]$  kubectl delete -n kube-system  pod kube-apiserver kube-apiserver- master-54 
pod "kube-apiserver" deleted
pod "kube-apiserver- master-54" deleted

The connection to the server 1x.xx.xx.58:6443 was refused - did you specify the right host or port?
### 等几分钟 ，自我修复。

4.x 执行过程（可略）

## 4.x.1  master节点执行 kubeadm upgrade node 的过程：
[root@ master-54 /home]$  kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade] Upgrading your Static Pod-hosted control plane instance to version "v1.24.0"...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/etcd] Non fatal issue encountered during upgrade: the desired etcd version "3.5.3-0" is older than the currently installed "3.5.6-0". Skipping etcd upgrade
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests4251516813"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-26-14-49-33/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-26-14-49-33/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-26-14-49-33/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade] The control plane instance for this node was successfully updated!
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

## 4.x.2 master节点执行kubeadm upgrade plan的过程  （这个指令在worker节点执行的话会报错）
xxxe]$  kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.24.0
[upgrade/versions] kubeadm version: v1.24.17
W0626 15:17:33.470007 1547246 version.go:104] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get "https://dl.k8s.io/release/stable.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0626 15:17:33.470093 1547246 version.go:105] falling back to the local client version: v1.24.17
[upgrade/versions] Target version: v1.24.17
W0626 15:17:43.483985 1547246 version.go:104] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.24.txt": Get "https://dl.k8s.io/release/stable-1.24.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0626 15:17:43.484014 1547246 version.go:105] falling back to the local client version: v1.24.17
[upgrade/versions] Latest version in the v1.24 series: v1.24.17

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     5 x v1.24.0   v1.24.17

Upgrade to the latest version in the v1.24 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.24.0   v1.24.17
kube-controller-manager   v1.24.0   v1.24.17
kube-scheduler            v1.24.0   v1.24.17
kube-proxy                v1.24.0   v1.24.17
CoreDNS                   v1.8.6    v1.8.6
etcd                      3.5.6-0   3.5.6-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.24.17

_____________________________________________________________________

The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

## 4.x.3 master节点执行kubeadm upgrade apply v1.24.17的过程  
[root@master-52 /home/]$  kubeadm upgrade apply v1.24.17
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.24.17"
[upgrade/versions] Cluster version: v1.24.0
[upgrade/versions] kubeadm version: v1.24.17
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.24.17" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-26-15-19-39/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 3 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests1516825297"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-26-15-19-39/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 3 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-26-15-19-39/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 3 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-06-26-15-19-39/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
[apiclient] Found 3 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Removing the deprecated label node-role.kubernetes.io/master='' from all control plane Nodes. After this step only the label node-role.kubernetes.io/control-plane='' will be present on control plane Nodes.
[upgrade/postupgrade] Adding the new taint &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} to all control plane Nodes. After this step both taints &Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,} and &Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,} should be present on control plane Nodes.
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.24.17". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

## 4.x.4 worker节点执行kubeadm upgrade node的过程  
	]$   kubeadm upgrade node
	[upgrade] Reading configuration from the cluster...
	[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
	[preflight] Running pre-flight checks
	[preflight] Skipping prepull. Not a control plane node.
	[upgrade] Skipping phase. Not a control plane node.
	[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
	[upgrade] The configuration for this node was successfully updated!
	[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

五、将另一个空闲节点加入k8s集群，再做升级

#master节点执行：
[root@master-54 /home/nwom/dockerImageTar]$  kubeadm token create --print-join-command
kubeadm join 1X.xx.xx.58:6443 --token 6h3mjx.xxx--discovery-token-ca-cert-hash sha256:f5c91cc3xxx


#worker节点执行：
[root@GX-slave-60 ~]$  kubeadm join 1X.xx.xx.58:6443 --token 6h3mjx.xxx --discovery-token-ca-cert-hash sha256:f5c91ccxxx
[preflight] Running pre-flight checks
        [WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature gate of the kubelet
        [WARNING FileExisting-tc]: tc not found in system path
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
        [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
[root@GX-slave-60 ~]$  echo $?
1

#报错处理 使用  kubeadm reset
[root@ slave-60 ~/k8s-bak]$  kubeadm reset
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0627 09:58:10.511418 3906685 removeetcdmember.go:80] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system s IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

###再次加入
[root@ slave-60 ~/k8s-bak]$  
 kubeadm join 1X.xx.xx.58:6443 --token 6h3mjx.xxx --discovery-token-ca-cert-hash sha256:f5c91ccxxx

[preflight] Running pre-flight checks
        [WARNING FileExisting-tc]: tc not found in system path
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@ slave-60 ~/k8s-bak]$  echo $?
0

#参考：
https://blog.csdn.net/qq_43591361/article/details/129598840 《K8S设置node加入集群》
https://www.cnblogs.com/wangzy-Zj/p/13130877.html 《Kubernetes 加入主节点报错》

六、 k8s处理污点，去除污点

原本master54是没有污点的。升级后自带原本污点了。

xxx-web-79c57d67d5-kn7nq           0/1     Pending   0               49m

kubectl describe -n xx pod xxx-web-79c57d67d5-kn7nq     
kubectl get pod -n xx-oyaml xxx-web-79c57d67d5-kn7nq  

##
#######
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  19m                default-scheduler  0/5 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) didn't match Pod's node affinity/selector, 2 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  14m                default-scheduler  0/5 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) didn't match Pod's node affinity/selector, 2 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  46m (x2 over 51m)  default-scheduler  0/5 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 1 node(s) were unschedulable, 2 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  26m (x4 over 41m)  default-scheduler  0/5 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  22m                default-scheduler  0/5 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.

###污点

NODE           TAINTS
master-52   [map[effect:NoSchedule key:node-role.kubernetes.io/master] map[effect:NoSchedule key:node-role.kubernetes.io/control-plane]]
master-54   [map[effect:NoSchedule key:node-role.kubernetes.io/control-plane]]
master-56   [map[effect:NoSchedule key:node-role.kubernetes.io/master] map[effect:NoSchedule key:node-role.kubernetes.io/control-plane]]
slave-55    <none>
slave-57    <none>


###处理方法：
kubectl taint node  master-54 node-role.kubernetes.io/control-plane:NoSchedule-