（六）Kubernetes - 手动部署(二进制方式安装)

Kubernetes - 手动部署 [ 5 ]

- 1 部署Nginx+Keepalived高可用负载均衡器
- - 1.1 安装软件包(Master1/Master2)
  - 1.2 Nginx配置文件(主备相同)
  - 1.3 keepalived配置文件(Master1)
  - 1.4 keepalived配置(master2)
  - 1.5 Nginx增加Steam模块
  - - 1.5.1 查看Nginx版本模块
    - 1.5.2 下载同一个版本的nginx
    - 1.5.3 备份原Nginx文件
    - 1.5.4 重新编译Nginx
    - 1.5.5 替换nginx到Master1/Master2
    - 1.5.6 修改nginx服务文件
  - 1.6 启动并设置开机自启(master1/master2)
  - 1.7 查看keepalived工作状态
  - 1.8 Nginx+keepalived高可用测试
  - 1.9 访问负载均衡器测试
  - 1.10 修改所有的Work Node连接LB VIP

1 部署Nginx+Keepalived高可用负载均衡器

可参考以下链接，了解nginx和keepalived，不看也不影响部署
CHAPTER 3 Web HA集群部署 - Keepalived
CHAPTER 1 Web Server - nginx 安装配置

在这里插入图片描述

Nginx是一个主流Web服务和反向代理服务器，这里用四层实现对apiserver实现负载均衡。
Keepalived是一个主流高可用软件，基于VIP绑定实现服务器双机热备，在上述拓扑中，Keepalived主要根据Nginx运行状态判断是否需要故障转移（漂移VIP），例如当Nginx主节点挂掉，VIP会自动绑定在Nginx备节点，从而保证VIP一直可用，实现Nginx高可用。
如果你是在公有云上，一般都不支持keepalived，那么你可以直接用它们的负载均衡器产品，直接负载均衡多台Master kube-apiserver，架构与上面一样。

在两台Master节点操作

1.1 安装软件包(Master1/Master2)

yum install epel-release -y
yum install nginx keepalived -y

1.2 Nginx配置文件(主备相同)

cat > /etc/nginx/nginx.conf << "EOF"
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {

    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
       server 192.168.71.251:6443;   # Master1 APISERVER IP:PORT
       server 192.168.71.254:6443;   # Master2 APISERVER IP:PORT
    }
    
    server {
       listen 16443; # 由于nginx与master节点复用，这个监听端口不能是6443，否则会冲突
       proxy_pass k8s-apiserver;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}
EOF

1.3 keepalived配置文件(Master1)

cat > /etc/keepalived/keepalived.conf << EOF
global_defs { 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_MASTER
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state MASTER 
    interface ens33  # 修改为实际网卡名
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 100    # 优先级，备服务器设置 90 
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    # 虚拟IP
    virtual_ipaddress { 
        192.168.71.250/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

vrrp_script：指定检查nginx工作状态脚本（根据nginx状态判断是否故障转移）
virtual_ipaddress：虚拟IP（VIP）

准备上述配置文件中检查Nginx运行状态的脚本

cat > /etc/keepalived/check_nginx.sh  << "EOF"
#!/bin/bash
count=$(ss -antp |grep 16443 |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    exit 1
else
    exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh

1.4 keepalived配置(master2)

cat > /etc/keepalived/keepalived.conf << EOF
global_defs { 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_BACKUP
} 

vrrp_script check_nginx {
    script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state BACKUP 
    interface ens33
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 90
    advert_int 1
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress { 
        192.168.71.250/24
    } 
    track_script {
        check_nginx
    } 
}
EOF

准备上述配置文件中检查nginx运行状态的脚本：

cat > /etc/keepalived/check_nginx.sh  << "EOF"
#!/bin/bash
count=$(ss -antp |grep 16443 |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    exit 1
else
    exit 0
fi
EOF

chmod +x /etc/keepalived/check_nginx.sh

说明：
keepalived根据脚本返回状态码（0为工作正常，非0不正常）判断是否故障转移。

1.5 Nginx增加Steam模块

1.5.1 查看Nginx版本模块

如果已经安装 --with-stream模块,后面的步骤可以跳过

[root@k8s-master2 nginx-1.20.1]# nginx -V
nginx version: nginx/1.20.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) 
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --with-stream

1.5.2 下载同一个版本的nginx

下载地址 : http://nginx.org/download/

1.5.3 备份原Nginx文件

mv /usr/sbin/nginx /usr/sbin/nginx.bak
cp -r /etc/nginx{,.bak}

1.5.4 重新编译Nginx

检查模块是否支持，比如这次添加 limit 限流模块 和 stream 模块：
./configure –help | grep limit

ps：-without-http_limit_conn_module disable 表示已有该模块，编译时，不需要添加

./configure –help | grep stream

ps：–with-stream enable 表示不支持，编译时要自己添加该模块

根据第1步查到已有的模块，加上本次需新增的模块: --with-stream

编译环境准备

yum -y install libxml2 libxml2-dev libxslt-devel 
yum -y install gd-devel 
yum -y install perl-devel perl-ExtUtils-Embed 
yum -y install GeoIP GeoIP-devel GeoIP-data
yum -y install pcre-devel
yum -y install openssl openssl-devel
yum -y install gcc make

编译

tar -xf nginx-1.20.1.tar.gz
cd nginx-1.20.1/
./configure --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf  --with-stream
make

说明:
make完成后不要继续输入“make install”，以免现在的nginx出现问题
以上完成后，会在objs目录下生成一个nginx文件，先验证:

[root@k8s-master2 nginx-1.20.1]# ./objs/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

1.5.5 替换nginx到Master1/Master2

cp ./objs/nginx /usr/sbin/ 
scp objs/nginx root@192.168.242.51:/usr/sbin/

1.5.6 修改nginx服务文件

vim /usr/lib/systemd/system/nginx.service
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/bin/rm -rf /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecStop=/usr/sbin/nginx -s stop
ExecReload=/usr/sbin/nginx -s reload
PrivateTmp=true
[Install]
WantedBy=multi-user.target

1.6 启动并设置开机自启(master1/master2)

systemctl daemon-reload
systemctl start nginx keepalived
systemctl enable nginx keepalived

nginx无法启动

[root@node-254 ~]# /usr/sbin/nginx -t
nginx: [emerg] unknown directive "stream" in /etc/nginx/nginx.conf:13
nginx: configuration file /etc/nginx/nginx.conf test failed

解决办法：

# 安装nginx源
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
# 先安装
yum -y install epel-release

#应该是缺少modules模块
yum -y install nginx-all-modules.noarch
然后在用nginx -t就好了
[root@k8s-node2 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

1.7 查看keepalived工作状态

[root@node-251 kubernetes]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:84:07:56 brd ff:ff:ff:ff:ff:ff
    inet 192.168.71.251/24 brd 192.168.71.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.71.250/24 scope global secondary ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::ca6d:22ce:c36:f87b/64 scope link tentative noprefixroute dadfailed
       valid_lft forever preferred_lft forever
    inet6 fe80::7fd7:bd03:f1c1:ee7c/64 scope link tentative noprefixroute dadfailed
       valid_lft forever preferred_lft forever
    inet6 fe80::a2f8:7cd5:7ddb:3558/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:08:fa:41:ae brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 172.16.52.192/32 scope global tunl0
       valid_lft forever preferred_lft forever
5: calid70e12abeda@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever

可以看到，在ens33网卡绑定了192.168.242.55 虚拟IP，说明工作正常。

1.8 Nginx+keepalived高可用测试

关闭主节点Nginx，测试VIP是否漂移到备节点服务器。
在Nginx Master执行 pkill nginx;
在Nginx Backup，ip addr命令查看已成功绑定VIP。

[root@node-254 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:c6:83:89 brd ff:ff:ff:ff:ff:ff
    inet 192.168.70.251/24 brd 192.168.70.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.71.250/24 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fec6:8389/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:b6:fb:9c:83 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: docker_gwbridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:dd:b0:70:37 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
5: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 172.16.212.128/32 scope global tunl0
       valid_lft forever preferred_lft forever

1.9 访问负载均衡器测试

找K8s集群中任意一个节点，使用curl查看K8s版本测试，使用VIP访问：

[root@node-251 kubernetes]# curl -k https://192.168.71.250:16443/version
{
  "major": "1",
  "minor": "20",
  "gitVersion": "v1.20.15",
  "gitCommit": "8f1e5bf0b9729a899b8df86249b56e2c74aebc55",
  "gitTreeState": "clean",
  "buildDate": "2022-01-19T17:23:01Z",
  "goVersion": "go1.15.15",
  "compiler": "gc",
  "platform": "linux/amd64"

可以正确获取到K8s版本信息，说明负载均衡器搭建正常。该请求数据流程：curl -> vip(nginx) -> apiserver
通过查看Nginx日志也可以看到转发apiserver IP：

[root@node-251 kubernetes]# tailf /var/log/nginx/k8s-access.log
192.168.71.251 192.168.71.251:6443 - [09/May/2023:05:18:17 -0400] 200 424
192.168.20.252 192.168.71.251:6443 - [09/May/2023:05:19:24 -0400] 200 541
192.168.71.251 192.168.71.251:6443 - [09/May/2023:05:22:07 -0400] 200 424

正常可以看到访问的IP地址在71.251和70.251切换，但是笔者的两个master不在同一网段，所以无法切换

1.10 修改所有的Work Node连接LB VIP

试想下，虽然我们增加了Master2 Node和负载均衡器，但是我们是从单Master架构扩容的，也就是说目前所有的Worker Node组件连接都还是Master1 Node，如果不改为连接VIP走负载均衡器，那么Master还是单点故障。

因此接下来就是要改所有Worker Node（kubectl get node命令查看到的节点）组件配置文件

由原来192.168.71.251修改为192.168.71.250（VIP）。

在所有Worker Node执行：

sed -i 's#192.168.71.251:6443#192.168.71.250:16443#' /opt/kubernetes/cfg/*
systemctl restart kubelet kube-proxy

检查节点状态

[root@node-251 kubernetes]# kubectl get nodes
NAME       STATUS   ROLES    AGE    VERSION
node-251   Ready    <none>   124m   v1.20.15
node-252   Ready    <none>   84m    v1.20.15
node-253   Ready    <none>   84m    v1.20.15
node-254   Ready    <none>   65m    v1.20.15

至此,一套高可用的k8s二进制可用集群就部署完成了~