华子目录
- `Kubernetes`部署说明
- 环境准备工作
- 主机准备
- `harbor`搭建
- `k8s`集群中的`主机名`和`ip`设定
- `k8s`集群中设置`hosts解析`
- `k8s`中的所有节点关闭`防火墙`和`selinux`
- `k8s`集群中`禁用swap分区`
- `k8s`集群中安装`docker-ce`
- `k8s`集群中`下载harbor证书`
- `k8s`集群中配置`harbor镜像加速器`
- `k8s`节点登录`harbor`测试
Kubernetes部署说明
K8S中文官网：https://kubernetes.io/zh-cn
| 主机名 | ip | 角色 |
|---|---|---|
harbor.huazi.org | 172.25.254.250 | harbor仓库 |
k8s-master.org | 172.25.254.100 | master，k8s集群控制节点 |
k8s-node1.org | 172.25.254.10 | worker，k8s集群工作节点 |
k8s-node2.org | 172.25.254.20 | worker，k8s集群工作节点 |
要求：除了harbor服务器的所有k8s节点
所有节点禁用selinux和防火墙所有节点同步时间和解析所有节点安装docker-ce所有节点禁用swap，注意注释掉/etc/fstab文件中的定义（因为k8s集群中容器对内存要求非常高，如果不禁用swap，一些容器就会存到swap中，这样可能会使你的容器暂停或者运行缓慢）
环境准备工作
主机准备
harbor搭建
harbor的搭建，我们可以参考这篇博客：- https://blog.csdn.net/huaz_md/article/details/142671140?spm=1001.2014.3001.5501
这里博主只写出几个关键的操作
- 制作
https的证书和key
[root@harbor ~]# openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/huazi.org.key -addext "subjectAltName = DNS:harbor.huazi.org" -x509 -days 365 -out certs/huazi.org.crt
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:shanxi
Locality Name (eg, city) [Default City]:xi'an
Organization Name (eg, company) [Default Company Ltd]:docker
Organizational Unit Name (eg, section) []:registry
Common Name (eg, your name or your server's hostname) []:harbor.huazi.org
Email Address []:admin@huazi.org
- 修改
harbor的配置文件
[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml
- 启动
harbor
[root@harbor harbor]# ./install.sh --with-chartmuseum
关闭防火墙和selinux
[root@harbor ~]# systemctl is-active firewalld.service
inactive
[root@harbor ~]# getenforce
Disabled
k8s集群中的主机名和ip设定
k8s-master
[root@k8s-master ~]# hostname -I
172.25.254.100
[root@k8s-master ~]# hostnamectl hostname k8s-master.org
[root@k8s-master ~]# hostname
k8s-master.org
k8s-node1
[root@k8s-node1 ~]# hostname -I
172.25.254.10
[root@k8s-node1 ~]# hostnamectl hostname k8s-node1.org
[root@k8s-node1 ~]# hostname
k8s-node1.org
k8s-node2
[root@k8s-node2 ~]# hostname -I
172.25.254.20
[root@k8s-node2 ~]# hostnamectl hostname k8s-node2.org
[root@k8s-node2 ~]# hostname
k8s-node2.org
k8s集群中设置hosts解析
k8s-master
[root@k8s-master ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.100 k8s-master.org
172.25.254.10 k8s-node1.org
172.25.254.20 k8s-node2.org
172.25.254.250 harbor.huazi.org
k8s-node1（这里我们使用scp命令）
[root@k8s-master ~]# scp /etc/hosts root@172.25.254.10:/etc/hosts
The authenticity of host '172.25.254.10 (172.25.254.10)' can't be established.
ED25519 key fingerprint is SHA256:oRI0QHrpuaAH8E6hepK2f2FymklDq9LifjGxkU86pMg.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '172.25.254.10' (ED25519) to the list of known hosts.
root@172.25.254.10's password:
hosts 100% 286 253.6KB/s 00:00
[root@k8s-node1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.100 k8s-master.org
172.25.254.10 k8s-node1.org
172.25.254.20 k8s-node2.org
172.25.254.250 harbor.huazi.org
k8s-node2（这里我们使用scp命令）
[root@k8s-master ~]# scp /etc/hosts root@172.25.254.20:/etc/hosts
The authenticity of host '172.25.254.20 (172.25.254.20)' can't be established.
ED25519 key fingerprint is SHA256:oRI0QHrpuaAH8E6hepK2f2FymklDq9LifjGxkU86pMg.
This host key is known by the following other names/addresses:
~/.ssh/known_hosts:1: 172.25.254.10
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '172.25.254.20' (ED25519) to the list of known hosts.
root@172.25.254.20's password:
hosts 100% 286 431.8KB/s 00:00
[root@k8s-node2 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.100 k8s-master.org
172.25.254.10 k8s-node1.org
172.25.254.20 k8s-node2.org
172.25.254.250 harbor.huazi.org
ping测试
[root@k8s-master ~]# ping harbor.huazi.org
PING harbor.huazi.org (172.25.254.250) 56(84) 比特的数据。
64 比特，来自 harbor.huazi.org (172.25.254.250): icmp_seq=1 ttl=64 时间=0.629 毫秒
64 比特，来自 harbor.huazi.org (172.25.254.250): icmp_seq=2 ttl=64 时间=0.288 毫秒
^C
--- harbor.huazi.org ping 统计 ---
已发送 2 个包， 已接收 2 个包, 0% packet loss, time 1061ms
rtt min/avg/max/mdev = 0.288/0.458/0.629/0.170 ms
[root@k8s-master ~]# ping k8s-node1.org
PING k8s-node1.org (172.25.254.10) 56(84) 比特的数据。
64 比特，来自 k8s-node1.org (172.25.254.10): icmp_seq=1 ttl=64 时间=0.422 毫秒
64 比特，来自 k8s-node1.org (172.25.254.10): icmp_seq=2 ttl=64 时间=0.339 毫秒
^C
--- k8s-node1.org ping 统计 ---
已发送 2 个包， 已接收 2 个包, 0% packet loss, time 1038ms
rtt min/avg/max/mdev = 0.339/0.380/0.422/0.041 ms
[root@k8s-master ~]# ping k8s-node2.org
PING k8s-node2.org (172.25.254.20) 56(84) 比特的数据。
64 比特，来自 k8s-node2.org (172.25.254.20): icmp_seq=1 ttl=64 时间=0.660 毫秒
64 比特，来自 k8s-node2.org (172.25.254.20): icmp_seq=2 ttl=64 时间=0.256 毫秒
64 比特，来自 k8s-node2.org (172.25.254.20): icmp_seq=3 ttl=64 时间=0.666 毫秒
^C
--- k8s-node2.org ping 统计 ---
已发送 3 个包， 已接收 3 个包, 0% packet loss, time 2078ms
rtt min/avg/max/mdev = 0.256/0.527/0.666/0.191 ms
k8s中的所有节点关闭防火墙和selinux
k8s-master
[root@k8s-master ~]# systemctl is-active firewalld.service
inactive
[root@k8s-master ~]# getenforce
Disabled
k8s-node1
[root@k8s-node1 ~]# systemctl is-active firewalld.service
inactive
[root@k8s-node1 ~]# getenforce
Disabled
k8s-node2
[root@k8s-node2 ~]# systemctl is-active firewalld.service
inactive
[root@k8s-node2 ~]# getenforce
Disabled
k8s集群中禁用swap分区
k8s-master
[root@k8s-master ~]# vim /etc/fstab
[root@k8s-master ~]# swapon -s
Filename Type Size Used Priority
/dev/nvme0n1p2 partition 2097148 0 -2
[root@k8s-master ~]# swapoff -a
[root@k8s-master ~]# swapon -s #发现swap分区已经没了
[root@k8s-master ~]#
k8s-node1
[root@k8s-node1 ~]# vim /etc/fstab
[root@k8s-node1 ~]# systemctl mask swap.target
Created symlink /etc/systemd/system/swap.target → /dev/null.
[root@k8s-node1 ~]# swapoff -a
[root@k8s-node1 ~]# swapon -s
k8s-node2
[root@k8s-node2 ~]# vim /etc/fstab
[root@k8s-node2 ~]# systemctl mask swap.target
Created symlink /etc/systemd/system/swap.target → /dev/null.
[root@k8s-node2 ~]# swapoff -a
[root@k8s-node2 ~]# swapon -s
k8s集群中安装docker-ce
[root@k8s-master yum.repos.d]# vim redhat.repo
[docker]
name=docker-ce
baseurl=https://mirrors.aliyun.com/docker-ce/linux/rhel/9/x86_64/stable/
gpgcheck=0
[root@k8s-master yum.repos.d]# scp /etc/yum.repos.d/redhat.repo root@172.25.254.10:/etc/yum.repos.d/
root@172.25.254.10's password:
redhat.repo 100% 108 130.7KB/s 00:00
[root@k8s-master yum.repos.d]# scp /etc/yum.repos.d/redhat.repo root@172.25.254.20:/etc/yum.repos.d/
root@172.25.254.20's password:
redhat.repo 100% 108 180.7KB/s 00:00
k8s-master
[root@k8s-master ~]# yum install docker-ce -y
[root@k8s-master ~]# systemctl enable --now docker
k8s-node1
[root@k8s-node1 ~]# yum install docker-ce -y
[root@k8s-node1 ~]# systemctl enable --now docker
k8s-node2
[root@k8s-node2 ~]# yum install docker-ce -y
[root@k8s-node2 ~]# systemctl enable --now docker
当我们在执行docker info后，出现了如下WARNING，如何处理呢
[root@k8s-master ~]# docker info
......
......
......
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
[root@k8s-master ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@k8s-master ~]# modprobe br_netfilter
[root@k8s-master ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
[root@k8s-master ~]# sysctl --system
[root@k8s-master ~]# systemctl restart docker
[root@k8s-node1 ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@k8s-node1 ~]# modprobe br_netfilter
[root@k8s-node1 ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
[root@k8s-node1 ~]# sysctl --system
[root@k8s-node1 ~]# systemctl restart docker
[root@k8s-node2 ~]# echo br_netfilter > /etc/modules-load.d/docker_mod.conf
[root@k8s-node2 ~]# modprobe br_netfilter
[root@k8s-node2 ~]# vim /etc/sysctl.d/docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
[root@k8s-node2 ~]# sysctl --system
[root@k8s-node2 ~]# systemctl restart docker
再次docker info时，就没有WARNING了
k8s集群中下载harbor证书
k8s-master
[root@k8s-master ~]# mkdir -p /etc/docker/certs.d/harbor.huazi.org/
[root@k8s-master ~]# scp root@harbor.huazi.org:/root/certs/huazi.org.crt /etc/docker/certs.d/harbor.huazi.org/ca.crt
The authenticity of host 'harbor.huazi.org (172.25.254.250)' can't be established.
ED25519 key fingerprint is SHA256:wkrDVNwOtwB4XhstKSlH+BEeO8JE3gp9NDIKRe6aMu0.
This host key is known by the following other names/addresses:
~/.ssh/known_hosts:5: 172.25.254.250
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'harbor.huazi.org' (ED25519) to the list of known hosts.
root@harbor.huazi.org's password:
huazi.org.crt 100% 2175 1.9MB/s 00:00
[root@k8s-master ~]# ll /etc/docker/certs.d/harbor.huazi.org/ca.crt
-rw-r--r-- 1 root root 2163 10月 2 05:19 /etc/docker/certs.d/harbor.huazi.org/ca.crt
[root@k8s-master ~]# systemctl restart docker
k8s-node1
[root@k8s-node1 ~]# mkdir -p /etc/docker/certs.d/harbor.huazi.org/
[root@k8s-node1 ~]# scp root@harbor.huazi.org:/root/certs/huazi.org.crt /etc/docker/certs.d/harbor.huazi.org/ca.crt
root@harbor.huazi.org's password:
huazi.org.crt 100% 2175 2.4MB/s 00:00
[root@k8s-node1 ~]# ll /etc/docker/certs.d/harbor.huazi.org/ca.crt
-rw-r--r-- 1 root root 2175 10月 2 05:55 /etc/docker/certs.d/harbor.huazi.org/ca.crt
[root@k8s-node1 ~]# systemctl restart docker
k8s-node2
[root@k8s-node2 ~]# mkdir -p /etc/docker/certs.d/harbor.huazi.org/
[root@k8s-node2 ~]# scp root@harbor.huazi.org:/root/certs/huazi.org.crt /etc/docker/certs.d/harbor.huazi.org/ca.crt
root@harbor.huazi.org's password:
huazi.org.crt 100% 2175 2.5MB/s 00:00
[root@k8s-node2 ~]# ll /etc/docker/certs.d/harbor.huazi.org/ca.crt
-rw-r--r-- 1 root root 2175 10月 2 05:57 /etc/docker/certs.d/harbor.huazi.org/ca.crt
[root@k8s-node2 ~]# systemctl restart docker
k8s集群中配置harbor镜像加速器
k8s-master
[root@k8s-master ~]# cd /etc/docker/
[root@k8s-master docker]# ls
certs.d
[root@k8s-master docker]# vim daemon.json
{
"registry-mirrors": ["https://harbor.huazi.org"]
}
[root@k8s-master docker]# systemctl restart docker
[root@k8s-master ~]# docker info
......
......
......
Registry Mirrors:
https://harbor.huazi.org/
k8s-node1
[root@k8s-node1 ~]# cd /etc/docker/
[root@k8s-node1 docker]# ls
certs.d
[root@k8s-node1 docker]# vim daemon.json
{
"registry-mirrors": ["https://harbor.huazi.org"]
}
[root@k8s-node1 docker]# cd
[root@k8s-node1 ~]# systemctl restart docker
[root@k8s-node1 ~]# docker info
......
......
......
Registry Mirrors:
https://harbor.huazi.org/
k8s-node2
[root@k8s-node2 ~]# cd /etc/docker/
[root@k8s-node2 docker]# ls
certs.d
[root@k8s-node2 docker]# vim daemon.json
{
"registry-mirrors": ["https://harbor.huazi.org"]
}
[root@k8s-node2 docker]# cd
[root@k8s-node2 ~]# systemctl restart docker
[root@k8s-node2 ~]# docker info
......
......
......
Registry Mirrors:
https://harbor.huazi.org/
k8s节点登录harbor测试
k8s-master
[root@k8s-master ~]# docker login harbor.huazi.org
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores
Login Succeeded
k8s-node1
[root@k8s-node1 ~]# docker login harbor.huazi.org
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores
Login Succeeded
k8s-node2
[root@k8s-node2 ~]# docker login harbor.huazi.org
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores
Login Succeeded
至此kubernetes环境搭建成功
