1. 简介

kubespray​ 是一个用于部署和管理 Kubernetes 集群的开源工具。它使用 Ansible 作为配置管理工具，可以根据用户需求灵活地配置和部署 Kubernetes 集群。它还支持各种功能，如高可用性、网络插件、存储插件、日志和监控等。

具有以下几个特点：

• 可以部署在 Ubuntu、CentOS、Red Hat、Google Cloud Platform、Amazon Web Services 和 Microsoft Azure。.
• 部署 High Available Kubernetes 集群.
• 可组合性 (Composable)，可自行选择 Network Plugin (flannel, calico, canal, weave) 来部署.
• 支持多种 Linux distributions(CoreOS, Debian Jessie, Ubuntu 16.04, CentOS/RHEL7).

本次部署说明：Kubespray v2.24.0 在线部署 kubernetes v1.28.6，软件版本支持如下

2. 预备条件

6 个节点：

角色	ip	cpu	mem	disk	ios	kernel
bastion01	192.168.23.30	4	8G	100G	Rocky 8.8	4.18+
kube-master01	192.168.23.31	4	8G	100G	Rocky 8.8	4.18+
kube-node01	192.168.23.32	16	32G	100G,200G	Rocky 8.8	4.18+
kube-node02	192.168.23.33	16	32G	100G,200G	Rocky 8.8	4.18+
kube-node03	192.168.23.34	16	32G	100G,200G	Rocky 8.8	4.18+
kube-node04	192.168.23.35	16	32G	100G,200G	Rocky 8.8	4.18+

注意：机器一定要检查：

• 是否有时间同步服务器：ntp or chrony
• dns
• yum

3. 基础配置

3.1 配置hostname

对应节点分别执行：

hostnamctl set-hostname bastion01
hostnamctl set-hostname kube-master01
hostnamctl set-hostname kube-node01
hostnamctl set-hostname kube-node02
hostnamctl set-hostname kube-node03
hostnamctl set-hostname kube-node04

3.2 配置互信

(bastion01操作)

ssh-keygen
for i in `cat inventory/sample/inventory.ini |grep host| awk '{print $2}' | awk -F '=' '{print $2}'`;do ssh-copy-id root@$i;done

4. 配置部署环境

(bastion01操作)

4.1 在线安装docker

sudo dnf check-update
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf  list docker-ce --showduplicates | sort -r
sudo dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl start docker && systemctl enable docker  && systemctl status docker
docker pull quay.io/kubespray/kubespray:v2.24.0

4.2 启动容器 kubespray

(bastion01操作)

docker run -it --net=host --mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa \
  quay.io/kubespray/kubespray:v2.24.0 bash

4.3 编写 inventory.ini

(bastion01操作)

$ vim inventory/sample/inventory.ini
[all]
kube-master01 ansible_host=192.168.23.31 ip=192.168.23.31 etcd_member_name=etcd1
kube-master02 ansible_host=192.168.23.32 ip=192.168.23.32 etcd_member_name=etcd2
kube-master03 ansible_host=192.168.23.33 ip=192.168.23.33 etcd_member_name=etcd3
kube-node01 ansible_host=192.168.23.34 ip=192.168.23.34
kube-node02 ansible_host=192.168.23.35 ip=192.168.23.35

[kube_control_plane]
kube-master01
kube-master02
kube-master03

[etcd]
kube-master01
kube-master02
kube-master03

[kube_node]
kube-node01
kube-node02

[calico_rr]

[k8s_cluster:children]
kube_control_plane
kube_node
calico_rr

查看默认的集群配置如下，你可以配置指定的 kubernetes 版本、网络插件、容器运行时等等，查看支持组件版本。

root@38261b1472a7:/kubespray# cat inventory/sample/group_vars/k8s_cluster/k8s-cluster.yml |grep -v "#" |grep -v "^$"
---
kube_config_dir: /etc/kubernetes
kube_script_dir: "{{ bin_dir }}/kubernetes-scripts"
kube_manifest_dir: "{{ kube_config_dir }}/manifests"
kube_cert_dir: "{{ kube_config_dir }}/ssl"
kube_token_dir: "{{ kube_config_dir }}/tokens"
kube_api_anonymous_auth: true
kube_version: v1.26.5
local_release_dir: "/tmp/releases"
retry_stagger: 5
kube_owner: kube
kube_cert_group: kube-cert
kube_log_level: 2
credentials_dir: "{{ inventory_dir }}/credentials"
kube_network_plugin: calico
kube_network_plugin_multus: false
kube_service_addresses: 10.233.0.0/18
kube_pods_subnet: 10.233.64.0/18
kube_network_node_prefix: 24
enable_dual_stack_networks: false
kube_service_addresses_ipv6: fd85:ee78:d8a6:8607::1000/116
kube_pods_subnet_ipv6: fd85:ee78:d8a6:8607::1:0000/112
kube_network_node_prefix_ipv6: 120
kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
kube_proxy_mode: ipvs
kube_proxy_strict_arp: false
kube_proxy_nodeport_addresses: >-
  {%- if kube_proxy_nodeport_addresses_cidr is defined -%}
  [{{ kube_proxy_nodeport_addresses_cidr }}]
  {%- else -%}
  []
  {%- endif -%}
kube_encrypt_secret_data: false
cluster_name: cluster.local
ndots: 2
dns_mode: coredns
enable_nodelocaldns: true
enable_nodelocaldns_secondary: false
nodelocaldns_ip: 169.254.25.10
nodelocaldns_health_port: 9254
nodelocaldns_second_health_port: 9256
nodelocaldns_bind_metrics_host_ip: false
nodelocaldns_secondary_skew_seconds: 5
enable_coredns_k8s_external: false
coredns_k8s_external_zone: k8s_external.local
enable_coredns_k8s_endpoint_pod_names: false
resolvconf_mode: host_resolvconf
deploy_netchecker: false
skydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"
skydns_server_secondary: "{{ kube_service_addresses|ipaddr('net')|ipaddr(4)|ipaddr('address') }}"
dns_domain: "{{ cluster_name }}"
container_manager: containerd
kata_containers_enabled: false
kubeadm_certificate_key: "{{ lookup('password', credentials_dir + '/kubeadm_certificate_key.creds length=64 chars=hexdigits') | lower }}"
k8s_image_pull_policy: IfNotPresent
kubernetes_audit: false
default_kubelet_config_dir: "{{ kube_config_dir }}/dynamic_kubelet_dir"
podsecuritypolicy_enabled: false
volume_cross_zone_attachment: false
persistent_volumes_enabled: false
event_ttl_duration: "1h0m0s"
auto_renew_certificates: false
kubeadm_patches:
  enabled: false
  source_dir: "{{ inventory_dir }}/patches"
  dest_dir: "{{ kube_config_dir }}/patches"

配置代理，否则 registry.k8s.io 镜像无法下载

$ vim  inventory/sample/group_vars/all/all.yml
...
http_proxy: "http://192.168.21.101:7890"
https_proxy: "http://192.168.21.101:7890"
no_proxy: "localhost,127.0.0.0/8,169.0.0.0/8,10.0.0.0/8,192.168.0.0/16,*.coding.net,*.tencentyun.com,*.myqcloud.com"
...

4.4 关闭防火墙、swap、selinux

(bastion01操作)

ansible -i inventory/sample/inventory.ini all -m ping
ansible -i inventory/sample/inventory.ini all -m systemd -a "name=firewalld state=stopped enabled=no"
ansible -i inventory/sample/inventory.ini all -m lineinfile -a "path=/etc/selinux/config regexp='^SELINUX=' line='SELINUX=disabled'" -b
ansible -i inventory/sample/inventory.ini all -m shell -a "getenforce 0"
ansible -i inventory/sample/inventory.ini all -m shell -a "sed -i '/.*swap.*/s/^/#/' /etc/fstab" -b
ansible -i inventory/sample/inventory.ini all -m shell -a " swapoff -a && sysctl -w vm.swappiness=0"

4.5 配置内核模块

ansible -i inventory/sample/inventory.ini all -m shell -a " modprobe bridge && modprobe br_netfilter && modprobe ip_conntrack"
ansible -i inventory/sample/inventory.ini all -m lineinfile -a "path=/etc/rc.local line='modprobe br_netfilter\nmodprobe ip_conntrack'"

5. 部署

ansible-playbook -i inventory/sample/inventory.ini cluster.yml
或者
ansible-playbook -i inventory/sample/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml
或者
ansible-playbook -i inventory/sample/inventory.ini --private-key /root/.ssh/id_rsa --become --become-user=root cluster.yml