k8s 的网络通信

1 k8s通信整体架构

2 flannel 网络插件

2.1 flannel 插件组成

2.2 flannel 插件的通信过程

2.3 flannel 支持的后端模式

3 calico 网络插件

3.1 calico 简介

3.2 calico 网络架构

3.3 部署 calico

1 k8s通信整体架构

k8s通过CNI接口接入其他插件来实现网络通讯。目前比较流行的插件有flannel，calico等
CNI插件存放位置：# cat /etc/cni/net.d/10-flannel.conflist
插件使用的解决方案如下
- 虚拟网桥，虚拟网卡，多个容器共用一个虚拟网卡进行通信。
- 多路复用：MacVLAN，多个容器共用一个物理网卡进行通信。
- 硬件交换：SR-LOV，一个物理网卡可以虚拟出多个接口，这个性能最好。
容器间通信：
- 同一个pod内的多个容器间的通信，通过lo即可实现pod之间的通信
- 同一节点的pod之间通过cni网桥转发数据包。
- 不同节点的pod之间的通信需要网络插件支持
pod和service通信: 通过iptables或ipvs实现通信，ipvs取代不了iptables，因为ipvs只能做负载均衡，而做不了nat转换
pod和外网通信：iptables的MASQUERADE
Service与集群外部客户端的通信；（ingress、nodeport、loadbalancer）

2 flannel 网络插件

2.1 flannel 插件组成

插件	功能
VXLAN	即Virtual Extensible LAN（虚拟可扩展局域网），是Linux本身支持的一网种网络虚拟化技术。VXLAN可以完全在内核态实现封装和解封装工作，从而通过“隧道”机制，构建出覆盖网络（Overlay Network）
VTEP	VXLAN Tunnel End Point（虚拟隧道端点），在Flannel中 VNI的默认值是1，这也是为什么宿主机的VTEP设备都叫flannel.1的原因
Cni0	网桥设备，每创建一个pod都会创建一对 veth pair。其中一端是pod中的eth0，另一端是Cni0网桥中的端口（网卡）
Flannel.1	TUN设备(虚拟网卡)，用来进行 vxlan 报文的处理（封包和解包）。不同node之间的pod数据流量都从overlay设备以隧道的形式发送到对端
Flanneld	flannel在每个主机中运行flanneld作为agent，它会为所在主机从集群的网络地址空间中，获取一个小的网段subnet，本主机内所有容器的IP地址都将从中分配。同时Flanneld监听K8s集群数据库，为flannel.1设备提供封装数据时必要的mac、ip等网络数据信息

2.2 flannel 插件的通信过程

当容器发送IP包，通过veth pair 发往cni网桥，再路由到本机的flannel.1设备进行处理。
VTEP设备之间通过二层数据帧进行通信，源VTEP设备收到原始IP包后，在上面加上一个目的MAC地址，封装成一个内部数据帧，发送给目的VTEP设备。
内部数据桢，并不能在宿主机的二层网络传输，Linux内核还需要把它进一步封装成为宿主机的一个普通的数据帧，承载着内部数据帧通过宿主机的eth0进行传输。
Linux会在内部数据帧前面，加上一个VXLAN头，VXLAN头里有一个重要的标志叫VNI，它是VTEP识别某个数据桢是不是应该归自己处理的重要标识。
flannel.1设备只知道另一端flannel.1设备的MAC地址，却不知道对应的宿主机地址是什么。在linux内核里面，网络设备进行转发的依据，来自FDB的转发数据库，这个flannel.1网桥对应的FDB信息，是由flanneld进程维护的。
linux内核在IP包前面再加上二层数据帧头，把目标节点的MAC地址填进去，MAC地址从宿主机的ARP表获取。
此时flannel.1设备就可以把这个数据帧从eth0发出去，再经过宿主机网络来到目标节点的eth0设备。目标主机内核网络栈会发现这个数据帧有VXLAN Header，并且VNI为1，Linux内核会对它进行拆包，拿到内部数据帧，根据VNI的值，交给本机flannel.1设备处理,flannel.1拆包，根据路由表发往cni网桥，最后到达目标容器。

# 默认网络通信路由

[root@k8s-master ~]# ip route 
default via 192.168.239.2 dev eth0 proto static metric 100 
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1 
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink     # flannel.1 onlink  即vxlan模式
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 
192.168.239.0/24 dev eth0 proto kernel scope link src 192.168.239.100 metric 100 

# 桥接转发数据库
[root@k8s-master ~]# bridge fdb
01:00:5e:00:00:01 dev eth0 self permanent
01:00:5e:00:00:fb dev eth0 self permanent
01:00:5e:00:00:01 dev virbr0 self permanent
01:00:5e:00:00:fb dev virbr0 self permanent
52:54:00:e9:79:9d dev virbr0-nic vlan 1 master virbr0 permanent
52:54:00:e9:79:9d dev virbr0-nic master virbr0 permanent
33:33:00:00:00:01 dev docker0 self permanent
01:00:5e:00:00:01 dev docker0 self permanent
01:00:5e:00:00:fb dev docker0 self permanent
02:42:c5:53:18:d6 dev docker0 vlan 1 master docker0 permanent
02:42:c5:53:18:d6 dev docker0 master docker0 permanent
33:33:00:00:00:01 dev dummy0 self permanent
33:33:00:00:00:01 dev kube-ipvs0 self permanent
aa:51:31:0d:75:9c dev flannel.1 dst 192.168.239.120 self permanent
ae:69:8d:48:73:fe dev flannel.1 dst 192.168.239.110 self permanent
33:33:00:00:00:01 dev cni0 self permanent
01:00:5e:00:00:01 dev cni0 self permanent
33:33:ff:3c:1a:c8 dev cni0 self permanent
01:00:5e:00:00:fb dev cni0 self permanent
32:70:25:3c:1a:c8 dev cni0 vlan 1 master cni0 permanent
32:70:25:3c:1a:c8 dev cni0 master cni0 permanent
d6:f4:e8:62:81:f4 dev veth981accb8 vlan 1 master cni0 permanent
e2:32:52:80:6b:7e dev veth981accb8 master cni0 
d6:f4:e8:62:81:f4 dev veth981accb8 master cni0 permanent
33:33:00:00:00:01 dev veth981accb8 self permanent
01:00:5e:00:00:01 dev veth981accb8 self permanent
33:33:ff:62:81:f4 dev veth981accb8 self permanent
2e:6d:1d:95:6d:60 dev veth77413d63 master cni0 permanent
2e:6d:1d:95:6d:60 dev veth77413d63 vlan 1 master cni0 permanent
be:f6:e4:24:ba:a1 dev veth77413d63 master cni0 
33:33:00:00:00:01 dev veth77413d63 self permanent
01:00:5e:00:00:01 dev veth77413d63 self permanent
33:33:ff:95:6d:60 dev veth77413d63 self permanent

# arp列表
[root@k8s-master ~]# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.239.1            ether   00:50:56:c0:00:08   C                     eth0
10.244.1.0               ether   ae:69:8d:48:73:fe   CM                    flannel.1
10.244.0.10              ether   e2:32:52:80:6b:7e   C                     cni0
10.244.0.11              ether   be:f6:e4:24:ba:a1   C                     cni0
192.168.239.110          ether   00:0c:29:e8:3a:34   C                     eth0
192.168.239.2            ether   00:50:56:e7:94:f6   C                     eth0
192.168.239.120          ether   00:0c:29:58:a0:fd   C                     eth0
10.244.2.0               ether   aa:51:31:0d:75:9c   CM                    flannel.1

2.3 flannel 支持的后端模式

网络模式	功能
vxlan	报文封装，默认模式
Directrouting	直接路由，跨网段使用vxlan，同网段使用host-gw模式
host-gw	主机网关，性能好，但只能在二层网络中，不支持跨网络如果有成千上万的Pod，容易产生广播风暴，不推荐
UDP	性能差，不推荐

[root@k8s-master ~]# kubectl -n kube-flannel edit configmaps kube-flannel-cfg 
apiVersion: v1
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "EnableNFTables": false,
      "Backend": {
        "Type": "host-gw"			# 默认为 vxlan 这里更改为 host-gw
      }
    }

#重启pod
[root@k8s-master ~]# kubectl -n kube-flannel delete pod --all
pod "kube-flannel-ds-6k567" deleted
pod "kube-flannel-ds-x7x7l" deleted
pod "kube-flannel-ds-xxp9b" deleted

[root@k8s-master ~]# ip route 
default via 192.168.239.2 dev eth0 proto static metric 100 
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1 
10.244.1.0/24 via 192.168.239.110 dev eth0 
10.244.2.0/24 via 192.168.239.120 dev eth0 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 
192.168.239.0/24 dev eth0 proto kernel scope link src 192.168.239.100 metric 100

3 calico 网络插件

3.1 calico 简介

纯三层的转发，中间没有任何的NAT和overlay，转发效率最好。
Calico 仅依赖三层路由可达。Calico 较少的依赖性使它能适配所有 VM、Container、白盒或者混合环境场景。

官网：Install Calico networking and network policy for on-premises deployments | Calico DocumentationInstall Calico networking and network policy for on-premises deployments.https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises

3.2 calico 网络架构

Felix：监听ECTD中心的存储获取事件，用户创建pod后，Felix负责将其网卡、IP、MAC都设置好，然后在内核的路由表里面写一条，注明这个IP应该到这张网卡。同样如果用户制定了隔离策略，Felix同样会将该策略创建到ACL中，以实现隔离。
BIRD：一个标准的路由程序，它会从内核里面获取哪一些IP的路由发生了变化，然后通过标准BGP的路由协议扩散到整个其他的宿主机上，让外界都知道这个IP在这里，路由的时候到这里

3.3 部署 calico

删除原有的网络插件 flannel

[root@k8s-master cri-plugin]# ls 
kube-flannel.yml

# 通过清单文件 删除 flannel 资源类型
[root@k8s-master cri-plugin]# kubectl delete -f kube-flannel.yml 
namespace "kube-flannel" deleted
serviceaccount "flannel" deleted
clusterrole.rbac.authorization.k8s.io "flannel" deleted
clusterrolebinding.rbac.authorization.k8s.io "flannel" deleted
configmap "kube-flannel-cfg" deleted
daemonset.apps "kube-flannel-ds" deleted

# 删除 Flannel 网络配置文件 防止 不同插件其冲突
[root@k8s-master cri-plugin]# rm -rf /etc/cni/net.d/10-flannel.conflist

官网地址下载 calico 清单文件

[root@k8s-master calico]# curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.2/manifests/calico-typha.yaml -o calico.yaml

将镜像使用 docker 拉取下来

[root@harbor ~]# docker pull docker.io/calico/typha:v3.28.2
[root@harbor ~]# docker pull docker.io/calico/node:v3.28.2
[root@harbor ~]# docker pull docker.io/calico/kube-controllers:v3.28.2
[root@harbor ~]# docker pull docker.io/calico/cni:v3.28.2

修改calico清单文件的配置

[root@k8s-master calico]# vim calico.yaml
4835           image: calico/cni:v3.28.2
4835           image: calico/cni:v3.28.2
4906           image: calico/node:v3.28.2
4932           image: calico/node:v3.28.2
5160           image: calico/kube-controllers:v3.28.2
5249         - image: calico/typha:v3.28.2

4973             - name: CALICO_IPV4POOL_VXLAN
4974               value: "Never"

4999             - name: CALICO_IPV4POOL_CIDR
5000               value: "10.244.0.0/16"
5001             - name: CALICO_AUTODETECTION_METHOD
5002               value: "interface=eth0"

声明清单文件

[root@k8s-master calico]# kubectl apply -f calico.yaml 
poddisruptionbudget.policy/calico-kube-controllers created
poddisruptionbudget.policy/calico-typha created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
serviceaccount/calico-cni-plugin created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrole.rbac.authorization.k8s.io/calico-cni-plugin created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-cni-plugin created
service/calico-typha created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created
deployment.apps/calico-typha created

是否加载了calico插件

[root@k8s-master calico]# kubectl -n kube-system get pods 
NAME                                       READY   STATUS    RESTARTS      AGE
calico-kube-controllers-56cd9cdfcd-mq9wq   1/1     Running   0             32s
calico-node-bgd7s                          1/1     Running   0             32s
calico-node-l957p                          1/1     Running   0             32s
calico-node-ts4mq                          1/1     Running   0             32s
calico-typha-7c66d894d-8t6rv               1/1     Running   0             32s
coredns-66d4c695bb-f7bvq                   1/1     Running   4 (27h ago)   10d
coredns-66d4c695bb-llnfv                   1/1     Running   4 (27h ago)   10d
etcd-k8s-master                            1/1     Running   4 (27h ago)   10d
kube-apiserver-k8s-master                  1/1     Running   4 (27h ago)   10d
kube-controller-manager-k8s-master         1/1     Running   5 (27h ago)   10d
kube-proxy-pgzvt                           1/1     Running   2 (27h ago)   7d6h
kube-proxy-qv44r                           1/1     Running   2 (27h ago)   7d6h
kube-proxy-zwt7c                           1/1     Running   2 (27h ago)   7d6h
kube-scheduler-k8s-master                  1/1     Running   5 (27h ago)   10d


[root@k8s-master calico]# kubectl get nodes 
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   10d   v1.30.0
k8s-node1    Ready    <none>          10d   v1.30.0
k8s-node2    Ready    <none>          10d   v1.30.0


[root@k8s-master calico]# kubectl run web --image nginx:latest --port 80 
pod/web created

[root@k8s-master calico]# curl 10.244.169.129
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>

这个时候需要重启重新加载网卡集群

不重启看不到网卡效果

[root@k8s-master ~]# reboot 
[root@k8s-node1 ~]# reboot 
[root@k8s-node2 ~]# reboot

查看是否重新加载

# master
[root@k8s-master ~]# ifconfig | grep -A 6 cali
calibb4b7ff266b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
--
calif66e318b6af: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# node1
[root@k8s-node1 ~]# ifconfig | grep -A 6 cali
cali38611089018: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
--
cali61bb03b2f20: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
--
cali96923616f01: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
--
cali9f2541b8e98: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# node2
[root@k8s-node2 ~]# ifconfig | grep -A 6 cali
cali290a9f54e70: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
--
cali4a60cab99b6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
--
cali68caf03a5f4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



# 最后一行倒数第三行
[root@k8s-master ~]# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.239.110          ether   00:0c:29:e8:3a:34   C                     eth0
192.168.239.120          ether   00:0c:29:58:a0:fd   C                     eth0
192.168.239.2            ether   00:50:56:e7:94:f6   C                     eth0
10.244.235.193           ether   ee:ba:6a:62:1e:19   C                     calibb4b7ff266b
192.168.239.1            ether   00:50:56:c0:00:08   C                     eth0
10.244.235.194           ether   aa:49:13:e7:00:a5   C                     calif66e318b6af

# 第二第三行使用bird路由程序，这个是获取路由变化的，并且是calico独有的
[root@k8s-master ~]# ip route 
default via 192.168.239.2 dev eth0 proto static metric 100 
10.244.36.64/26 via 192.168.239.110 dev tunl0 proto bird onlink 
10.244.169.128/26 via 192.168.239.120 dev tunl0 proto bird onlink 
blackhole 10.244.235.192/26 proto bird 
10.244.235.193 dev calibb4b7ff266b scope link 
10.244.235.194 dev calif66e318b6af scope link 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 
192.168.239.0/24 dev eth0 proto kernel scope link src 192.168.239.100 metric 100