环境
两台机器两网卡同网段接入同一个二层交换机。
机器A ens38 00:0c:29:a4:8b:fb 10.0.0.11/24 ens39 00:0c:29:a4:8b:05 10.0.0.12/24
机器B ens38 00:0c:29:4f:a6:c4 10.0.0.21/24 ens39 00:0c:29:4f:a6:ce 10.0.0.22/24
初始ARP表
只有管理口接口的ARP表项，10.0.0.0/24网段没有
机器A
root@ubuntu22:~# arp -n
Address HWtype HWaddress Flags Mask Iface
172.0.1.13 ether 00:0c:29:c1:16:a8 C ens33
172.0.1.131 ether b4:56:b9:f0:02:a5 C ens33
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
机器B
root@u22s:~# arp -n
Address HWtype HWaddress Flags Mask Iface
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
172.0.1.12 ether 00:0c:29:a4:8b:f1 C ens33
A ens38 10.0.0.11 ----> B 10.0.0.21
root@ubuntu22:~# ping -I ens38 10.0.0.21 -c 5 -w 5
PING 10.0.0.21 (10.0.0.21) from 10.0.0.11 ens38: 56(84) bytes of data.
64 bytes from 10.0.0.21: icmp_seq=1 ttl=64 time=0.447 ms
64 bytes from 10.0.0.21: icmp_seq=2 ttl=64 time=0.236 ms
64 bytes from 10.0.0.21: icmp_seq=3 ttl=64 time=0.223 ms
64 bytes from 10.0.0.21: icmp_seq=4 ttl=64 time=0.231 ms
64 bytes from 10.0.0.21: icmp_seq=5 ttl=64 time=0.377 ms
--- 10.0.0.21 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4097ms
rtt min/avg/max/mdev = 0.223/0.302/0.447/0.091 ms
抓包
arp表项
root@ubuntu22:~# arp -n
Address HWtype HWaddress Flags Mask Iface
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
10.0.0.22 ether 00:0c:29:4f:a6:ce C ens38
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
172.0.1.131 ether b4:56:b9:f0:02:a5 C ens33
10.0.0.21 ether 00:0c:29:4f:a6:ce C ens38
172.0.1.13 ether 00:0c:29:c1:16:a8 C ens33
172.0.1.25 ether 00:0c:29:4f:a6:ba C ens33
数据走向图
A ens38 10.0.0.11 —> B 10.0.0.22
root@ubuntu22:~# ping -I ens38 10.0.0.22 -c 5 -w 5
PING 10.0.0.22 (10.0.0.22) from 10.0.0.11 ens38: 56(84) bytes of data.
64 bytes from 10.0.0.22: icmp_seq=1 ttl=64 time=0.479 ms
64 bytes from 10.0.0.22: icmp_seq=2 ttl=64 time=0.317 ms
64 bytes from 10.0.0.22: icmp_seq=3 ttl=64 time=0.208 ms
64 bytes from 10.0.0.22: icmp_seq=4 ttl=64 time=0.976 ms
64 bytes from 10.0.0.22: icmp_seq=5 ttl=64 time=0.294 ms
--- 10.0.0.22 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4065ms
rtt min/avg/max/mdev = 0.208/0.454/0.976/0.274 ms
抓包
arp表项
root@ubuntu22:~# arp -n
Address HWtype HWaddress Flags Mask Iface
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
10.0.0.22 ether 00:0c:29:4f:a6:ce C ens38
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
172.0.1.131 ether b4:56:b9:f0:02:a5 C ens33
10.0.0.21 ether 00:0c:29:4f:a6:ce C ens38
172.0.1.13 ether 00:0c:29:c1:16:a8 C ens33
172.0.1.25 ether 00:0c:29:4f:a6:ba C ens33
机器B的ARP
root@u22s:~# arp -n
Address HWtype HWaddress Flags Mask Iface
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
10.0.0.11 ether 00:0c:29:a4:8b:fb C ens39
172.0.1.12 ether 00:0c:29:a4:8b:f1 C ens33
10.0.0.11 ether 00:0c:29:a4:8b:fb C ens38
数据走向图
A ens39 10.0.0.12 —> B 10.0.0.21
root@ubuntu22:~# ping -I ens39 10.0.0.21 -c 5 -w 5
PING 10.0.0.21 (10.0.0.21) from 10.0.0.12 ens39: 56(84) bytes of data.
64 bytes from 10.0.0.21: icmp_seq=1 ttl=64 time=0.415 ms
64 bytes from 10.0.0.21: icmp_seq=2 ttl=64 time=0.288 ms
64 bytes from 10.0.0.21: icmp_seq=3 ttl=64 time=0.202 ms
64 bytes from 10.0.0.21: icmp_seq=4 ttl=64 time=0.205 ms
64 bytes from 10.0.0.21: icmp_seq=5 ttl=64 time=0.195 ms
--- 10.0.0.21 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4079ms
rtt min/avg/max/mdev = 0.195/0.261/0.415/0.084 ms
抓包
ARP表
root@ubuntu22:~# arp -n
Address HWtype HWaddress Flags Mask Iface
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
10.0.0.22 ether 00:0c:29:4f:a6:ce C ens38
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
172.0.1.131 ether b4:56:b9:f0:02:a5 C ens33
10.0.0.21 ether 00:0c:29:4f:a6:ce C ens39
10.0.0.21 ether 00:0c:29:4f:a6:ce C ens38
172.0.1.13 ether 00:0c:29:c1:16:a8 C ens33
172.0.1.25 ether 00:0c:29:4f:a6:ba C ens33
10.0.0.22 ether 00:0c:29:4f:a6:ce C ens39
数据走向图
A ens39 10.0.0.12 —> 10.0.0.22
root@ubuntu22:~# ping -I ens39 10.0.0.22 -c 5 -w 5
PING 10.0.0.22 (10.0.0.22) from 10.0.0.12 ens39: 56(84) bytes of data.
64 bytes from 10.0.0.22: icmp_seq=1 ttl=64 time=0.224 ms
64 bytes from 10.0.0.22: icmp_seq=2 ttl=64 time=0.233 ms
64 bytes from 10.0.0.22: icmp_seq=3 ttl=64 time=0.330 ms
64 bytes from 10.0.0.22: icmp_seq=4 ttl=64 time=0.279 ms
64 bytes from 10.0.0.22: icmp_seq=5 ttl=64 time=0.230 ms
--- 10.0.0.22 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4100ms
rtt min/avg/max/mdev = 0.224/0.259/0.330/0.040 ms
抓包
ARP表项
root@ubuntu22:~# arp -n
Address HWtype HWaddress Flags Mask Iface
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
10.0.0.22 ether 00:0c:29:4f:a6:ce C ens38
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
172.0.1.131 ether b4:56:b9:f0:02:a5 C ens33
10.0.0.21 ether 00:0c:29:4f:a6:ce C ens39
10.0.0.21 ether 00:0c:29:4f:a6:ce C ens38
172.0.1.13 ether 00:0c:29:c1:16:a8 C ens33
172.0.1.25 ether 00:0c:29:4f:a6:ba C ens33
10.0.0.22 ether 00:0c:29:4f:a6:ce C ens39
机器B ARP表项
root@u22s:~# arp -n
Address HWtype HWaddress Flags Mask Iface
10.0.0.11 ether 00:0c:29:a4:8b:fb C ens38
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
10.0.0.12 ether 00:0c:29:a4:8b:05 C ens39
10.0.0.11 ether 00:0c:29:a4:8b:fb C ens39
172.0.1.12 ether 00:0c:29:a4:8b:f1 C ens33
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
10.0.0.12 ether 00:0c:29:a4:8b:05 C ens38
数据走向图
机器B ping 机器A
root@u22s:~# ping -I ens39 10.0.0.12 -c 3
PING 10.0.0.12 (10.0.0.12) from 10.0.0.22 ens39: 56(84) bytes of data.
64 bytes from 10.0.0.12: icmp_seq=1 ttl=64 time=0.199 ms
64 bytes from 10.0.0.12: icmp_seq=2 ttl=64 time=0.233 ms
64 bytes from 10.0.0.12: icmp_seq=3 ttl=64 time=0.244 ms
--- 10.0.0.12 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2052ms
rtt min/avg/max/mdev = 0.199/0.225/0.244/0.019 ms
root@u22s:~# ping -I ens39 10.0.0.11 -c 3
PING 10.0.0.11 (10.0.0.11) from 10.0.0.22 ens39: 56(84) bytes of data.
64 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=0.192 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=64 time=0.228 ms
64 bytes from 10.0.0.11: icmp_seq=3 ttl=64 time=0.292 ms
--- 10.0.0.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2042ms
rtt min/avg/max/mdev = 0.192/0.237/0.292/0.041 ms
root@u22s:~# ^C
root@u22s:~#
root@u22s:~# ping -I ens38 10.0.0.12 -c 3
PING 10.0.0.12 (10.0.0.12) from 10.0.0.21 ens38: 56(84) bytes of data.
64 bytes from 10.0.0.12: icmp_seq=1 ttl=64 time=0.192 ms
64 bytes from 10.0.0.12: icmp_seq=2 ttl=64 time=0.201 ms
64 bytes from 10.0.0.12: icmp_seq=3 ttl=64 time=0.930 ms
--- 10.0.0.12 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2020ms
rtt min/avg/max/mdev = 0.192/0.441/0.930/0.345 ms
root@u22s:~# ping -I ens38 10.0.0.11 -c 3
PING 10.0.0.11 (10.0.0.11) from 10.0.0.21 ens38: 56(84) bytes of data.
64 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=0.195 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=64 time=0.632 ms
64 bytes from 10.0.0.11: icmp_seq=3 ttl=64 time=0.251 ms
en39 10.0.0.22 --> 10.0.0.12
ens39 10.0.0.22 --> 10.0.0.11
ens38 10.0.0.21 --> 10.0.0.12
ens38 10.0.0.21 --> 10.0.0.11
数据走向图
arp表
root@u22s:~# arp -n
Address HWtype HWaddress Flags Mask Iface
10.0.0.11 ether 00:0c:29:a4:8b:fb C ens38
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
10.0.0.12 ether 00:0c:29:a4:8b:05 C ens39
10.0.0.11 ether 00:0c:29:a4:8b:fb C ens39
172.0.1.12 ether 00:0c:29:a4:8b:f1 C ens33
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
10.0.0.12 ether 00:0c:29:a4:8b:05 C ens38
现象分析
Node A ping Node B时， 不论目的IP是B.ens38还是B.ens39所有包都从 ens39 进入Node B
从ARP报文来看，10.0.0.21的MAC响应ens38和ens39都有，但是ens39的MAC响应是先收到的，ARP表中 10.0.0.21,10.0.0.22的mac地址都是 ens39 00:0c:29:4f:a6:ce。
Node B ping Node A时，数据包从对应IP的接口进入， ARP表中IP与MAC也是正确对应的，抓包到的ARP报文响应也是对应接口的MAC
小结
双机多网口二层连接，配置同网段地址，那么IP-MAC映射会存在不确定性，会受到ARP响应的先后顺序影响。如果多机多网口这样连接，恐怕不能通过指定源目的IP来实现指定接口到指定接口的通信。
而主机对ARP请求的响应，可能是都从一个接口处，也可能各个接口都出，这种行为的区别暂不清楚受到哪些内核配置的影响。
解决方法
设置规则
# 关闭ARP代答
sudo sysctl -w net.ipv4.conf.all.proxy_arp=0
sudo sysctl -w net.ipv4.conf.default.proxy_arp=0
# 设置ARP过滤规则
sudo sysctl -w net.ipv4.conf.all.arp_ignore=1
sudo sysctl -w net.ipv4.conf.all.arp_announce=2
sudo sysctl -w net.ipv4.conf.default.arp_ignore=1
sudo sysctl -w net.ipv4.conf.default.arp_announce=2
# For each specific interface
sudo sysctl -w net.ipv4.conf.ens38.arp_ignore=1
sudo sysctl -w net.ipv4.conf.ens38.arp_announce=2
sudo sysctl -w net.ipv4.conf.ens39.arp_ignore=1
sudo sysctl -w net.ipv4.conf.ens39.arp_announce=2
-
arp_ignore- 值为0：对所有请求进行响应。
- 值为1：只响应目标IP地址是接口上本地地址的请求，不响应非本地地址的请求。
-
arp_announce- 值为0：默认行为，使用最合适的本地地址进行ARP应答。
- 值为1：仅使用目标网络的最佳地址进行ARP应答。
- 值为2：仅使用发送接口上的地址进行ARP应答。
抓包
没有多余ARP响应
ARP表
root@ubuntu22:~# arp -n
Address HWtype HWaddress Flags Mask Iface
10.0.0.22 ether 00:0c:29:4f:a6:ce C ens38
172.0.1.131 ether b4:56:b9:f0:02:a5 C ens33
172.0.1.32 ether 00:e0:97:1c:20:1a C ens33
172.0.1.1 ether 00:e0:97:1c:20:1a C ens33
10.0.0.21 ether 00:0c:29:4f:a6:c4 C ens38
172.0.1.13 ether 00:0c:29:c1:16:a8 C ens33
10.0.0.22 ether 00:0c:29:4f:a6:ce C ens39
IP与MAC映射关系正常了。
