看到网上介绍可以通过Linux bridge 开启hairpin方式测试macvlan vepa模式，但是没有找到详细资料。我尝试测试总提示错误信息，无法实现，经过几天的研究，我总算实现模拟测试，记录如下：
参考
1.Linux Macvlan
2.图解几个与Linux网络虚拟化相关的虚拟网卡-VETH/MACVLAN/MACVTAP/IPVLAN
3.kube-proxy IPVS 模式的工作原理
4.Linux brctl 命令，虚拟网络设备 LinuxBridge 管理工具
5.Linux 虚拟网络设备 bridge
6.Linux虚拟网络设备—之使用Veth pair连接linux网桥bridge
7.brctl快速入门与基础
环境
1. 操作系统
Centos7.9
2. 安装包
安装测试环境需要的包
[root@centos7-10 ~]# yum install -y net-tools iputils telnet traceroute iproute bridge-utils NetworkManager
- net-tools：netstat命令
- iputils：ping命令
- telnet：telnet命令
- traceroute：traceroute命令
- iproute：ip命令
- bridge-utils：brctl命令
- NetworkManager 网络管理命令
如果是ubuntu 命令如下：
apt install -y net-tools inetutils-ping telnet traceroute iproute2 bridge-utils network-manager
Linux bridge 介绍
Bridge概念详见：Linux brctl 命令，虚拟网络设备 LinuxBridge 管理工具
Macvlan 介绍
Macvlan概念详见：Linux brctl 命令，虚拟网络设备 LinuxBridge 管理工具
模拟测试
1. 测试流程
因交换不支持802.1q，故采用Linux bridge开启服务器托管网hairpin方式，模拟测试macvlan vepa模式，步骤如下：
- 创建Linux bridge br0
- 创建veth pair：veth0和veth0_1
- veth0加入br0
- veth0_1作为父网卡，创建两个macvlan子网卡veth0_1.101****和veth0_1.102，模式vepa
- 创建两个namespace：ns101和ns102
- veth0_1.101加入ns101，配置IP 10.211.55.101，启用
- veth0_1.102加入ns102，配置IP 10.211.55.102，启用
- 测试br0下关闭和开启接口veth0 hairpin时，macvlan vepa网络通讯情况
详见下图：
2. 创建Linux bridge br0
- 查看当前bridge
[root@centos7-18 ~]# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.5254009f1377 yes virbr0-nic
- 创建bridge br0
// 创建br0
[root@centos7-18 ~]# brctl addbr br0
// 启用br0
[root@centos7-18 ~]# ip link set br0 up
// 查看bridge
[root@centos7-18 ~]# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.000000000000 no
virbr0 8000.5254009f1377 yes virbr0-nic
3. 创建veth pair：veth0和veth0_1
- 查看当前网卡
[root@centos7-18 ~]# ip a
1: lo: LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s5: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
inet 10.211.55.18/24 brd 10.211.55.255 scope global enp0s5
valid_lft forever preferred_lft forever
inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic
valid_lft 2591486sec preferred_lft 604286sec
inet6 fe80::21c:42ff:fe60:87b2/64 scope link
valid_lft forever preferred_lft forever
3: enp0s6: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:1c:42:d1:70:62 brd ff:ff:ff:ff:ff:ff
inet 10.211.55.21/24 brd 10.211.55.255 scope global noprefixroute dynamic enp0s6
valid_lft 1322sec preferred_lft 1322sec
inet6 fdb2:2c26:f4e4:0:2a52:f262:86d:6cd5/64 scope global noprefixroute dynamic
valid_lft 2591486sec preferred_lft 604286sec
inet6 fe80::bfab:127:7500:dd3c/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: virbr0: NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:9f:13:77 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
5: virbr0-nic: BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:9f:13:77 brd ff:ff:ff:ff:ff:ff
6: br0: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 86:13:97:70:a2:e2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::8413:97ff:fe70:a2e2/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
- 创建虚拟网卡veth0和veth0_1
// 创建veth0和veth0_1
[root@centos7-18 ~]# ip link add veth0 type veth peer name veth0_1
// 启用veth0和veth0_1
[root@centos7-18 ~]# ip link set veth0 up
[root@centos7-18 ~]# ip link set veth0_1 up
// 查看veth
[root@centos7-18 ~]# ip a | grep -A4 veth0
7: veth0_1@veth0: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7a:87:ef:c6:77:9b brd ff:ff:ff:ff:ff:ff
inet6 fe80::7887:efff:fec6:779b/64 scope link
valid_lft forever preferred_lft forever
8: veth0@veth0_1: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 86:08:8e:91:09:fe brd ff:ff:ff:ff:ff:ff
inet6 fe80::8408:8eff:fe91:9fe/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
4. veth0加入br0
// veth0加入br0
[root@centos7-18 ~]# brctl addif br0 veth0
[root@centos7-18 ~]# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.86088e9109fe no veth0
virbr0 8000.5254009f1377 yes virbr0-nic
5. veth0_1作为父网卡，创建两个macvlan子网卡veth0_1.101和veth0_1.102，模式vepa
- 父网卡 veth0_1
- 子网卡 veth0_1.101
- 子网卡 veth0_1.102
// 创建两个macvlan子网卡veth0_1.101和veth0_1.102，模式vepa
[root@centos7-18 ~]# ip link add link veth0_1 name veth0_1.101 type macvlan mode vepa
[root@centos7-18 ~]# ip link add link veth0_1 name veth0_1.102 type macvlan mode vepa
// 查看创建结果
[root@centos7-18 ~]# ip a | grep -A5 veth0
7: veth0_1@veth0: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7a:87:ef:c6:77:9b brd ff:ff:ff:ff:ff:ff
inet6 fe80::7887:efff:fec6:779b/64 scope link
valid_lft forever preferred_lft forever
8: veth0@veth0_1: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether 86:08:8e:91:09:fe brd ff:ff:ff:ff:ff:ff
inet6 fe80::8408:8eff:fe91:9fe/64 scope link
valid_lft forever preferred_lft forever
9: veth0_1.101@veth0_1: BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff
10: veth0_1.102@veth0_1: BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 22:f8:d5:8b:c1:63 brd ff:ff:ff:ff:ff:ff
6. 创建两个namespace：ns101和ns102
// 创建ns101和ns102
[root@centos7-18 ~]# ip netns add ns101
[root@centos7-18 ~]# ip netns add ns102
// 查看结果
[root@centos7-18 ~]# ip netns list
ns102
ns101
7. veth子网卡加入namespace，配置网卡并启用
ns101和ns102网络隔离，将两个macvlan子网卡（veth0_1.101和veth0_1.102）分别加入其中
- veth0_1.101加入ns101，配置IP 10.211.55.101，启用
// veth0_1.101加入ns101
[root@centos7-18 ~]# ip link set veth0_1.101 netns ns101
// 查看ns101的网卡
[root@centos7-18 ~]# ip netns exec ns101 ip a
1: lo: LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: veth0_1.101@if7: BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
// ns101启用lo。不启用ping自己的IP，会不通
[root@centos7-18 ~]# ip netns exec ns101 ip link set lo up
// ns101配置IP 10.211.55.101
[root@centos7-18 ~]# ip netns exec ns101 ip addr add 10.211.55.101/24 dev veth0_1.101
// ns101启用veth0_1.101。
[root@centos7-18 ~]# ip netns exec ns101 ip link set veth0_1.101 up
// 查看ns101 网卡
[root@centos7-18 ~]# ip netns exec ns101 ip a
1: lo: LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
9: veth0_1.101@if7: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.101/24 scope global veth0_1.101
valid_lft forever preferred_lft forever
inet6 fe80::b03e:6eff:feae:7457/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.101
PING 10.211.55.101 (10.211.55.101) 56(84) bytes of data.
6服务器托管网4 bytes from 10.211.55.101: icmp_seq=1 ttl=64 time=0.037 ms
64 bytes from 10.211.55.101: icmp_seq=2 ttl=64 time=0.058 ms
--- 10.211.55.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.037/0.047/0.058/0.012 ms
- veth0_1.102加入ns102，配置IP 10.211.55.102，启用
// veth0_1.102加入ns102
[root@centos7-18 ~]# ip link set veth0_1.102 netns ns102
[root@centos7-18 ~]# ip netns exec ns102 ip link set lo up
[root@centos7-18 ~]# ip netns exec ns102 ip addr add 10.211.55.102/24 dev veth0_1.102
[root@centos7-18 ~]# ip netns exec ns102 ip link set veth0_1.102 up
[root@centos7-18 ~]#
// 查看ns102 网卡
[root@centos7-18 ~]# ip netns exec ns102 ip a
1: lo: LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
10: veth0_1.102@if7: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 22:f8:d5:8b:c1:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.102/24 scope global veth0_1.102
valid_lft forever preferred_lft forever
inet6 fe80::20f8:d5ff:fe8b:c163/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]# ip netns exec ns102 ping -c2 10.211.55.102
PING 10.211.55.102 (10.211.55.102) 56(84) bytes of data.
64 bytes from 10.211.55.102: icmp_seq=1 ttl=64 time=0.035 ms
64 bytes from 10.211.55.102: icmp_seq=2 ttl=64 time=0.055 ms
--- 10.211.55.102 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.035/0.045/0.055/0.010 ms
[root@centos7-18 ~]#
8. 模拟测试macvlan vepa网络通讯情况
测试br0在关闭和开启接口veth0 hairpin时，macvlan vepa网络通讯情况
- br0关闭veth0 hairpin时（默认状态是off），macvlan子网卡无法互相访问
// ns101 无法ping通 ns102的10.211.55.102
[root@centos7-18 ~]# ip netns exec ns101 ip a | grep veth
9: veth0_1.101@if7: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.211.55.101/24 scope global veth0_1.101
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.102
PING 10.211.55.102 (10.211.55.102) 56(84) bytes of data.
From 10.211.55.18 icmp_seq=1 Destination Host Unreachable
From 10.211.55.18 icmp_seq=2 Destination Host Unreachable
--- 10.211.55.102 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1000ms
// ns102 无法ping通 ns101的10.211.55.101
[root@centos7-18 ~]# ip netns exec ns102 ip a | grep veth
10: veth0_1.102@if7: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.211.55.102/24 scope global veth0_1.102
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip netns exec ns102 ping -c2 10.211.55.101
PING 10.211.55.101 (10.211.55.101) 56(84) bytes of data.
From 10.211.55.18 icmp_seq=1 Destination Host Unreachable
From 10.211.55.18 icmp_seq=2 Destination Host Unreachable
--- 10.211.55.101 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 999ms
-
br0开启veth0 hairpin时，macvlan子网卡经过br0转发，macvlan子网卡可以互相访问
- 开启veth0 hairpin
- 测试macvaln网络联通性
// 开启hairpin
[root@centos7-18 ~]# brctl hairpin br0 veth0 on
// 查看br0的veth0开启hairpin结果
[root@centos7-18 ~]# bridge -d link | grep -A5 veth0
8: veth0 state UP @veth0_1: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 2
hairpin on guard off root_block off fastleave off learning on flood on mcast_flood on
[root@centos7-18 ~]#
// ns101 可以ping通 ns102的10.211.55.102
[root@centos7-18 ~]# ip netns exec ns101 ip a | grep -A5 veth0
9: veth0_1.101@if7: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.101/24 scope global veth0_1.101
valid_lft forever preferred_lft forever
inet6 fe80::b03e:6eff:feae:7457/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.102
PING 10.211.55.102 (10.211.55.102) 56(84) bytes of data.
64 bytes from 10.211.55.102: icmp_seq=1 ttl=64 time=0.048 ms
64 bytes from 10.211.55.102: icmp_seq=2 ttl=64 time=0.095 ms
--- 10.211.55.102 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.048/0.071/0.095/0.024 ms
[root@centos7-18 ~]#
// ns102 可以ping通 ns101的10.211.55.101
[root@centos7-18 ~]# ip netns exec ns102 ip a | grep -A5 veth0
10: veth0_1.102@if7: BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 22:f8:d5:8b:c1:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.211.55.102/24 scope global veth0_1.102
valid_lft forever preferred_lft forever
inet6 fe80::20f8:d5ff:fe8b:c163/64 scope link
valid_lft forever preferred_lft forever
[root@centos7-18 ~]#
[root@centos7-18 ~]# ip netns exec ns102 ping -c2 10.211.55.101
PING 10.211.55.101 (10.211.55.101) 56(84) bytes of data.
64 bytes from 10.211.55.101: icmp_seq=1 ttl=64 time=0.047 ms
64 bytes from 10.211.55.101: icmp_seq=2 ttl=64 time=0.077 ms
--- 10.211.55.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.047/0.062/0.077/0.015 ms
[root@centos7-18 ~]#
总结
通过Linux bridge 开启接口hairpin的方式，模拟macvlan vepa在外部交换支持802.1q的情况下，同一父网卡下的多个子网卡之间是可以通讯的。
之所以使用bridge、veth pair和macvlan组合进行模拟测试，是因为macvlan的父网卡不能属于其它bridge，如果尝试加入会报以下错误信息：
[root@centos7-18 ~]# brctl addif br0 veth0_1
device veth0_1 is already a member of a bridge; can't enslave it to bridge br0.
[root@centos7-18 ~]#
服务器托管，北京服务器托管，服务器租用 http://www.fwqtg.net
相关推荐: iOS Class Guard 成功了，但无法区分差异
转载： iOS Class Guard 成功了，但无法区分差异 我正在开发一个静态库，并使用 Polidea 的 iOS Class Guard 来混淆我的静态库。我按照步骤在项目的根路径中下载 obfuscate_project，更改其中所需的名称，最后在终…
