bridge table of allowed-address-pair is corruption after live-migration
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R2.20 |
Fix Committed
|
High
|
Naveen N | |||
R2.21.x |
Fix Committed
|
High
|
Naveen N | |||
R2.22.x |
Fix Committed
|
High
|
Naveen N | |||
Trunk |
Fix Committed
|
High
|
Naveen N |
Bug Description
Thare are three compute nodes and two virtual-machines which are running vrrp are on two compute nodes.
We move one virtual-machine by live migration which is Master VRRP to other compute node where Standby VRRP is running.
(Note, this issue is not seen if Master VRRP moves to same compute node where Standby VRRP is running.)
We also caputured at Virtual-machine, the Virtual-machine receaved two arp request to both instance-ip and VIP by vrouter-agnet, then the virtual-machine replied them.
As a result, inet table seems to be correct, but bridge table seems to be wrong.
Please see logs below, next hop of 10.0.1.254 is 31 which means local compute, but 0:0:5e:0:1:a is 24 which means remote compute.
It seems cruptin.
Compute-1(VRRP Master)---------
root@sv-6:~# rt --dump 1 | egrep "1.254/"
10.0.1.254/32 32 P - 31 0:0:5e:0:1:a(86308) <<<<<
root@sv-6:~# rt --dump 1 --family bridge
Kernel L2 Bridge table 0/1
Flags: L=Label Valid, Df=DHCP flood
Index DestMac Flags Label/VNID Nexthop
6444 52:54:0:6a:c0:92 Df - 3
21424 0:1:0:0:5:78 LDf 4 25
51272 2:51:b:a0:7c:b9 - 34
52932 2:97:39:39:bb:d0 LDf 19 21
86308 0:0:5e:0:1:a LDf 19 21 <<<<<
97192 ff:ff:ff:ff:ff:ff LDf 4 37
237456 2:d5:ac:a1:72:2 LDf 20 20
252916 0:0:5e:0:1:0 Df - 3
Routing table was fixed after vrrp Master was switched to another then back to origin virtual-machine.
Compute-1(VRRP Master)---------
root@sv-6:~# rt --dump 1 | grep "1.254/"
10.0.1.254/32 32 P - 31 0:0:5e:0:1:a(86308) <<<<<
root@sv-6:~# rt --dump 1 --family bridge
Kernel L2 Bridge table 0/1
Flags: L=Label Valid, Df=DHCP flood
Index DestMac Flags Label/VNID Nexthop
6444 52:54:0:6a:c0:92 Df - 3
21424 0:1:0:0:5:78 LDf 4 25
51272 2:51:b:a0:7c:b9 - 34
52932 2:97:39:39:bb:d0 LDf 19 21
86308 0:0:5e:0:1:a Df - 34 <<<<<
97192 ff:ff:ff:ff:ff:ff LDf 4 37
237456 2:d5:ac:a1:72:2 LDf 20 20
252916 0:0:5e:0:1:0 Df - 3
Compute-
root@sv-8:~# rt --dump 1 | grep .1.254/
10.0.1.254/32 32 LP 18 19 0:0:5e:0:1:a(86308)
root@sv-8:~# rt --dump 1 --family bridge
Kernel L2 Bridge table 0/1
Flags: L=Label Valid, Df=DHCP flood
Index DestMac Flags Label/VNID Nexthop
21424 0:1:0:0:5:78 LDf 4 21
51272 2:51:b:a0:7c:b9 LDf 19 19
52932 2:97:39:39:bb:d0 LDf 19 23
86308 0:0:5e:0:1:a LDf 19 19
97192 ff:ff:ff:ff:ff:ff LDf 4 28
111580 52:54:0:9b:10:5e Df - 3
237456 2:d5:ac:a1:72:2 - 41
252916 0:0:5e:0:1:0 Df - 3
information type: | Proprietary → Public Security |
information type: | Public Security → Public |
Changed in juniperopenstack: | |
assignee: | nobody → Hari Prasad Killi (haripk) |
tags: | added: vrouter |
Hi Daisuke-San,
We have tried the same scenario but not seeing the corruption. After migrating to new compute still VRRP MAC (Allowed Address Pair) point to local next hop. I have tried with Latest Mainline Build 2711 Kilo.
Please find observation below. Please let us know if we are trying something different.
Thanks and Regards,
Chhandak
VRRP master test-1: /home/ubuntu# ip a show UP,LOWER_ UP> mtu 16436 qdisc noqueue state UNKNOWN MULTICAST, UP,LOWER_ UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 >>>>>>> >>>>>>> >>VIP e9ff:fe48: 1882/64 scope link
--------------
root@vm-
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,
link/ether 02:1e:e9:48:18:82 brd ff:ff:ff:ff:ff:ff
inet 1.1.1.3/24 brd 1.1.1.255 scope global eth0
inet 1.1.1.50/32 scope global eth0>>>
inet6 fe80::1e:
valid_lft forever preferred_lft forever
Respective Compute ------- ------- ------- ------ ceph-perf2: ~# rt --dump 3 --family bridge>>>>>>>>>>>> cmbu-ceph-perf2: is current compute
-------
root@cmbu-
Kernel L2 Bridge table 0/3
Flags: L=Label Valid, Df=DHCP flood
Index DestMac Flags Label/VNID Nexthop ceph-perf2: ~# nh --get 72
Flags:Valid, Policy,
EncapFmly: 0806 Oif:6 Len:14 >>>>>>> >>>Pointing Local
2168 2:1e:e9:48:18:82 - 72
5084 ff:ff:ff:ff:ff:ff LDf 6 75
52076 0:0:5e:0:1:0 Df - 3
104012 2:6d:31:f9:dd:8e LDf 21 50
158240 0:0:5e:0:1:4 Df - 72 >>>>>>>>>>> VRRP MAC
178124 0:25:90:35:8a:1e Df - 3
root@cmbu-
Id:72 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:3
Encap Data: 02 1e e9 48 18 82 00 00 5e 00 01 00 08 00 >>>>>>>
root@vm- test-1: /home/ubuntu# tcpdump -ni eth0 not port 22 e9ff:fe48: 1882.546 > ff02::1:2.547: dhcp6 request e9ff:fe48: 1882.546: dhcp6 reply e9ff:fe48: 1882 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
05:03:52.900295 IP6 fe80::1e:
05:03:52.900663 IP6 ::.547 > fe80::1e:
05:03:52.922331 IP6 fe80::1e:
05:03:53.646801 IP 1.1.1.4 > 1.1.1.50: ICMP echo request, id 1944, seq 684, length 64
05:03:53.646825 IP 1.1.1.50 > 1.1.1.4: ICMP echo reply, id 1944, seq 684, length 64
05:03:54.647946 IP 1.1.1.4 > 1.1.1.50: ICMP echo request, id 1944, seq 685, length 64
05:03:54.647964 IP 1.1.1.50 > 1.1.1.4: ICMP echo reply, id 1944, seq 685, length 64>>>>> Responding to PING for VIP IP
05:03:54.702586 IP6 fe80::6d:31
Migrating VRRP Master VM to New Compute ------- ------- ------- ------- ------- ------- ------- ------- ---- ceph-perf1: ~# nova live-migration 6b483f3b- daaf-44df- ba16-fe0fb7daf3 75 cmbu-ceph-perf3
-------
root@cmbu-
root@cmbu- ceph-perf3: ~# rt --dump 1...