rarp packet will be dropped in flows cause vm connectivity broken after live-migration

[Yang Li]

When we live-migration a VM, at the moment of VM start in destination nodes, the VM will send 5 rarp packets(broadcast) to trigger mac learn for switch, then other same network VMs can send packets to this VM directly, but actually these rarp packets will be dropped in flows of br-int, they won't trigger mac learn for switch, then the connectivity of other VMs to this VM will be broken, they need a broadcast to search the new position of this VM, it will waste some few seconds.

Without live-migration, you can use nping command in VM to simulate rarp packets sent from VM:
nping --arp --arp-type rarp --arp-target-mac <vm-mac> --ether-type rarp <vm-dhcp-host-ip>
For example:
nping --arp --arp-type rarp --arp-target-mac fa:16:3e:02:49:d3 --ether-type rarp 192.168.111.2

And you can use tcpdump to capture these packets, and also you can see these packets are dropped in (table 71 , priority=10, in_port=16), 16 is the tap device port in br-int

# ovs-ofctl show br-int | grep 02:49:d3
 16(tap80be4c89-6d): addr:fe:16:3e:02:49:d3
#
# ovs-ofctl dump-flows br-int | grep table=71 | grep in_port=16 | grep priority=10
 cookie=0xbb76a2777919bc71, duration=1037.306s, table=71, n_packets=13, n_bytes=594, idle_age=12, priority=10,ct_state=-trk,reg5=0x10,in_port=16 actions=drop
#
# tcpdump 'rarp or arp' -i tap80be4c89-6d -nne
tcpdump: WARNING: tap80be4c89-6d: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap80be4c89-6d, link-type EN10MB (Ethernet), capture size 65535 bytes
10:11:11.895958 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28
10:11:12.896280 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28
10:11:13.897538 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28
10:11:14.898815 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28
10:11:15.900093 fa:16:3e:02:49:d3 > ff:ff:ff:ff:ff:ff, ethertype Reverse ARP (0x8035), length 42: Reverse Request who-is fa:16:3e:02:49:d3 tell fa:16:3e:02:49:d3, length 28

^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel
# ovs-ofctl dump-flows br-int | grep table=71 | grep in_port=16 | grep priority=10
 cookie=0xbb76a2777919bc71, duration=1063.414s, table=71, n_packets=18, n_bytes=804, idle_age=8, priority=10,ct_state=-trk,reg5=0x10,in_port=16 actions=drop

You can see 5 rarp packets are captured by tcpdump, and in the drop flow, the count of n_packets was increased from 13 to 18, this means these 5 rarp packets are dropped by this flow.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/661938
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=7d11cd6211a047d73c2a4795e8daa1b6a4851fa6
Submitter: Zuul
Branch: master

commit 7d11cd6211a047d73c2a4795e8daa1b6a4851fa6
Author: Yang Li <email address hidden>
Date: Wed May 29 17:21:53 2019 +0800

    Add rarp protocol for neutron constants

    We should support rarp protocol(0x8035) for live-migration
    A release note is also included.

    Change-Id: I52c115acdb1a8c464f5fc6273f6d8c4d0c6526e9
    Partial-Bug: #1831404

[norman shen (jshen28)]

Hi, I am curious about the reason why vm sending rarp packets after live-migration? what os is affected? as well as, is there a way to reproduce this issue stably? Any reference related to this bug will be appreciated. Thank you.

[Yang Li (yang-li)]

It was designed by qemu, when the vm is live-migrate completed, the vm will send 5 rarp packets:
  88 static int announce_self_create(uint8_t *buf,
  89 uint8_t *mac_addr)
  90 {
  91 /* Ethernet header. */
  92 memset(buf, 0xff, 6); /* destination MAC addr */
  93 memcpy(buf + 6, mac_addr, 6); /* source MAC addr */
  94 *(uint16_t *)(buf + 12) = htons(ETH_P_RARP); /* ethertype */
  95
  96 /* RARP header. */
  97 *(uint16_t *)(buf + 14) = htons(ARP_HTYPE_ETH); /* hardware addr space */
  98 *(uint16_t *)(buf + 16) = htons(ARP_PTYPE_IP); /* protocol addr space */
  99 *(buf + 18) = 6; /* hardware addr length (ethernet) */
 100 *(buf + 19) = 4; /* protocol addr length (IPv4) */
 101 *(uint16_t *)(buf + 20) = htons(ARP_OP_REQUEST_REV); /* opcode */
 102 memcpy(buf + 22, mac_addr, 6); /* source hw addr */
 103 memset(buf + 28, 0x00, 4); /* source protocol addr */
 104 memcpy(buf + 32, mac_addr, 6); /* target hw addr */
 105 memset(buf + 38, 0x00, 4); /* target protocol addr */
 106
 107 /* Padding to get up to 60 bytes (ethernet min packet size, minus FCS). */
 108 memset(buf + 42, 0x00, 18);
 109
 110 return 60; /* len (FCS will be added by hardware) */
 111 }

Use some big flavor with high memory, it can be reproduced easily.

[Yang Li (yang-li)]

This patch showed why the vm send rarp packets after migration.

https://lists.gnu.org/archive/html/qemu-devel/2009-10/msg01457.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/661921
Committed: https://opendev.org/openstack/neutron/commit/5c246e6b578cda9e051f8a36b9b1687963e8aeb0
Submitter: "Zuul (22348)"
Branch: master

commit 5c246e6b578cda9e051f8a36b9b1687963e8aeb0
Author: Yang Li <email address hidden>
Date: Wed May 29 17:13:44 2019 +0800

    Support rarp protocol for egress table

    When vm live-migration is done, at the vm start moment, the
    vm will send rarp packet, but the table 71 of openflow will
    drop these packets, because they cannot match any flow in
    high priority(65 70 80 95), only the priority 10 flow can
    match them, and its aciton is drop, so all the rarp packets
    will be dropped, and the connectivity will be broken after
    live-migration.

    Change-Id: Id1cc34f0f544b2cb8cc89af2cb81bb15e4605a0e
    Partial-Bug: #1831404
    Depends-On: https://review.openstack.org/661938