Live migration between hosts with provider networks works sporadically

[Kyle Mestery]

Using this OVS and OVN version:

vagrant@ovn-controller:~/devstack$ sudo ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.5.90
Compiled Feb 15 2016 03:13:53
DB Schema 7.12.1
vagrant@ovn-controller:~/devstack$

With Neutron from master (2-15-2016).

I'm using provider networks for my networking with OVN. When I migrate a VM across a compute host, the networking is sometimes retained, and sometimes it's not. Sometimes it works the first migration, but a subsequent migration fails.

Before a migration, things look like this:

vagrant@ovn-controller:~/devstack$ ovn-nbctl show
    lswitch 6eb40d01-091a-4384-84f4-fcebc7157fbd (neutron-36eaece0-a01b-4a39-b35f-9098067f0ed6)
        lport 98cbd16d-d281-41d2-b5f2-6b9ac50ebdf2
            addresses: ["fa:16:3e:89:4d:5a 172.16.1.3", "fa:16:3e:89:4d:5a fd64:3109:4ba3:0:f816:3eff:fe89:4d5a"]
        lport 4a0dbb36-3229-4fc5-9436-4faeb3b2238d
            addresses: ["fa:16:3e:d7:3e:bc 172.16.1.1"]
        lport b7a641e8-e90c-4f01-a568-8c34a03313fa
            addresses: ["fa:16:3e:80:55:6d 172.16.1.2", "fa:16:3e:80:55:6d fd64:3109:4ba3:0:f816:3eff:fe80:556d"]
    lswitch cc4c73ae-2f1b-46bb-8eaa-d5933693a7f1 (neutron-1ec64402-eff6-4244-857b-cf0dce8e2977)
        lport 1ec64402-eff6-4244-857b-cf0dce8e2977
            addresses: ["fa:16:3e:6b:94:d7 10.10.0.51"]
        lport provnet-1ec64402-eff6-4244-857b-cf0dce8e2977
            addresses: ["unknown"]
    lswitch 8c23c1ce-214c-44cc-ae7f-fcd54b73aff2 (neutron-179edc2a-bccd-4046-ac38-81d69833440a)
        lport provnet-179edc2a-bccd-4046-ac38-81d69833440a
            addresses: ["unknown"]
        lport 179edc2a-bccd-4046-ac38-81d69833440a
            addresses: ["fa:16:3e:90:14:91 10.10.0.53"]
    lswitch 81a582ea-0249-4654-8d83-3d9982fef43e (neutron-583c7fe4-3de5-4f11-820d-b4fc627e4ef8)
        lport 583c7fe4-3de5-4f11-820d-b4fc627e4ef8
            addresses: ["fa:16:3e:3c:61:74 10.10.0.59"]
        lport provnet-583c7fe4-3de5-4f11-820d-b4fc627e4ef8
            addresses: ["unknown"]
    lswitch b2b0a19e-022a-44a4-adec-5f27acbdfbf0 (neutron-ce0c0d02-b2dd-48c7-9baa-df12bba022ac)
        lport ce0c0d02-b2dd-48c7-9baa-df12bba022ac
            addresses: ["fa:16:3e:d1:c3:6c 10.10.0.52"]
        lport provnet-ce0c0d02-b2dd-48c7-9baa-df12bba022ac
            addresses: ["unknown"]
    lswitch f8cd658b-e2d9-415f-9fd5-db00dbbd894e (neutron-7b50dd6f-c2f5-4ef8-abdd-9d6d879bcd09)
vagrant@ovn-controller:~/devstack$ ovn-sbctl show
Chassis "0378fa8e-52c2-4cc6-b85d-87c1edab37ae"
    Encap geneve
        ip: "192.168.33.32"
    Port_Binding "179edc2a-bccd-4046-ac38-81d69833440a"
    Port_Binding "98cbd16d-d281-41d2-b5f2-6b9ac50ebdf2"
Chassis "e4e6d72c-ac0c-4e2d-84f7-e0d67d5b1177"
    Encap geneve
        ip: "192.168.33.31"
    Port_Binding "b7a641e8-e90c-4f01-a568-8c34a03313fa"
    Port_Binding "583c7fe4-3de5-4f11-820d-b4fc627e4ef8"
    Port_Binding "ce0c0d02-b2dd-48c7-9baa-df12bba022ac"
Chassis "2cc10a56-7f95-406c-bd9e-e09c142de7e4"
    Encap geneve
        ip: "192.168.33.12"
vagrant@ovn-controller:~/devstack$

Post the first migration, they look like this (in a failing condition):

vagrant@ovn-controller:~/devstack$ ovn-nbctl show
    lswitch 6eb40d01-091a-4384-84f4-fcebc7157fbd (neutron-36eaece0-a01b-4a39-b35f-9098067f0ed6)
        lport 98cbd16d-d281-41d2-b5f2-6b9ac50ebdf2
            addresses: ["fa:16:3e:89:4d:5a 172.16.1.3", "fa:16:3e:89:4d:5a fd64:3109:4ba3:0:f816:3eff:fe89:4d5a"]
        lport 4a0dbb36-3229-4fc5-9436-4faeb3b2238d
            addresses: ["fa:16:3e:d7:3e:bc 172.16.1.1"]
        lport b7a641e8-e90c-4f01-a568-8c34a03313fa
            addresses: ["fa:16:3e:80:55:6d 172.16.1.2", "fa:16:3e:80:55:6d fd64:3109:4ba3:0:f816:3eff:fe80:556d"]
    lswitch cc4c73ae-2f1b-46bb-8eaa-d5933693a7f1 (neutron-1ec64402-eff6-4244-857b-cf0dce8e2977)
        lport 1ec64402-eff6-4244-857b-cf0dce8e2977
            addresses: ["fa:16:3e:6b:94:d7 10.10.0.51"]
        lport provnet-1ec64402-eff6-4244-857b-cf0dce8e2977
            addresses: ["unknown"]
    lswitch 8c23c1ce-214c-44cc-ae7f-fcd54b73aff2 (neutron-179edc2a-bccd-4046-ac38-81d69833440a)
        lport provnet-179edc2a-bccd-4046-ac38-81d69833440a
            addresses: ["unknown"]
        lport 179edc2a-bccd-4046-ac38-81d69833440a
            addresses: ["fa:16:3e:90:14:91 10.10.0.53"]
    lswitch 81a582ea-0249-4654-8d83-3d9982fef43e (neutron-583c7fe4-3de5-4f11-820d-b4fc627e4ef8)
        lport 583c7fe4-3de5-4f11-820d-b4fc627e4ef8
            addresses: ["fa:16:3e:3c:61:74 10.10.0.59"]
        lport provnet-583c7fe4-3de5-4f11-820d-b4fc627e4ef8
            addresses: ["unknown"]
    lswitch b2b0a19e-022a-44a4-adec-5f27acbdfbf0 (neutron-ce0c0d02-b2dd-48c7-9baa-df12bba022ac)
        lport ce0c0d02-b2dd-48c7-9baa-df12bba022ac
            addresses: ["fa:16:3e:d1:c3:6c 10.10.0.52"]
        lport provnet-ce0c0d02-b2dd-48c7-9baa-df12bba022ac
            addresses: ["unknown"]
    lswitch f8cd658b-e2d9-415f-9fd5-db00dbbd894e (neutron-7b50dd6f-c2f5-4ef8-abdd-9d6d879bcd09)
vagrant@ovn-controller:~/devstack$ ovn-sbctl show
Chassis "0378fa8e-52c2-4cc6-b85d-87c1edab37ae"
    Encap geneve
        ip: "192.168.33.32"
    Port_Binding "179edc2a-bccd-4046-ac38-81d69833440a"
    Port_Binding "98cbd16d-d281-41d2-b5f2-6b9ac50ebdf2"
    Port_Binding "583c7fe4-3de5-4f11-820d-b4fc627e4ef8"
Chassis "e4e6d72c-ac0c-4e2d-84f7-e0d67d5b1177"
    Encap geneve
        ip: "192.168.33.31"
    Port_Binding "b7a641e8-e90c-4f01-a568-8c34a03313fa"
    Port_Binding "ce0c0d02-b2dd-48c7-9baa-df12bba022ac"
Chassis "2cc10a56-7f95-406c-bd9e-e09c142de7e4"
    Encap geneve
        ip: "192.168.33.12"
vagrant@ovn-controller:~/devstack$

The VM itself is ok:

agrant@ovn-controller:~/devstack$ nova list --all
+--------------------------------------+------+----------------------------------+--------+------------+-------------+---------------------+
| ID | Name | Tenant ID | Status | Task State | Power State | Networks |
+--------------------------------------+------+----------------------------------+--------+------------+-------------+---------------------+
| 8ba3e11e-8305-49d6-8c9b-689dcc4f6723 | vm1 | e588180c1c9a44c8982389e4dd617348 | ACTIVE | - | Running | provider=10.10.0.59 |
+--------------------------------------+------+----------------------------------+--------+------------+-------------+---------------------+
vagrant@ovn-controller:~/devstack$

vagrant@ovn-controller:~/devstack$ nova show 8ba3e11e-8305-49d6-8c9b-689dcc4f6723
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | ovn-compute2 |
| OS-EXT-SRV-ATTR:hostname | vm1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | ovn-compute2 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000006 |
| OS-EXT-SRV-ATTR:kernel_id | 8ed23a40-f1eb-48a1-ab56-d079683df869 |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | ddd15fe1-1643-4ca4-9c59-ba6d4a674651 |
| OS-EXT-SRV-ATTR:reservation_id | r-p9l7widi |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-02-16T01:00:09.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2016-02-16T01:00:04Z |
| flavor | ovntenant (27) |
| hostId | 1aa531f37cbdfbbddec744df074a2e26e6bec7bbcc46baf6ea098f38 |
| id | 8ba3e11e-8305-49d6-8c9b-689dcc4f6723 |
| image | cirros-0.3.4-x86_64-uec (2b31db62-33ac-4079-9d93-ea7d1597323d) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | vm1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 80 |
| provider network | 10.10.0.59 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | e588180c1c9a44c8982389e4dd617348 |
| updated | 2016-02-16T01:03:16Z |
| user_id | 4ead3522ebbf46c3a98d5a23ed40a2b1 |
+--------------------------------------+----------------------------------------------------------------+
vagrant@ovn-controller:~/devstack$

I cannot get to the console however:

vagrant@ovn-controller:~/devstack$ nova console-log 8ba3e11e-8305-49d6-8c9b-689dcc4f6723

vagrant@ovn-controller:~/devstack$

[Russell Bryant (russellb)]

Is there any chance you could provide access to a compute node with a VM in the failed state? If not, maybe we could just get together on IRC for some live debugging.

[Russell Bryant (russellb)]

that is, if you'd like help. Let me know. :-)

[Russell Bryant (russellb)]

as discussed on IRC, this may be due to OVN not generating a gratuitous ARP in this case. That's something we need to add even if there's more going on here.

[Kyle Mestery (mestery)]

I think the problem is the lack of the gratuitous arp as you say in comment #3 Russell. Once that happens, we'll be in good shape here.

[Richard Theis (rtheis)]

Hi Kyle,

First, thank you for https://review.openstack.org/#/c/280728/.

What is the failure condition that you noticed in your tests? Also, does this impact project networks or just provider networks?

Thanks,
Richard

[Richard Theis (rtheis)]

I think I was able to reproduce the problem. I lost network connectivity to the VM after the live migration. After hard rebooting the VM, network connectivity is restored.

[Richard Theis (rtheis)]

I haven't seen an issue with project networks.

With respect to provider networks, I haven't been able to confirm or deny that a gratuitous arp fixes everything. This is because I'm periodically seeing (from the Dashboard console) the VM operating system crash when using cirros 0.3.4. I'll see if another image has better luck.

[Han Zhou (zhouhan)]

Sorry that I don't understand this problem yet. If it is lack of GARP, then waiting for some time (300 secs?) should have the problem fixed without rebooting the VM, because the learned mac will be aged in ovs bridges and physical switches (by default 300 secs) and flooding should reach the new VIF finally. Isn't it?

[Richard Theis (rtheis)]

That's what I'm trying to sort out. Waiting or clearing the ARP cache doesn't work for me. This is because I think the VM is crashing as a result of the live migration. I can confirm this sometimes using the Dashboard console and other times I cannot. I'll keep investigating.

That said, Kyle was able to wait and have the problem fixed.

[Richard Theis (rtheis)]

Adding link to ovs mailing list bug report: http://openvswitch.org/pipermail/discuss/2016-February/020275.html

[Richard Theis (rtheis)]

I've just recreated the issue using the private network.

[Russell Bryant (russellb)]

> I've just recreated the issue using the private network.

That's interesting, because lack of GARP would not explain that.

I would make sure the VM is functional and that the problem really is the network. Beyond that, we'll have to dig into OVN to figure out what happened. My process would be roughly something like.

1) Checking "ovn-sbctl show" to make sure that the port bindings properly reflect that a port has moved.

2) Assuming #1 is good, I'd be digging into flows to see where my packets are going. Unfortunately, this isn't the easiest thing to do. It's easiest on a system that's otherwise idle, as you can just look at packet counters in the flows and see what's increasing.

A helpful command for watching counters increase: watch -d -n 0.5 "sudo ovs-ofctl -O OpenFlow13 dump-flows br-int | cut -f3- -d','"

[Richard Theis (rtheis)]

Hi Russell, thank you for the debug tips. I'll give them a try today.

[Richard Theis (rtheis)]

I verified that cold migrations on private or provider networks are successful, which I would expect. Then I went back to live migrations and verified the port bindings properly reflect that a port has moved. However, once again I'm seeing the VM crash after the live migration and am unable to reproduce what I saw yesterday. :(

[Richard Theis (rtheis)]

The VM didn't crash for several live migrations and here is what I observed...

I did as suggested and watched (watch -d -n 5 "sudo ovs-ofctl -O OpenFlow13 dump-flows br-int | cut -f3- -d','") the packet counters in the flows on both compute nodes during the live migration and after when I attempted to ping my VM over the provider network (ping -c1 10.10.0.115). When I saw the ping lose the packet, the counters increased on the wrong compute node (i.e. the compute node where the VM use to be). Trying the ping about a minute later was successful and the counters increased on the correct compute node. I observed this behavior a couple times.

I'm going to try redeploying my environment with larger compute nodes to see if that help prevent the VMs from crashing. I tried using a ubuntu image instead of cirros but didn't have much luck with that...maybe larger compute nodes will help. When the VMs crash, top shows libvirt+ consuming 90+% of the cpu. I would really like to eliminate the VM crash from the picture.

[Russell Bryant (russellb)]

To be clear, was this last test with a provider network? If so, the lack of GARP would describe what you just saw.

[Richard Theis (rtheis)]

Yes, the last test was with a provider network.

[Richard Theis (rtheis)]

I tried the following to prevent the VMs from occasionally crashing after the live migration but none solved the problem:
(1) Redeployed environment with larger compute nodes
(2) Launched VMs on provider or private network
(3) Launched VMs as demo or admin user
(4) Launched VMs using cirros or ubuntu trusty image
(5) Redeployed environment using latest ubuntu/trusty64 vagrant box

I suspect the VM crash may be a qemu-kvm bug. I may try installing a newer version per https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1297218

I did notice the time on the vagrant VMs was drifting which could impact live migration and possibly cause the VMs to crash. I'll look into fixing that as part of the networking-ovn vagrant deployment.

Now with respect to the lack of GARP... As was noted several times in this bug, the lack of GARP when deploying on a provider network can be recreated and needs to be fixed. I was NOT able to recreate the same problem on a private network. I *think* my comment #11 about recreating on the private network was NOT accurate and may have been a result of problems with the VM.

@Russell, do you know the status of the OVN GARP fix for provider networks?

[Russell Bryant (russellb)]

There's no status update other than it being a confirmed issue and TODO item. Nobody is working on it yet that I know of.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/293419

[Richard Theis (rtheis)]

VM crash still occurs with updated qemu per https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1297218 :(

[Ramu Ramamurthy (ramu-ramamurthy)]

We are able to consistently see a closely-related ping problem on provider networks.

1) VM-1 is booted with IP1,mac1 and after a while destroyed, and then VM2 is booted with IP1->mac2.
This can happen when a pool of IP addresses on provider network is reused by VMs

2) Then, attempt to ping IP1, and we find that the router on the providernet is
sending ICMP echos with dest mac1 instead of dest mac2 . The router ages out the arp entry after
a long time (4 hours or so - which is the cisco arp timeout default) . During that period, IP1 is unpingable.

But this may not be a serious problem because VM2 stayed "silent", and didnt do any network activity - during that period,
which is probably uncommon - ie normally VMs would arp for their default router and have continuous
network activity with the default router.

In any case, the following changes are suggested by the above problem, and I can volunteer to make code
changes for these if there is nobody else working on them.

1) OVN must NOT respond to unsolicited self arp requests from VMs on localnet and must flood them.
Currently, OVN responds to self-arp requests also. This is likely a simple change in the l2 arp response table,
and is probably needed anyway.

2) However, the above does not solve the problem because, many VMs do not send garps by themselves when they
come up, and so ideally OVN should send GARP on the localnet for attached logical ports - when they get bound,
so routers and switches on the broadcast segment can update their arp caches. At the outset this does not seem
like a simple change.

[Amitabha Biswas (azbiswas)]

Hi Ramu,

Can you explain a little more why proposal change 1 is needed?

The latest version of OVN replies to ARPs requests using lflows (these will have the latest ip->mac mapping) for ARP requests received on non-localnet ports. ARP requests received on localnet ports should have been flooded already - is this not happening?

For 2: yes we need to generate a GARP.

Amitabha

[Ramu Ramamurthy (ramu-ramamurthy)]

@Amitabha, What I meant by was the following:

Currently, If a VM (IP,mac) on the logical-switch for provider-network sends a broadcast gratuitous ARP request, then OVN on the same hypervisor responds to that ARP request without flooding it out onto the localnet physical network.
This behavior needs to change.

Try the following test from your VM (IP) on a provider network.

case 1)
arping -U -c 1 -I eth1 IP (request mode to update neighbors)
 -- You will see a response from OVN, and not flooded to provider net

case 2)
arping -D -c 1 -I eth1 IP (ARP Probing for Duplicate Address Detection)
-- you will see a response from OVN, and not flooded to provider net

case 3)
arping -A -c1 -I eth1 IP (answer mode to update neighbors)
-- you will not see a response from OVN, and you will see this flooded to provider net

In cases 1 and 2 above, I believe OVN is behaving wrongly for provider network,
In both of those cases, a) OVN should not respond, and b) It should flood the request onto
localnet

Cases 1) and 2) above are used in practice in the following ways.
Case 1) is used in VRRP for example -
https://tools.ietf.org/html/rfc3768 (section 6.4.1)

and Case 2) is used for ARP probing for duplicate address detection - see:
https://tools.ietf.org/html/rfc5227#section-2

But, the above usecase may not apply to the usecases OVN is targeted for and so, this
may not be a serious problem.

[Amitabha Biswas (azbiswas)]

Thanks for the examples and use cases, it makes the reason for desired change very clear. We should have this conversation in the ovs-dev channel as well.

[Richard Theis (rtheis)]

Hi Ramu, should a separate networking-ovn bug be opened for the use cases that you described?

[Ramu Ramamurthy (ramu-ramamurthy)]

@rtheis, we can fix the above cases separately - its different from the vm migration usecase
that is the subject of this bug.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/293419
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=72b68151facdd5e46e15160df990100d2bec0e82
Submitter: Jenkins
Branch: master

commit 72b68151facdd5e46e15160df990100d2bec0e82
Author: Richard Theis <email address hidden>
Date: Tue Mar 15 11:46:34 2016 -0500

    Vagrant: Synchronize clock for live migration

    Live migration can have problems if there is significant clock
    drift on the host. This patch set updates the vagrant deployment
    to install ntp on the VMs and to customize the VirtualBox
    --timesync-set-threshold for the VMs (see [1]).

    [1] https://www.virtualbox.org/manual/ch09.html#changetimesync

    Change-Id: I432767423b8a84e9fc0e9e74d28a0dfcbbe0f684
    Related-Bug: #1545897

[Richard Theis (rtheis)]

Hi Amitabha, do you have any updates on the OVN fix for this bug?

[Amitabha Biswas (azbiswas)]

http://patchwork.ozlabs.org/patch/605696/

[Richard Theis (rtheis)]

Thank you for the link Amitabha. Is this patch ready to test against this bug or should I wait?

[Amitabha Biswas (azbiswas)]

It's Ramu's patch, I would email him. The last I heard was that ARP type used for the GARP was under discussion.

[Ramu Ramamurthy (ramu-ramamurthy)]

The following patchset (v2) - to send garp on provider network,
is posted to the ovsdev mailing list for review:

https://patchwork.ozlabs.org/patch/615312/
https://patchwork.ozlabs.org/patch/615313/

[Kyle Mestery (mestery)]

Ramu's patch has now landed in OVS master [1]. I'll try to verify it works today.

[1] https://github.com/openvswitch/ovs/commit/0ee8aaf658dde99144c5145eff81edc88ee018da

[Kyle Mestery (mestery)]

I have verified this is fixed with Ramu's GARP patch in the OVS repository.

[Richard Theis (rtheis)]

Thank you Kyle.