vif plugging fails with ovn when trying to set mtu on qvb device that does not exist

[Matt Riedemann]

Richard Theis reported the regression here:

https://review.openstack.org/#/c/370681/9/nova/virt/libvirt/vif.py

Shown here:

http://logs.openstack.org/75/371175/1/check/gate-tempest-dsvm-networking-ovn/7e52927/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-09-16_11_20_39_627

2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [req-78de89a8-2053-42d9-899f-04aa2b8f25c7 tempest-FloatingIPsTestJSON-1913520192 tempest-FloatingIPsTestJSON-1913520192] [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Instance failed to spawn
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Traceback (most recent call last):
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/compute/manager.py", line 2078, in _build_resources
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] yield resources
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/compute/manager.py", line 1920, in _build_and_run_instance
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] block_device_info=block_device_info)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2583, in spawn
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] post_xml_callback=gen_confdrive)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4814, in _create_domain_and_network
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] self.plug_vifs(instance, network_info)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 684, in plug_vifs
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] self.vif_driver.plug(instance, vif)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 817, in plug
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] self._plug_os_vif(instance, vif_obj, vif)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/virt/libvirt/vif.py", line 799, in _plug_os_vif
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] linux_net._set_device_mtu(veth, mtu)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/network/linux_net.py", line 1237, in _set_device_mtu
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] check_exit_code=[0, 2, 254])
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/utils.py", line 295, in execute
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] return RootwrapProcessHelper().execute(*cmd, **kwargs)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/opt/stack/new/nova/nova/utils.py", line 178, in execute
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] return processutils.execute(*cmd, **kwargs)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] cmd=sanitized_cmd)
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] ProcessExecutionError: Unexpected error while running command.
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link set qvb00b6a38f-60 mtu 1442
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Exit code: 1
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stdout: u''
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221] Stderr: u'Cannot find device "qvb00b6a38f-60"\n'
2016-09-16 11:21:29.140 15810 ERROR nova.compute.manager [instance: fa58495c-1f29-4149-9efb-2bbf7624d221]

The nova team isn't sure if ovn should have those devices or not by the time we've plugged the vif, or if ovn is a special case and those just won't exist.

We're going to add device exists checks around that code to be safe and get that into newton-rc2 and move forward unless someone more that knows more about OVN can tell us otherwise.

[Daniel Berrange (berrange)]

The nova call to set_device_mtu is protected by this condition:

        if (isinstance(vif, os_vif.objects.vif.VIFBridge) and
            hasattr(vif, "port_profile") and

The 'vif' object here comes from nova.network.os_vif_util.nova_to_osvif_vif().

nova_to_osvif_vif() calls one of two methods:

_nova_to_osvif_vif_bridge
_nova_to_osvif_vif_ovs

Only the _nova_to_osvif_vif_ovs() method sets the port_profile attribute AFAICS, and that always sets the plugin=ovs.

The vif_plug_ovs impl will call either _plug_bridge or _plug_bridge_windows(). _plug_bridge() will always create the veth pair, but _plugin_bridge_windows() will not.

So, AFAICT, the only way we should get that Nova error is if we are running on windows ?!?!?

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22mtu%201442%5C%22%20AND%20message%3A%5C%22_set_device_mtu%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

[Daniel Berrange (berrange)]

Actually I spot the mistake - this check:

        if (isinstance(vif, os_vif.objects.vif.VIFBridge) and

is wrong because 'VIFOpenVSwitch' is a subclass of VIFBridge. So this was not filtering out the VIFOpenVSwitch scenario correctly. We need an exact class match here against VIFBridge, not merely an instance check

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/371543

[Richard Theis (rtheis)]

OVN does not create those devices.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/371543
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fc0e281743917dbda25fb1911500439abed192ca
Submitter: Jenkins
Branch: master

commit fc0e281743917dbda25fb1911500439abed192ca
Author: John Garbutt <email address hidden>
Date: Fri Sep 16 14:51:10 2016 +0100

    Stop ovn networking failing on mtu

    The type check needed to be more specific. There are some subclasses
    that don't create the devices we were trying to set the mtu on.

    Change-Id: Icc628a2dbde137d320fb78ad45b2ee0f7b5775fa
    Closes-Bug: #1624383

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/371836

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/371836
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dbca6efcc47930bff37579493e2429cbd52bcd42
Submitter: Jenkins
Branch: stable/newton

commit dbca6efcc47930bff37579493e2429cbd52bcd42
Author: John Garbutt <email address hidden>
Date: Fri Sep 16 14:51:10 2016 +0100

    Stop ovn networking failing on mtu

    The type check needed to be more specific. There are some subclasses
    that don't create the devices we were trying to set the mtu on.

    Change-Id: Icc628a2dbde137d320fb78ad45b2ee0f7b5775fa
    Closes-Bug: #1624383
    (cherry picked from commit fc0e281743917dbda25fb1911500439abed192ca)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.