OS-VIF: Interface missing from Route Object. (OVS)

[Mark Doffman]

When trying to boot and delete instances (Rally) saw this stack trace:

2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2075, in build_resources
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] yield resources
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1919, in build_and_run_instance
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] block_device_info=block_device_info)
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2658, in spawn
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] write_to_disk=True)
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4693, in _get_guest_xml
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] context)
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4525, in _get_guest_config
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] flavor, virt_type, self._host)
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 507, in get_config
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] vif_obj = os_vif_util.nova_to_osvif_vif(vif)
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 371, in nova_to_osvif_vif
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] vifobj = func(vif)
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 276, in _nova_to_osvif_vif_ovs
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] vif_name=_get_vif_name(vif))
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 239, in _get_vif_instance
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] network=_nova_to_osvif_network(vif['network']),
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 205, in _nova_to_osvif_network
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] subnets=_nova_to_osvif_subnets(network['subnets']))
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 191, in _nova_to_osvif_subnets
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] objects=[_nova_to_osvif_subnet(subnet) for subnet in subnets])
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 173, in _nova_to_osvif_subnet
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] routes=_nova_to_osvif_routes(subnet['routes']))
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 157, in _nova_to_osvif_routes
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] objects=[_nova_to_osvif_route(route) for route in routes])
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/network/os_vif_util.py", line 139, in _nova_to_osvif_route
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] interface=route['interface'])
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 307, in __init
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] setattr(self, key, kwargs[key])
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] field_value = field.coerce(self, name, value)
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 193, in coerce
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] return self.null(obj, attr)
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 171, in _null
2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] raise ValueError(("Field %s' cannot be None") % attr) 2016-08-12 15:21:58.859 38365 ERROR nova.compute.manager [instance: 47fbd50b-94ec-4395-884c-9131e0e3f335] ValueError: Fieldinterface' cannot be None

I cannot see where the 'interface' is set on the Nova 'Route' object. The object only seems to be created in one place. nova/network/neutronv2/api.py:2160

Version: Master
Networking: OVN ML2 driver. OVS VIF.

[Matt Riedemann (mriedem)]

Some notes:

(4:04:03 PM) mriedem: for neutron the route gets added here https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L2158
(4:04:06 PM) mriedem: but interface isn't set
(4:04:25 PM) patrickeast: mriedem: let me know what you'd like to have happen there, where its more info or a bug or whatever
(4:04:29 PM) mriedem: https://github.com/openstack/nova/blob/master/nova/network/model.py#L223
(4:04:44 PM) mriedem: set on the os-vif route object here https://github.com/openstack/nova/blob/master/nova/network/os_vif_util.py#L139
(4:04:53 PM) mriedem: but we don't ever set nova.network.model.Route.interface anywhere in nova
(4:04:57 PM) mriedem: AND
(4:05:04 PM) mriedem: i don't see anything in the neutron data model for Route.interface
(4:05:37 PM) mriedem: https://github.com/openstack/neutron/blob/master/neutron/objects/subnet.py#L67
(4:05:41 PM) mriedem: armax: HenryG: ^
(4:05:44 PM) mriedem: am i missing something?
(4:05:45 PM) doffm: os-vif object with presumably non-optional interface field here https://github.com/openstack/os-vif/blob/master/os_vif/objects/route.py#L28
(4:05:56 PM) mriedem: or was Route.interface just added to nova's nw model but never used?
(4:06:39 PM) mriedem: apparently we don't fail on this in the openstack CI b/c we aren't using subnets with routes
(4:06:40 PM) mriedem: http://logs.openstack.org/99/340599/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/3128550/logs/screen-n-cpu.txt.gz#_2016-08-12_17_08_55_248
(4:06:46 PM) mriedem: 'routes': []
(4:07:55 PM) mriedem: so looks to me like a lie was propagated to os-vif
(4:08:01 PM) mriedem: and the interface field should be nullable=True

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/355093

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/355098

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/355098
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1d57c1fd53e930b02c3ce0e9914f95ef68dd1f87
Submitter: Jenkins
Branch: master

commit 1d57c1fd53e930b02c3ce0e9914f95ef68dd1f87
Author: Matt Riedemann <email address hidden>
Date: Fri Aug 12 17:42:47 2016 -0400

    os-vif: do not set Route.interface if None

    The Route.interface field in os-vif is currently
    non-nullable, which means it must be something other
    than None if set.

    There is no code in nova that sets the interface field
    on the nova.network.model.Route object, so it defaults
    to None and is always set to None on the os-vif Route
    object, which fails with:

    "ValueError: Fieldinterface' cannot be None"

    We don't hit this in the gate because we don't test with
    subnets that have host_routes set.

    Change-Id: Ideadf1fc009b3ee496acb4696226a3cc9722fb29
    Closes-Bug: #1612812

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/355093
Committed: https://git.openstack.org/cgit/openstack/os-vif/commit/?id=b088d21b91dbfda0f18b1e4886f6aa01f2c72cbe
Submitter: Jenkins
Branch: master

commit b088d21b91dbfda0f18b1e4886f6aa01f2c72cbe
Author: Matt Riedemann <email address hidden>
Date: Fri Aug 12 17:32:51 2016 -0400

    Add a reminder to remove Route.interface field

    Nova never sets the Route.interface value to anything but
    None which fails with an error:

    "ValueError: Fieldinterface' cannot be None"

    This looks like a carry-over from the nova.network.model.Route
    class which has an interface field which is set to None by default
    but that field is never set to anything else in Nova, neither
    for nova-network or Neutron.

    Furthermore, it looks like 'interface' is not something that's
    in the Route data model in Neutron either.

    We don't hit this in the gate because the subnets we're testing
    with don't have host_routes set.

    The ValueError was fixed in Nova by not setting the attribute:

    1d57c1fd53e930b02c3ce0e9914f95ef68dd1f87

    This change adds a TODO to remove it in version 2.0 of the Route object.

    Change-Id: Ib25a79514fe4335f4df222c02fefc9da62fe04ce
    Closes-Bug: #1612812

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/os-vif 1.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.0.0b3 development milestone.