neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_legacy_router_lifecycle* fail

Bug #1795548 reported by Brian Haley
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Brian Haley

Bug Description

I keep seeing these two tests fail:

http://logs.openstack.org/05/606205/2/check/neutron-functional/66c9475/logs/testr_results.html.gz

Example stack trace:

Traceback (most recent call last):
  File "neutron/tests/base.py", line 137, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/functional/agent/l3/test_legacy_router.py", line 85, in test_legacy_router_lifecycle
    self._router_lifecycle(enable_ha=False, dual_stack=True)
  File "neutron/tests/functional/agent/l3/framework.py", line 302, in _router_lifecycle
    self._assert_onlink_subnet_routes(router, ip_versions)
  File "neutron/tests/functional/agent/l3/framework.py", line 526, in _assert_onlink_subnet_routes
    namespace=ns_name)
  File "neutron/agent/linux/ip_lib.py", line 1030, in get_routing_table
    return list(privileged.get_routing_table(ip_version, namespace))
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
    raise exc_type(*result[2])
KeyError

I think the problem is that in the privsep get_routing_table code, one of the IPv6 routes does not have an integer in the route['oif'] element, it is None, raising the KeyError.

For example, here is a list of IPv6 routes on my local system:

--> ip -6 r
2601:18f:700:c12d::/64 dev enp0s31f6 proto ra metric 100 pref medium
fe80::/64 dev enp0s31f6 proto kernel metric 256 pref medium
fe80::/64 dev tun0 proto kernel metric 256 pref medium
default via fe80::9ade:d0ff:fe25:7710 dev enp0s31f6 proto static metric 100 pref medium

But a little test program I wrote shows iproute2 returns no 'oif':

dst: fe80::/64
gateway: None
oif: None
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./getroute.py", line 78, in foo
    routes = list(get_routing_table(6))
  File "./getroute.py", line 50, in get_routing_table
    print 'interface: %s' % ipdb_interfaces[route['oif']]['ifname']
KeyError: None

Digging further, since there are two routes to fe80::/64, it's returned differently from pyroute2:

{'metrics': {}, 'dst_len': 64, 'family': 10, 'proto': 2, 'tos': 0, 'dst': 'fe80::/64', 'pref': '00', 'ipdb_priority': 0, 'priority': 256, 'flags': 0, 'encap': {}, 'src_len': 0, 'table': 254, 'multipath': ({'oif': 2, 'family': 10}, {'oif': 6, 'dst_len': 64, 'family': 10, 'proto': 2, 'tos': 0, 'pref': '00', 'priority': 256, 'flags': 0, 'encap': {}, 'src_len': 0, 'table': 254, 'type': 1, 'scope': 0}), 'type': 1, 'scope': 0, 'ipdb_scope': 'system'}

So it looks like we need to parse this 'multipath' element, but there are two items in the list, so we have to parse them both.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/607093

Changed in neutron:
status: New → In Progress
tags: added: functional-tests gate-failure
Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

It's probably related with new version of pyroute2 (0.5.3) which was released 30.09

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Brian Haley (<email address hidden>) on branch: master
Review: https://review.openstack.org/607093

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/607009
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=00de8f9a9e36bbca6ef7f0c17c2e6f74a144a358
Submitter: Zuul
Branch: master

commit 00de8f9a9e36bbca6ef7f0c17c2e6f74a144a358
Author: Brian Haley <email address hidden>
Date: Mon Oct 1 13:49:32 2018 -0400

    Do not fail deleting namespace if it does not exist

    Note: this is a squash of two changes since they are
    dependent on each other, and are currently blocking
    the gate queue.

    Sometimes cleanup methods are failing in the check and
    gate queues trying to delete non-existing namespaces.
    Since they could have been deleted asynchronously, don't
    raise if the failure is "No such file or directory" since
    the system is in the intended state.

    Cleaned-up the DHCP agent to longer check for existence
    first, and the tests to longer mock-out the namespace
    exists check.

    Fix test_legacy_router_lifecycle failures

    Multi-path routes returned via the pyroute2 library have
    their outgoing interfaces in the 'multipath' dictionary
    element, not in the route dictionary. In that case return
    all the multipath routes correctly.

    Change-Id: I5415cb3a88ff2640a19598a1fcb2278388815343
    Closes-bug: #1795482
    Closes-bug: #1795548

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.0.0b1

This issue was fixed in the openstack/neutron 14.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/753255

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/753256

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/753256
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5ee6e5bb31179acf6e86357b0b0b34ef051af161
Submitter: Zuul
Branch: stable/queens

commit 5ee6e5bb31179acf6e86357b0b0b34ef051af161
Author: Brian Haley <email address hidden>
Date: Mon Oct 1 13:49:32 2018 -0400

    Do not fail deleting namespace if it does not exist

    Note: this is a squash of two changes since they are
    dependent on each other, and are currently blocking
    the gate queue.

    Sometimes cleanup methods are failing in the check and
    gate queues trying to delete non-existing namespaces.
    Since they could have been deleted asynchronously, don't
    raise if the failure is "No such file or directory" since
    the system is in the intended state.

    Cleaned-up the DHCP agent to longer check for existence
    first, and the tests to longer mock-out the namespace
    exists check.

    Fix test_legacy_router_lifecycle failures

    Multi-path routes returned via the pyroute2 library have
    their outgoing interfaces in the 'multipath' dictionary
    element, not in the route dictionary. In that case return
    all the multipath routes correctly.

    Change-Id: I5415cb3a88ff2640a19598a1fcb2278388815343
    Closes-bug: #1795482
    Closes-bug: #1795548

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/753255
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7e90c5fb73612d73b393a7aa0517fb17d81faf77
Submitter: Zuul
Branch: stable/rocky

commit 7e90c5fb73612d73b393a7aa0517fb17d81faf77
Author: Brian Haley <email address hidden>
Date: Mon Oct 1 13:49:32 2018 -0400

    Do not fail deleting namespace if it does not exist

    Note: this is a squash of two changes since they are
    dependent on each other, and are currently blocking
    the gate queue.

    Sometimes cleanup methods are failing in the check and
    gate queues trying to delete non-existing namespaces.
    Since they could have been deleted asynchronously, don't
    raise if the failure is "No such file or directory" since
    the system is in the intended state.

    Cleaned-up the DHCP agent to longer check for existence
    first, and the tests to longer mock-out the namespace
    exists check.

    Fix test_legacy_router_lifecycle failures

    Multi-path routes returned via the pyroute2 library have
    their outgoing interfaces in the 'multipath' dictionary
    element, not in the route dictionary. In that case return
    all the multipath routes correctly.

    Change-Id: I5415cb3a88ff2640a19598a1fcb2278388815343
    Closes-bug: #1795482
    Closes-bug: #1795548

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron queens-eol

This issue was fixed in the openstack/neutron queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron rocky-eol

This issue was fixed in the openstack/neutron rocky-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.