neutron

functional: test_legacy_router_ns_rebuild is unstable

Bug #1604370 reported by Jakub Libosvar on 2016-07-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Terry Wilson

Bug Description

logstash query: http://goo.gl/xfeSlj
13 hits in last 7 days

Example of failure: http://logs.openstack.org/01/312401/8/check/gate-neutron-dsvm-functional/6aedd17/logs/dsvm-functional-logs/neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_legacy_router_ns_rebuild.txt.gz#_2016-07-19_10_45_28_248

Traceback:
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info [req-8e9eaf17-43d7-46e4-8179-84cb18fa106e - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "qg-85bf65ee-a2"
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/common/utils.py", line 239, in call
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 1035, in process
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info self.process_external(agent)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 817, in process_external
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info self._process_external_gateway(ex_gw_port, agent.pd)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 695, in _process_external_gateway
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info self.external_gateway_added(ex_gw_port, interface_name)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 660, in external_gateway_added
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info ex_gw_port, interface_name, self.ns_name, preserve_ips)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 606, in _external_gateway_added
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info self._plug_external_gateway(ex_gw_port, interface_name, ns_name)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/l3/router_info.py", line 573, in _plug_external_gateway
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info mtu=ex_gw_port.get('mtu'))
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/linux/interface.py", line 251, in plug
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info bridge, namespace, prefix, mtu)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/linux/interface.py", line 344, in plug_new
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info ns_dev.link.set_address(mac_address)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 496, in set_address
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info self._as_root([], ('set', self.name, 'address', mac_address))
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 362, in _as_root
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info use_root_namespace=use_root_namespace)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info log_fail_as_error=log_fail_as_error)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info File "neutron/agent/linux/utils.py", line 138, in execute
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info raise RuntimeError(msg)
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "qg-85bf65ee-a2"
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info
2016-07-19 10:45:28.248 27850 ERROR neutron.agent.l3.router_info

Tags:

Jakub Libosvar (libosvar) on 2016-07-19

tags:

added: functional-tests

Miguel Lavalle (minsel) on 2016-07-20

tags:

added: l3-ipam-dhcp

Miguel Lavalle (minsel) on 2016-07-21

Changed in neutron:
importance:	Undecided → High

Revision history for this message

Assaf Muller (amuller) wrote on 2016-07-21:

It looks like a race between OVS adding the interface and querying for its existence via the Linux network stack. Likely resolved by https://review.openstack.org/#/c/344859/.

tags:	added: gate-failure
Changed in neutron:
status:	New → Confirmed

Revision history for this message

Miguel Lavalle (minsel) wrote on 2016-07-21:

Adding a slightly better formatted Traceback to facilitate debugging: http://paste.openstack.org/show/539312/

Revision history for this message

Miguel Lavalle (minsel) wrote on 2016-07-22:

Yes, it seems likely to be resolved by https://review.openstack.org/#/c/344859/. Will keep an eye on that patchset

Armando Migliaccio (armando-migliaccio) on 2016-07-25

Changed in neutron:
status:	Confirmed → In Progress
assignee:	nobody → Terry Wilson (otherwiseguy)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-07-29: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/344859
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=11dc21d3a6d765f7bcc95548b55ff13c4397c2e7
Submitter: Jenkins
Branch: master

commit 11dc21d3a6d765f7bcc95548b55ff13c4397c2e7
Author: Terry Wilson <email address hidden>
Date: Fri Apr 22 08:55:11 2016 -0500

Wait for vswitchd to add interfaces in native ovsdb

    ovs-vsctl, unless --no-wait is passed, will wait until ovs-vswitchd
    has reacted to a successful transaction. This patch implements
    the same logic, waiting for next_cfg to be incremented and checking
    that any added interfaces have actually been assigned ofports.

    Closes-Bug: #1604816
    Closes-Bug: #1604370
    Related-Bug: #1604115
    Change-Id: I638b82c13394f150c0bd23301285bd3375e66139

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-08-01: Fix proposed to neutron (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/349416

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-08-03: Fix merged to neutron (stable/mitaka)

Reviewed: https://review.openstack.org/349416
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a98b6ebe48f4ecda689646d05050e156bc28be4d
Submitter: Jenkins
Branch: stable/mitaka

commit a98b6ebe48f4ecda689646d05050e156bc28be4d
Author: Terry Wilson <email address hidden>
Date: Fri Apr 22 08:55:11 2016 -0500

Wait for vswitchd to add interfaces in native ovsdb

    Closes-Bug: #1604816
    Closes-Bug: #1604370
    Related-Bug: #1604115
    Change-Id: I638b82c13394f150c0bd23301285bd3375e66139
    (cherry picked from commit 11dc21d3a6d765f7bcc95548b55ff13c4397c2e7)