neutron

  1. Bug #1662804
  2. Activity log

Activity log for bug #1662804

Date Who What changed Old value New value Message
2017-02-08 08:27:46 venkata anil bug added bug
2017-02-08 08:27:52 venkata anil neutron: assignee venkata anil (anil-venkata)
2017-02-08 13:26:39 Jakub Libosvar tags l3-ha
2017-02-08 15:51:20 OpenStack Infra neutron: status New In Progress
2017-02-09 10:43:53 Ann Taraday neutron: importance Undecided Medium
2017-03-17 21:15:14 OpenStack Infra neutron: assignee venkata anil (anil-venkata) Brian Haley (brian-haley)
2017-03-22 05:11:16 OpenStack Infra neutron: assignee Brian Haley (brian-haley) venkata anil (anil-venkata)
2017-03-30 19:03:22 OpenStack Infra neutron: status In Progress Fix Released
2017-04-02 09:04:14 OpenStack Infra tags l3-ha in-stable-ocata l3-ha
2017-05-23 14:33:34 Edward Hope-Morley bug task added neutron (Ubuntu)
2017-05-23 14:33:44 Edward Hope-Morley neutron (Ubuntu): status New Fix Released
2017-05-23 14:33:56 Edward Hope-Morley nominated for series Ubuntu Xenial
2017-05-23 14:34:09 Edward Hope-Morley bug task added cloud-archive
2017-05-23 14:34:22 Edward Hope-Morley nominated for series cloud-archive/mitaka
2017-05-23 14:34:22 Edward Hope-Morley nominated for series cloud-archive/newton
2017-05-23 21:20:19 Edward Hope-Morley cloud-archive: status New Fix Released
2017-05-23 21:20:35 Edward Hope-Morley summary Agent is failing to process HA router if initialize() fails [SRU] Agent is failing to process HA router if initialize() fails
2017-05-23 21:21:04 Edward Hope-Morley description When HA router initialize() function fails for some reason(rabbitmq restart or no ha_port), keepalived_manager or KeepalivedInstance won't be configured. In this case, _process_router_if_compatible fails with exception, then _resync_router(update) will again try to process this router in loop. As we try initialize() only once(which was failed), retry of _process_router_if_compatible will always fail(no keepalived manager or instance) and router is never configured(see below trace). 2017-02-06 18:34:18.539 26120 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-114a72fe-02ae-4b87-a2e7-70f962df0951', 'ip', '-o', 'link', 'show', 'qr-e6 3406e1-e7'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101 2017-02-06 18:34:18.544 26120 DEBUG neutron.agent.linux.utils [-] Command: ['ip', 'netns', 'exec', u'qrouter-114a72fe-02ae-4b87-a2e7-70f962df0951', 'ip', '-o', 'link', 'show', u'qr-e63406e1-e7'] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:156 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'get_process' 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info Traceback (most recent call last): 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 359, in call 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info return func(*args, **kwargs) 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 744, in process 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd) 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 394, in _process_internal_ports 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info self.internal_network_added(p) 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 275, in internal_network_added 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info self._disable_ipv6_addressing_on_interface(interface_name) 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 235, in _disable_ipv6_addressing_on_interface 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info if self._should_delete_ipv6_lladdr(ipv6_lladdr): 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 217, in _should_delete_ipv6_lladdr 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info if manager.get_process().active: 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info AttributeError: 'NoneType' object has no attribute 'get_process' 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '114a72fe-02ae-4b87-a2e7-70f962df0951' 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 506, in _process_router_update 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 445, in _process_router_if_compatible 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._process_updated_router(router) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 459, in _process_updated_router 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent ri.process(self) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 377, in process 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 362, in call 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self.logger(e) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 359, in call 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent return func(*args, **kwargs) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 744, in process 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._process_internal_ports(agent.pd) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 394, in _process_internal_ports 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self.internal_network_added(p) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 275, in internal_network_added 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._disable_ipv6_addressing_on_interface(interface_name) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 235, in _disable_ipv6_addressing_on_interface 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent if self._should_delete_ipv6_lladdr(ipv6_lladdr): 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 217, in _should_delete_ipv6_lladdr 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent if manager.get_process().active: 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'get_process' [Impact] This patch resolves, amongst other things, issues with a create and delete router request race condition when using l3 HA. At the time of backport this patch is already available from Ocata onwards and has been verified as sufficiently minimal and safe for backport to Newton and Mitaka. Essentially the error case is a result of an incorrectly intialised router update action being executed without proper checks and this patch fixes this. [Test Case] * Deploy Openstack Mitaka - http://pastebin.ubuntu.com/24637244/ - with neutron-l3-agent configured to provide HA (vrrp) routers. * Repeatedly create and delete routers in rapid succession and check that the l3 agent does not go into an infinite error loop i.e. run http://pastebin.ubuntu.com/24634950/ and run do tail -F /var/log/neutron/neutron-l3-agent.log on all units of l3 agent. Also check that qrouter- namepspaces are not stacking up. For Mitaka I typically hit the error after ~20 create/deletes. [Regression Potential] * I do not envisage any regression potential from this patch. ==== When HA router initialize() function fails for some reason(rabbitmq restart or no ha_port), keepalived_manager or KeepalivedInstance won't be configured. In this case, _process_router_if_compatible fails with exception, then _resync_router(update) will again try to process this router in loop. As we try initialize() only once(which was failed), retry of _process_router_if_compatible will always fail(no keepalived manager or instance) and router is never configured(see below trace). 2017-02-06 18:34:18.539 26120 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-114a72fe-02ae-4b87-a2e7-70f962df0951', 'ip', '-o', 'link', 'show', 'qr-e6 3406e1-e7'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101 2017-02-06 18:34:18.544 26120 DEBUG neutron.agent.linux.utils [-] Command: ['ip', 'netns', 'exec', u'qrouter-114a72fe-02ae-4b87-a2e7-70f962df0951', 'ip', '-o', 'link', 'show', u'qr-e63406e1-e7'] Exit code: 0  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:156 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'get_process' 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info Traceback (most recent call last): 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 359, in call 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info return func(*args, **kwargs) 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 744, in process 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd) 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 394, in _process_internal_ports 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info self.internal_network_added(p) 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 275, in internal_network_added 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info self._disable_ipv6_addressing_on_interface(interface_name) 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 235, in _disable_ipv6_addressing_on_interface 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info if self._should_delete_ipv6_lladdr(ipv6_lladdr): 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 217, in _should_delete_ipv6_lladdr 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info if manager.get_process().active: 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info AttributeError: 'NoneType' object has no attribute 'get_process' 2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '114a72fe-02ae-4b87-a2e7-70f962df0951' 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 506, in _process_router_update 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 445, in _process_router_if_compatible 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._process_updated_router(router) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 459, in _process_updated_router 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent ri.process(self) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 377, in process 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 362, in call 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self.logger(e) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 359, in call 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent return func(*args, **kwargs) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 744, in process 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._process_internal_ports(agent.pd) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 394, in _process_internal_ports 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self.internal_network_added(p) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 275, in internal_network_added 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._disable_ipv6_addressing_on_interface(interface_name) 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 235, in _disable_ipv6_addressing_on_interface 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent if self._should_delete_ipv6_lladdr(ipv6_lladdr): 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 217, in _should_delete_ipv6_lladdr 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent if manager.get_process().active: 2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'get_process'
2017-05-23 21:22:38 Edward Hope-Morley tags in-stable-ocata l3-ha in-stable-ocata l3-ha sts sts-sru-needed
2017-05-23 21:23:52 Edward Hope-Morley attachment added lp1662804-newton.debdiff https://bugs.launchpad.net/neutron/+bug/1662804/+attachment/4882150/+files/lp1662804-newton.debdiff
2017-05-23 21:25:51 Edward Hope-Morley attachment added lp1662804-mitaka.debdiff https://bugs.launchpad.net/neutron/+bug/1662804/+attachment/4882151/+files/lp1662804-mitaka.debdiff
2017-05-25 21:21:42 Ihar Hrachyshka neutron: importance Medium High
2017-05-26 02:58:26 OpenStack Infra tags in-stable-ocata l3-ha sts sts-sru-needed in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed
2017-05-30 07:59:57 Edward Hope-Morley nominated for series Ubuntu Yakkety
2017-05-30 08:18:45 Edward Hope-Morley attachment added lp1662804-yakkety-newton.debdiff https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1662804/+attachment/4886119/+files/lp1662804-yakkety-newton.debdiff
2017-05-31 10:40:39 Edward Hope-Morley bug added subscriber Ubuntu Sponsors Team
2017-06-05 10:33:40 James Page bug task added neutron (Ubuntu Xenial)
2017-06-05 10:33:50 James Page bug task added neutron (Ubuntu Yakkety)
2017-06-05 10:34:04 James Page bug task added cloud-archive/newton
2017-06-05 10:34:17 James Page bug task added cloud-archive/mitaka
2017-06-06 14:12:34 Edward Hope-Morley cloud-archive/mitaka: assignee Edward Hope-Morley (hopem)
2017-06-06 14:12:37 Edward Hope-Morley cloud-archive/newton: assignee Edward Hope-Morley (hopem)
2017-06-06 14:12:40 Edward Hope-Morley neutron (Ubuntu Xenial): assignee Edward Hope-Morley (hopem)
2017-06-06 14:12:46 Edward Hope-Morley neutron (Ubuntu Yakkety): assignee Edward Hope-Morley (hopem)
2017-06-07 08:19:21 James Page removed subscriber Ubuntu Sponsors Team
2017-06-07 08:19:28 James Page bug added subscriber Ubuntu Stable Release Updates Team
2017-06-07 09:14:13 James Page neutron (Ubuntu Xenial): status New In Progress
2017-06-07 09:14:16 James Page cloud-archive/mitaka: status New In Progress
2017-06-07 10:36:32 James Page neutron (Ubuntu Yakkety): status New In Progress
2017-06-07 10:36:34 James Page cloud-archive/newton: status New In Progress
2017-06-22 10:45:40 Łukasz Zemczak neutron (Ubuntu Xenial): status In Progress Fix Committed
2017-06-22 10:45:46 Łukasz Zemczak bug added subscriber SRU Verification
2017-06-22 10:45:50 Łukasz Zemczak tags in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-needed
2017-06-22 10:50:46 Łukasz Zemczak neutron (Ubuntu Yakkety): status In Progress Fix Committed
2017-06-27 14:13:16 James Page cloud-archive/newton: status In Progress Fix Committed
2017-06-27 14:13:21 James Page tags in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-needed in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-needed verification-newton-needed
2017-06-30 08:43:45 James Page cloud-archive/mitaka: status In Progress Fix Committed
2017-06-30 08:43:48 James Page tags in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-needed verification-newton-needed in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-mitaka-needed verification-needed verification-newton-needed
2017-07-03 14:45:10 Edward Hope-Morley tags in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-mitaka-needed verification-needed verification-newton-needed in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-done-yakkety verification-mitaka-needed verification-needed verification-newton-needed
2017-07-03 17:19:17 Edward Hope-Morley tags in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-done-yakkety verification-mitaka-needed verification-needed verification-newton-needed in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-done-yakkety verification-mitaka-needed verification-needed verification-newton-done
2017-07-04 11:25:33 Edward Hope-Morley tags in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-done-yakkety verification-mitaka-needed verification-needed verification-newton-done in-stable-newton in-stable-ocata l3-ha sts sts-sru-needed verification-done-xenial verification-done-yakkety verification-mitaka-done verification-newton-done
2017-07-05 00:38:48 Launchpad Janitor neutron (Ubuntu Yakkety): status Fix Committed Fix Released
2017-07-05 00:38:56 Chris Halse Rogers removed subscriber Ubuntu Stable Release Updates Team
2017-07-05 00:54:52 Launchpad Janitor neutron (Ubuntu Xenial): status Fix Committed Fix Released
2017-07-05 10:17:28 James Page cloud-archive/newton: status Fix Committed Fix Released
2017-07-05 10:20:01 James Page cloud-archive/mitaka: status Fix Committed Fix Released