neutron

tempest test api.network;l3 agent can't delete HA-router

Bug #1510757 reported by zhang sheng
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Invalid
Medium
jessegler

Bug Description

I use tempest test my company's openstack environment.

# tox -eall -- tempest.api.network

When tox finished. Log in L3-agent always show:

2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent [-] Error while deleting router da4b28ce-33b1-4000-8609-a41a2ab8c982
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 344, in _safe_router_removed
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent self._router_removed(router_id)
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 362, in _router_removed
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent ri.delete(self)
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 364, in delete
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent super(HaRouter, self).delete(agent)
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 273, in delete
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent self.process(agent)
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 370, in process
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent)
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 359, in call
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent self.logger(e)
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/common/utils.py", line 356, in call
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 695, in process
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent self.routes_updated()
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 181, in routes_updated
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent instance = self._get_keepalived_instance()
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 131, in _get_keepalived_instance
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent return self.keepalived_manager.config.get_instance(self.ha_vr_id)
2015-10-27 19:32:03.281 23885 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'config'

I think the reason is tempest create and delete router too fast.
when l3-agent create ha-router,tempest delete the router,and neutron-server delete ha-interface

https://github.com/openstack/neutron/blob/master/neutron/agent/l3/ha_router.py#L79

class HaRouter can't initialized without ha-interface information,it just return without _init_keepalived_manager
When l3-agent delete router,it report AttributeError: 'NoneType'.

When l3 agent can't delete the router,l3 agent always fullsync with neutron-server every 30 seconds.
In controller, the neutron-server cpu always 70%..^-^

L3 agent should add a check before create HArouter.If it find ha-interface is none ,means router has been deleted in neutron-server.

Tags: l3-ha
Hong Hui Xiao (xiaohhui)
Changed in neutron:
assignee: nobody → Hong Hui Xiao (xiaohhui)
Revision history for this message
Hong Hui Xiao (xiaohhui) wrote :

Although it is reported as a tempest bug, it might happen in a real world.

Changed in neutron:
status: New → Confirmed
Revision history for this message
Hong Hui Xiao (xiaohhui) wrote :
Download full text (4.4 KiB)

I simulate that ha_interface is not in router. And get this logs in l3-agent all the time.

2015-10-29 10:20:57.169 7079 ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'config'
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 343, in call
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info return func(*args, **kwargs)
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 632, in process
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info self.routes_updated()
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 182, in routes_updated
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info instance = self._get_keepalived_instance()
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 132, in _get_keepalived_instance
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info return self.keepalived_manager.config.get_instance(self.ha_vr_id)
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info AttributeError: 'NoneType' object has no attribute 'config'
2015-10-29 10:20:57.169 7079 TRACE neutron.agent.l3.router_info
2015-10-29 10:20:57.174 7079 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'b2c8808d-8d21-44a2-85bf-b166507de1ae'
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 465, in _process_router_update
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent self._process_router_if_compatible(router)
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 412, in _process_router_if_compatible
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent self._process_updated_router(router)
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 426, in _process_updated_router
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent ri.process(self)
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 363, in process
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent super(HaRouter, self).process(agent)
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 346, in call
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent self.logger(e)
2015-10-29 10:20:57.174 7079 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
...

Read more...

OpenStack Infra (hudson-openstack)
Changed in neutron:
status: Confirmed → In Progress
Assaf Muller (amuller)
tags: added: l3-ha
Revision history for this message
LIU Yulong (dragon889) wrote :

Rally scenarios test create_and_delete_routers can also produce this error.

Revision history for this message
Ann Taraday (akamyshnikova) wrote :

Proposed change - https://review.openstack.org/#/c/240662/

Changed in neutron:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/240662
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Armando Migliaccio (armando-migliaccio)
Changed in neutron:
status: In Progress → Incomplete
assignee: Hong Hui Xiao (xiaohhui) → nobody
Revision history for this message
Ann Taraday (akamyshnikova) wrote :

This issue can be still reproduced after rally tests execution http://paste.openstack.org/show/494161/

Changed in neutron:
status: Incomplete → Confirmed
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

This bug is > 180 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

Changed in neutron:
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
Revision history for this message
Alejandro Comisario (alejandro-f) wrote :
Download full text (8.0 KiB)

Happening on mitaka ubuntu packages as of today.
And also, when you create the router, you also get on l3-agent :

2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'config'
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 379, in call
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 1002, in process
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info self.process_external(agent)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 515, in process_external
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info super(DvrLocalRouter, self).process_external(agent)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 827, in process_external
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info self._process_external_gateway(ex_gw_port, agent.pd)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 701, in _process_external_gateway
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info self.external_gateway_added(ex_gw_port, interface_name)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_edge_ha_router.py", line 61, in external_gateway_added
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info ex_gw_port, interface_name)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_edge_router.py", line 45, in external_gateway_added
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info self.routes_updated([], self.router['routes'])
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 203, in routes_updated
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info instance = self._get_keepalived_instance()
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/ha_router.py", line 150, in _get_keepalived_instance
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info return self.keepalived_manager.config.get_instance(self.ha_vr_id)
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info AttributeError: 'NoneType' object has no attribute 'config'
2017-04-03 12:49:52.153 13617 ERROR neutron.agent.l3.router_info
2017-04-03 12:49:52.157 13617 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '5875dcd3-58d7-4e11-...

Read more...

Changed in neutron:
status: Expired → Confirmed
Revision history for this message
Alejandro Comisario (alejandro-f) wrote :

This happens on a real world cloud whitout tempest

Revision history for this message
Deepak Mourya (mourya007) wrote :

hey, i couldn't reproduce this error while running tox -eall -- tempest.api.network . Is this bug still occurring ?

jessegler (je808k)
Changed in neutron:
assignee: nobody → jessegler (je808k)
Revision history for this message
Brian Haley (brian-haley) wrote :

Closing as I don't think this happens any more.

Changed in neutron:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.