When HA router initialize() function fails for some reason(rabbitmq restart or no ha_port), keepalived_manager or KeepalivedInstance won't be configured. In this case, _process_router_if_compatible fails with exception, then _resync_router(update) will again try to process this router in loop. As we try initialize() only once(which was failed), retry of _process_router_if_compatible will always fail(no keepalived manager or instance) and router is never configured(see below trace).
2017-02-06 18:34:18.539 26120 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-114a72fe-02ae-4b87-a2e7-70f962df0951', 'ip', '-o', 'link', 'show', 'qr-e6
3406e1-e7'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:101
2017-02-06 18:34:18.544 26120 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', 'netns', 'exec', u'qrouter-114a72fe-02ae-4b87-a2e7-70f962df0951', 'ip', '-o', 'link', 'show', u'qr-e63406e1-e7']
Exit code: 0
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:156
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'get_process'
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 359, in call
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 744, in process
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd)
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 394, in _process_internal_ports
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info self.internal_network_added(p)
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 275, in internal_network_added
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info self._disable_ipv6_addressing_on_interface(interface_name)
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 235, in _disable_ipv6_addressing_on_interface
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info if self._should_delete_ipv6_lladdr(ipv6_lladdr):
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 217, in _should_delete_ipv6_lladdr
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info if manager.get_process().active:
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info AttributeError: 'NoneType' object has no attribute 'get_process'
2017-02-06 18:34:18.544 26120 ERROR neutron.agent.l3.router_info
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '114a72fe-02ae-4b87-a2e7-70f962df0951'
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 506, in _process_router_update
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 445, in _process_router_if_compatible
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 459, in _process_updated_router
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent ri.process(self)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 377, in process
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent super(HaRouter, self).process(agent)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 362, in call
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self.logger(e)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 359, in call
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 744, in process
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._process_internal_ports(agent.pd)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 394, in _process_internal_ports
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self.internal_network_added(p)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 275, in internal_network_added
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent self._disable_ipv6_addressing_on_interface(interface_name)
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 235, in _disable_ipv6_addressing_on_interface
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent if self._should_delete_ipv6_lladdr(ipv6_lladdr):
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 217, in _should_delete_ipv6_lladdr
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent if manager.get_process().active:
2017-02-06 18:34:18.549 26120 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'get_process'
When HA router initialize() function fails for some reason(rabbitmq restart or no ha_port), keepalived_manager or KeepalivedInstance won't be configured. In this case, _process_ router_ if_compatible fails with exception, then _resync_ router( update) will again try to process this router in loop. As we try initialize() only once(which was failed), retry of _process_ router_ if_compatible will always fail(no keepalived manager or instance) and router is never configured(see below trace).
2017-02-06 18:34:18.539 26120 DEBUG neutron. agent.linux. utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter- 114a72fe- 02ae-4b87- a2e7-70f962df09 51', 'ip', '-o', 'link', 'show', 'qr-e6 rootwrap_ daemon /usr/lib/ python2. 7/site- packages/ neutron/ agent/linux/ utils.py: 101 agent.linux. utils [-] 114a72fe- 02ae-4b87- a2e7-70f962df09 51', 'ip', '-o', 'link', 'show', u'qr-e63406e1-e7'] python2. 7/site- packages/ neutron/ agent/linux/ utils.py: 156 agent.l3. router_ info [-] 'NoneType' object has no attribute 'get_process' agent.l3. router_ info Traceback (most recent call last): agent.l3. router_ info File "/usr/lib/ python2. 7/site- packages/ neutron/ common/ utils.py" , line 359, in call agent.l3. router_ info return func(*args, **kwargs) agent.l3. router_ info File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ router_ info.py" , line 744, in process agent.l3. router_ info self._process_ internal_ ports(agent. pd) agent.l3. router_ info File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ router_ info.py" , line 394, in _process_ internal_ ports agent.l3. router_ info self.internal_ network_ added(p) agent.l3. router_ info File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ ha_router. py", line 275, in internal_ network_ added agent.l3. router_ info self._disable_ ipv6_addressing _on_interface( interface_ name) agent.l3. router_ info File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ ha_router. py", line 235, in _disable_ ipv6_addressing _on_interface agent.l3. router_ info if self._should_ delete_ ipv6_lladdr( ipv6_lladdr) : agent.l3. router_ info File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ ha_router. py", line 217, in _should_ delete_ ipv6_lladdr agent.l3. router_ info if manager. get_process( ).active: agent.l3. router_ info AttributeError: 'NoneType' object has no attribute 'get_process' agent.l3. router_ info agent.l3. agent [-] Failed to process compatible router '114a72fe- 02ae-4b87- a2e7-70f962df09 51' agent.l3. agent Traceback (most recent call last): agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ agent.py" , line 506, in _process_ router_ update agent.l3. agent self._process_ router_ if_compatible( router) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ agent.py" , line 445, in _process_ router_ if_compatible agent.l3. agent self._process_ updated_ router( router) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ agent.py" , line 459, in _process_ updated_ router agent.l3. agent ri.process(self) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ ha_router. py", line 377, in process agent.l3. agent super(HaRouter, self).process( agent) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ common/ utils.py" , line 362, in call agent.l3. agent self.logger(e) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ oslo_utils/ excutils. py", line 204, in __exit__ agent.l3. agent six.reraise( self.type_ , self.value, self.tb) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ common/ utils.py" , line 359, in call agent.l3. agent return func(*args, **kwargs) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ router_ info.py" , line 744, in process agent.l3. agent self._process_ internal_ ports(agent. pd) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ router_ info.py" , line 394, in _process_ internal_ ports agent.l3. agent self.internal_ network_ added(p) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ ha_router. py", line 275, in internal_ network_ added agent.l3. agent self._disable_ ipv6_addressing _on_interface( interface_ name) agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ ha_router. py", line 235, in _disable_ ipv6_addressing _on_interface agent.l3. agent if self._should_ delete_ ipv6_lladdr( ipv6_lladdr) : agent.l3. agent File "/usr/lib/ python2. 7/site- packages/ neutron/ agent/l3/ ha_router. py", line 217, in _should_ delete_ ipv6_lladdr agent.l3. agent if manager. get_process( ).active: agent.l3. agent AttributeError: 'NoneType' object has no attribute 'get_process'
3406e1-e7'] execute_
2017-02-06 18:34:18.544 26120 DEBUG neutron.
Command: ['ip', 'netns', 'exec', u'qrouter-
Exit code: 0
execute /usr/lib/
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.544 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.
2017-02-06 18:34:18.549 26120 ERROR neutron.