AttributeError in l3-agent changing HA router to non-HA

[Slawek Kaplonski]

In case when router is migrated from HA to non-HA one, it has to be disabled and then migrated.
If that is done fast enough, and some error will occur in router_remove method (https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L547) old router_info will be added to the self.router_info dict.
In such case it will be still HA router added to self.router_info but as this router is already switched to be non-HA, it will use wrong RouterInfo class and errors like:

2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'remove_vip_by_ip_address': AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 158, in call
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1186, in process
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info self._process_internal_ports()
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 594, in _process_internal_ports
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info self.internal_network_added(p)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 356, in internal_network_added
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info port, self.get_internal_device_name, router.INTERNAL_DEV_PREFIX)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 351, in _plug_ha_router_port
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info self._disable_ipv6_addressing_on_interface(interface_name)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 312, in _disable_ipv6_addressing_on_interface
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info self._remove_vip(ipv6_lladdr)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 236, in _remove_vip
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info instance.remove_vip_by_ip_address(ip_cidr)
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
2020-08-05 10:17:11.323 142395 ERROR neutron.agent.l3.router_info
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 8a8e034d-0c53-4d06-a936-9bb53b064f02: AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 694, in _process_routers_if_compatible
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 540, in _process_router_if_compatible
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 581, in _process_updated_router
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent ri.process()
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 485, in process
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent super(HaRouter, self).process()
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 161, in call
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent self.logger(e)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent self.force_reraise()
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent raise value
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 158, in call
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1186, in process
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent self._process_internal_ports()
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 594, in _process_internal_ports
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent self.internal_network_added(p)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 356, in internal_network_added
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent port, self.get_internal_device_name, router.INTERNAL_DEV_PREFIX)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 351, in _plug_ha_router_port
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent self._disable_ipv6_addressing_on_interface(interface_name)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 312, in _disable_ipv6_addressing_on_interface
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent self._remove_vip(ipv6_lladdr)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/ha_router.py", line 236, in _remove_vip
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent instance.remove_vip_by_ip_address(ip_cidr)
2020-08-05 10:17:11.324 142395 ERROR neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'remove_vip_by_ip_address'

will occur in L3 agent.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/747922

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747922
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=489e0ead7297e3b17ca2bf8c4bea9701ad14a939
Submitter: Zuul
Branch: master

commit 489e0ead7297e3b17ca2bf8c4bea9701ad14a939
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 14:44:26 2020 +0200

    Fix migration from the HA to non-HA routers

    In case if during switching HA router to be down, there will be any
    failure, router_info will be stored in L3 agent's cache as HaRouter.
    In case when next update on the router is migration to non-HA router
    this is wrong class and it causes other issues, e.g. with
    remove_vip_by_ip_address() which is correct only for HA routers.

    This patch fixes that issue by adding check of the router's ha and
    distributed flags and update local cache with new router_info class
    in case if at least one of those flags don't match.

    Change-Id: Ib0d3a501f88c149baea7d715c7cfe5811bc85e4f
    Closes-Bug: #1892846

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/763344

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/763345

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/763346

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/763347

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/763348

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/763349

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron stein-eol release.