FileNotFoundError not cached during cleaning of HA router

[Slawek Kaplonski]

L3 agent can't handle properly the case when it has to remove HA router and for some reason keepalived's config is already deleted.
It can happen e.g. in case when router is deleted, there is some error during processing this deletion so agent tries to process it again but then ha_conf/<router_id> is already removed.

There is error like:

Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent [-] Error while deleting router e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0: FileNotFoundError: [Errno 2] No such file or directory: '/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 514, in _safe_router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent self._router_removed(ri, router_id)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 551, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent self.router_info[router_id] = ri
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent self.force_reraise()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent raise value
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 547, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent ri.delete()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 504, in delete
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent self.disable_keepalived()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 204, in disable_keepalived
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent shutil.rmtree(conf_dir)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/shutil.py", line 477, in rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent onerror(os.lstat, path, sys.exc_info())
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/shutil.py", line 475, in rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent orig_st = os.lstat(path)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent FileNotFoundError: [Errno 2] No such file or directory: '/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'

in such case.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/747926

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747926
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a08893368a9f323d41f490ee043ef7bd50c4f1fa
Submitter: Zuul
Branch: master

commit a08893368a9f323d41f490ee043ef7bd50c4f1fa
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/748958

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/748959

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/748960

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/748961

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/748962

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748958
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7fd54d08762c9800549f9230123e304f284e5385
Submitter: Zuul
Branch: stable/ussuri

commit 7fd54d08762c9800549f9230123e304f284e5385
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748959
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=13e32a4e4d3e83603c2104d13b559ba4ba657299
Submitter: Zuul
Branch: stable/train

commit 13e32a4e4d3e83603c2104d13b559ba4ba657299
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748960
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5656f3cc89b7bfb76dccfe234aa82c08fbb0fc2c
Submitter: Zuul
Branch: stable/stein

commit 5656f3cc89b7bfb76dccfe234aa82c08fbb0fc2c
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748961
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bf73cd98aae1b702b00f6dca4c3e6e1ceeb5ae48
Submitter: Zuul
Branch: stable/rocky

commit bf73cd98aae1b702b00f6dca4c3e6e1ceeb5ae48
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    In stable/rocky branch, where we are using Python 2.7 it requires to
    catch EnvironmentError and check errno instead of catching
    FileNotFoundError which isn't available in Python 2.7.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748962
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=da7e9c5408011bc863b114e3566c18ba203ae959
Submitter: Zuul
Branch: stable/queens

commit da7e9c5408011bc863b114e3566c18ba203ae959
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    In stable/queens branch, where we are using Python 2.7 it requires to
    catch EnvironmentError and check errno instead of catching
    FileNotFoundError which isn't available in Python 2.7.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.