FileNotFoundError not cached during cleaning of HA router

Bug #1892866 reported by Slawek Kaplonski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Medium
Slawek Kaplonski

Bug Description

L3 agent can't handle properly the case when it has to remove HA router and for some reason keepalived's config is already deleted.
It can happen e.g. in case when router is deleted, there is some error during processing this deletion so agent tries to process it again but then ha_conf/<router_id> is already removed.

There is error like:

Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent [-] Error while deleting router e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0: FileNotFoundError: [Errno 2] No such file or directory: '/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 514, in _safe_router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent self._router_removed(ri, router_id)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 551, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent self.router_info[router_id] = ri
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent self.force_reraise()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent raise value
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 547, in _router_removed
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent ri.delete()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 504, in delete
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent self.disable_keepalived()
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/ha_router.py", line 204, in disable_keepalived
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent shutil.rmtree(conf_dir)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/shutil.py", line 477, in rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent onerror(os.lstat, path, sys.exc_info())
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/shutil.py", line 475, in rmtree
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent orig_st = os.lstat(path)
Aug 25 13:30:50 devstack-ubuntu-ovs neutron-l3-agent[29748]: ERROR neutron.agent.l3.agent FileNotFoundError: [Errno 2] No such file or directory: '/opt/stack/data/neutron/ha_confs/e41ab1d4-69cf-4dcb-877e-7d57fa90d3c0'

in such case.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/747926

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/747926
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a08893368a9f323d41f490ee043ef7bd50c4f1fa
Submitter: Zuul
Branch: master

commit a08893368a9f323d41f490ee043ef7bd50c4f1fa
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/748958

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/748959

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/748960

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/748961

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/748962

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/748958
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7fd54d08762c9800549f9230123e304f284e5385
Submitter: Zuul
Branch: stable/ussuri

commit 7fd54d08762c9800549f9230123e304f284e5385
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

tags: added: in-stable-ussuri
tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/748959
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=13e32a4e4d3e83603c2104d13b559ba4ba657299
Submitter: Zuul
Branch: stable/train

commit 13e32a4e4d3e83603c2104d13b559ba4ba657299
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/748960
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5656f3cc89b7bfb76dccfe234aa82c08fbb0fc2c
Submitter: Zuul
Branch: stable/stein

commit 5656f3cc89b7bfb76dccfe234aa82c08fbb0fc2c
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/748961
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bf73cd98aae1b702b00f6dca4c3e6e1ceeb5ae48
Submitter: Zuul
Branch: stable/rocky

commit bf73cd98aae1b702b00f6dca4c3e6e1ceeb5ae48
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    In stable/rocky branch, where we are using Python 2.7 it requires to
    catch EnvironmentError and check errno instead of catching
    FileNotFoundError which isn't available in Python 2.7.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/748962
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=da7e9c5408011bc863b114e3566c18ba203ae959
Submitter: Zuul
Branch: stable/queens

commit da7e9c5408011bc863b114e3566c18ba203ae959
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 25 15:06:04 2020 +0200

    Don't raise FileNotFoundError during disabling keepalived

    In case when keepalived's config is not existing already, there is no
    need to raise any exception while L3 agent is trying to clean this
    config.

    In stable/queens branch, where we are using Python 2.7 it requires to
    catch EnvironmentError and check errno instead of catching
    FileNotFoundError which isn't available in Python 2.7.

    Change-Id: I9ec81ad0c10379294d3145c5902e8b81b65c0221
    Closes-Bug: #1892866
    (cherry picked from commit a08893368a9f323d41f490ee043ef7bd50c4f1fa)

tags: added: in-stable-queens
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers