Race during removal of the network from DHCP agent

[Slawek Kaplonski]

It may happen that 2 neutron-server workers will try to remove network from the DHCP agent in almost the same way. On one of them all will work fine:

2022-04-19 18:51:26.579 50 WARNING neutron.db.agentschedulers_db [req-73439ae5-d6a4-432c-92f0-872cf5d19bec - - - - -] Removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent 1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report to the server in the last 150 seconds.

but on second one it will fail with error:

2022-04-19 18:51:26.634 50 WARNING neutron.db.agentschedulers_db [req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent 1ddb078b-8cfa-48b4-8a87-228f32f7a947 because the agent did not report to the server in the last 150 seconds.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db [req-ee6aab2f-14e8-4080-a1a6-841421cb28e9 - - - - -] Unexpected exception occurred while removing network d2ee3359-3ddb-44ba-ba89-68c8698d550a from agent 1ddb078b-8cfa-48b4-8a87-228f32f7a947: neutron_lib.exceptions.ObjectNotFound: Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a, dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db Traceback (most recent call last):
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File "/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 342, in remove_networks_from_down_agents
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db notify=False)
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File "/usr/lib/python3.6/site-packages/neutron/db/agentschedulers_db.py", line 431, in remove_network_from_dhcp_agent
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db binding_obj.delete()
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File "/usr/lib/python3.6/site-packages/neutron/objects/base.py", line 912, in delete
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db self._get_composite_keys()))
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 94, in delete_object
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db db_obj = _safe_get_object(obj_cls, context, **kwargs)
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db File "/usr/lib/python3.6/site-packages/neutron/objects/db/api.py", line 80, in _safe_get_object
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db id="%s(%s)" % (obj_cls.db_model.__name__, key))
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db neutron_lib.exceptions.ObjectNotFound: Object NetworkDhcpAgentBinding(network_id=d2ee3359-3ddb-44ba-ba89-68c8698d550a, dhcp_agent_id=1ddb078b-8cfa-48b4-8a87-228f32f7a947) not found.
2022-04-19 18:51:26.732 50 ERROR neutron.db.agentschedulers_db

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/839779

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/839779
Committed: https://opendev.org/openstack/neutron/commit/8df2f69b6f93fd964cc96efb42d085063b90a8e6
Submitter: "Zuul (22348)"
Branch: master

commit 8df2f69b6f93fd964cc96efb42d085063b90a8e6
Author: Slawek Kaplonski <email address hidden>
Date: Thu Apr 28 16:51:25 2022 +0200

    Handle properly ObjectNotFound while deleting network from DHCP agent

    In case when 2 neutron servers are trying to remove network from the
    same DHCP agent it may happend that one of them will not be able
    to find binding object anymore thus deletion of that binding object
    will raise ObjectNotFound exception.
    This patch adds proper handling of such case to not raise ugly
    stacktrace in neutron logs.

    Closes-Bug: #1970759
    Change-Id: I67d516c4583aa0c20416114b92a6d69ece5b970c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/840480

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/840481

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/840482

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/840483

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/840484

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/840485

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840480
Committed: https://opendev.org/openstack/neutron/commit/db2c12636c151d9ee0fa61f302b4ba1b02218df3
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit db2c12636c151d9ee0fa61f302b4ba1b02218df3
Author: Slawek Kaplonski <email address hidden>
Date: Thu Apr 28 16:51:25 2022 +0200

    Handle properly ObjectNotFound while deleting network from DHCP agent

    In case when 2 neutron servers are trying to remove network from the
    same DHCP agent it may happend that one of them will not be able
    to find binding object anymore thus deletion of that binding object
    will raise ObjectNotFound exception.
    This patch adds proper handling of such case to not raise ugly
    stacktrace in neutron logs.

    Closes-Bug: #1970759
    Change-Id: I67d516c4583aa0c20416114b92a6d69ece5b970c
    (cherry picked from commit 8df2f69b6f93fd964cc96efb42d085063b90a8e6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840481
Committed: https://opendev.org/openstack/neutron/commit/97efa2a5ad5be4d5aa13441ae7ef83de6286a211
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 97efa2a5ad5be4d5aa13441ae7ef83de6286a211
Author: Slawek Kaplonski <email address hidden>
Date: Thu Apr 28 16:51:25 2022 +0200

    Handle properly ObjectNotFound while deleting network from DHCP agent

    In case when 2 neutron servers are trying to remove network from the
    same DHCP agent it may happend that one of them will not be able
    to find binding object anymore thus deletion of that binding object
    will raise ObjectNotFound exception.
    This patch adds proper handling of such case to not raise ugly
    stacktrace in neutron logs.

    Closes-Bug: #1970759
    Change-Id: I67d516c4583aa0c20416114b92a6d69ece5b970c
    (cherry picked from commit 8df2f69b6f93fd964cc96efb42d085063b90a8e6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840483
Committed: https://opendev.org/openstack/neutron/commit/4c1d9b2102240404020bd46e9952361f1804c3fa
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 4c1d9b2102240404020bd46e9952361f1804c3fa
Author: Slawek Kaplonski <email address hidden>
Date: Thu Apr 28 16:51:25 2022 +0200

    Handle properly ObjectNotFound while deleting network from DHCP agent

    In case when 2 neutron servers are trying to remove network from the
    same DHCP agent it may happend that one of them will not be able
    to find binding object anymore thus deletion of that binding object
    will raise ObjectNotFound exception.
    This patch adds proper handling of such case to not raise ugly
    stacktrace in neutron logs.

    Closes-Bug: #1970759
    Change-Id: I67d516c4583aa0c20416114b92a6d69ece5b970c
    (cherry picked from commit 8df2f69b6f93fd964cc96efb42d085063b90a8e6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840484
Committed: https://opendev.org/openstack/neutron/commit/882b381ccca18cea37d574ba561928d9536722da
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 882b381ccca18cea37d574ba561928d9536722da
Author: Slawek Kaplonski <email address hidden>
Date: Thu Apr 28 16:51:25 2022 +0200

    Handle properly ObjectNotFound while deleting network from DHCP agent

    In case when 2 neutron servers are trying to remove network from the
    same DHCP agent it may happend that one of them will not be able
    to find binding object anymore thus deletion of that binding object
    will raise ObjectNotFound exception.
    This patch adds proper handling of such case to not raise ugly
    stacktrace in neutron logs.

    Closes-Bug: #1970759
    Change-Id: I67d516c4583aa0c20416114b92a6d69ece5b970c
    (cherry picked from commit 8df2f69b6f93fd964cc96efb42d085063b90a8e6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840485
Committed: https://opendev.org/openstack/neutron/commit/d739ceec23b3a17eae7160ce63193c66842d206e
Submitter: "Zuul (22348)"
Branch: stable/train

commit d739ceec23b3a17eae7160ce63193c66842d206e
Author: Slawek Kaplonski <email address hidden>
Date: Thu Apr 28 16:51:25 2022 +0200

    Handle properly ObjectNotFound while deleting network from DHCP agent

    In case when 2 neutron servers are trying to remove network from the
    same DHCP agent it may happend that one of them will not be able
    to find binding object anymore thus deletion of that binding object
    will raise ObjectNotFound exception.
    This patch adds proper handling of such case to not raise ugly
    stacktrace in neutron logs.

    Closes-Bug: #1970759
    Change-Id: I67d516c4583aa0c20416114b92a6d69ece5b970c
    (cherry picked from commit 8df2f69b6f93fd964cc96efb42d085063b90a8e6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840482
Committed: https://opendev.org/openstack/neutron/commit/ef2873a9f92f82f115c0c7d7fe48490137e0dd44
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit ef2873a9f92f82f115c0c7d7fe48490137e0dd44
Author: Slawek Kaplonski <email address hidden>
Date: Thu Apr 28 16:51:25 2022 +0200

    Handle properly ObjectNotFound while deleting network from DHCP agent

    In case when 2 neutron servers are trying to remove network from the
    same DHCP agent it may happend that one of them will not be able
    to find binding object anymore thus deletion of that binding object
    will raise ObjectNotFound exception.
    This patch adds proper handling of such case to not raise ugly
    stacktrace in neutron logs.

    Closes-Bug: #1970759
    Change-Id: I67d516c4583aa0c20416114b92a6d69ece5b970c
    (cherry picked from commit 8df2f69b6f93fd964cc96efb42d085063b90a8e6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron victoria-eom release.