Notifications to nova disabled causing tests failures

[Slawek Kaplonski]

For some reason (I have no idea why) during the scenario tests neutron stops sending notifications about port status change to nova. In the logs of neutron server there is info that notifications are disabled, like:

Jan 18 12:05:45.415318 ubuntu-focal-inmotion-iad3-0028053821 neutron-server[71111]: DEBUG neutron.db.provisioning_blocks [None req-b33edd99-de3e-4c86-9fb8-405862418a59 None None] Provisioning complete for port 52a33477-c818-456c-b47f-a323e04cce42 triggered by entity L2. {{(pid=71111) provisioning_complete /opt/stack/neutron/neutron/db/provisioning_blocks.py:139}}
Jan 18 12:05:45.415822 ubuntu-focal-inmotion-iad3-0028053821 neutron-server[71111]: DEBUG neutron_lib.callbacks.manager [None req-b33edd99-de3e-4c86-9fb8-405862418a59 None None] Publish callbacks ['neutron.plugins.ml2.plugin.Ml2Plugin._port_provisioned-1745609'] for port (52a33477-c818-456c-b47f-a323e04cce42), provisioning_complete {{(pid=71111) _notify_loop /usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py:176}}
Jan 18 12:05:45.483625 ubuntu-focal-inmotion-iad3-0028053821 neutron-server[71111]: DEBUG neutron.plugins.ml2.plugin [None req-b33edd99-de3e-4c86-9fb8-405862418a59 None None] Current status of the port 52a33477-c818-456c-b47f-a323e04cce42 is: DOWN; New status is: ACTIVE {{(pid=71111) _update_individual_port_db_status /opt/stack/neutron/neutron/plugins/ml2/plugin.py:2213}}
Jan 18 12:05:45.484006 ubuntu-focal-inmotion-iad3-0028053821 neutron-server[71111]: DEBUG neutron_lib.callbacks.manager [None req-b33edd99-de3e-4c86-9fb8-405862418a59 None None] Publish callbacks ['neutron.plugins.ml2.plugin.SecurityGroupDbMixin._ensure_default_security_group_handler-1688939', 'neutron.services.qos.qos_plugin.QoSPlugin._check_port_for_placement_allocation_change-102324'] for port (52a33477-c818-456c-b47f-a323e04cce42), before_update {{(pid=71111) _notify_loop /usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py:176}}
Jan 18 12:05:45.514465 ubuntu-focal-inmotion-iad3-0028053821 neutron-server[71111]: DEBUG neutron.notifiers.nova [None req-b33edd99-de3e-4c86-9fb8-405862418a59 None None] Nova notifier disabled {{(pid=71111) _can_notify /opt/stack/neutron/neutron/notifiers/nova.py:180}}

Example of failure: https://0cc0fb9d5c802f1c7b9c-6aa1154e79cc29d864ed6f661bf68125.ssl.cf5.rackcdn.com/824982/2/check/neutron-ovs-tempest-multinode-full/1e135d1/testr_results.html

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/825513

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/825513
Committed: https://opendev.org/openstack/neutron/commit/ac718804de7cb2766420ed1eff4e4bc10190fb1c
Submitter: "Zuul (22348)"
Branch: master

commit ac718804de7cb2766420ed1eff4e4bc10190fb1c
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jan 20 15:40:05 2022 +0100

    Add extra logs around enabling/disabling nova notifier code

    I should helps with debugging why the nova notifier is at some point
    disabled for one neutron worker thus notifications aren't send to
    nova from that worker.

    Related-Bug: #1958363
    Change-Id: Ifd88d5bc7515b5f29240bd12be1d3257b308fd67

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/827331

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/828155

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/828156

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/827331
Committed: https://opendev.org/openstack/neutron/commit/507989fc622a6633926c4288827f482bae6054dd
Submitter: "Zuul (22348)"
Branch: master

commit 507989fc622a6633926c4288827f482bae6054dd
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sat Jan 29 10:10:22 2022 +0000

    Use a thread local variable to store the Nova Notifier enable flag

    The Nova Notifier can be called simultaneously by several RPC
    callbacks from the agents (DHCP, L2), trying to update the provisioning
    status of a port. In order to handle each context notifier enable flag,
    a thread local variable is used. This will isolate the flag update
    if two entities inform at the same time and one RPC callback is attended
    during the processing of the other one.

    This patch also removes the debug messages added to debug this issue.

    Closes-Bug: #1958363
    Change-Id: Ie670fba4b3afe427747732d2c3948d92311e960e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/828155
Committed: https://opendev.org/openstack/neutron/commit/68458c369e69605d4c3a75eb3cb21cbbbd68cb87
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 68458c369e69605d4c3a75eb3cb21cbbbd68cb87
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sat Jan 29 10:10:22 2022 +0000

    Use a thread local variable to store the Nova Notifier enable flag

    The Nova Notifier can be called simultaneously by several RPC
    callbacks from the agents (DHCP, L2), trying to update the provisioning
    status of a port. In order to handle each context notifier enable flag,
    a thread local variable is used. This will isolate the flag update
    if two entities inform at the same time and one RPC callback is attended
    during the processing of the other one.

    This patch also removes the debug messages added to debug this issue.

    Closes-Bug: #1958363

    Conflicts:
        neutron/notifiers/nova.py

    Change-Id: Ie670fba4b3afe427747732d2c3948d92311e960e
    (cherry picked from commit 507989fc622a6633926c4288827f482bae6054dd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/828156
Committed: https://opendev.org/openstack/neutron/commit/ec4e453671b0ca1f984c081bdb023be5f674ce0c
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit ec4e453671b0ca1f984c081bdb023be5f674ce0c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sat Jan 29 10:10:22 2022 +0000

    Use a thread local variable to store the Nova Notifier enable flag

    The Nova Notifier can be called simultaneously by several RPC
    callbacks from the agents (DHCP, L2), trying to update the provisioning
    status of a port. In order to handle each context notifier enable flag,
    a thread local variable is used. This will isolate the flag update
    if two entities inform at the same time and one RPC callback is attended
    during the processing of the other one.

    This patch also removes the debug messages added to debug this issue.

    Closes-Bug: #1958363

    Conflicts:
        neutron/notifiers/nova.py

    Change-Id: Ie670fba4b3afe427747732d2c3948d92311e960e
    (cherry picked from commit 507989fc622a6633926c4288827f482bae6054dd)
    (cherry picked from commit 68458c369e69605d4c3a75eb3cb21cbbbd68cb87)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.2.0 release.