neutron-ovs-grenade-dvr-multinode job failing in check queue

[Brian Haley]

The neutron-ovs-grenade-dvr-multinode job started failing in the check queue sometime today.

Typical trace coming from the compute1 node:

[...]
+ inc/python:setup_package:445 : safe_chown -R stack /opt/stack/old/neutron/neutron.egg-info
+ functions-common:safe_chown:2340 : _safe_permission_operation chown -R stack /opt/stack/old/neutron/neutron.egg-info
+ functions-common:_safe_permission_operation:2188 : sudo chown -R stack /opt/stack/old/neutron/neutron.egg-info
+ inc/python:_setup_package_with_constraints_edit:390 : use_library_from_git /opt/stack/old/neutron
+ inc/python:use_library_from_git:246 : local name=/opt/stack/old/neutron
+ inc/python:use_library_from_git:247 : local enabled=1
+ inc/python:use_library_from_git:248 : [[ cinder,devstack,glance,grenade,keystone,neutron,nova,placement,requirements,swift,tempest = \A\L\L ]]
+ inc/python:use_library_from_git:248 : [[ ,cinder,devstack,glance,grenade,keystone,neutron,nova,placement,requirements,swift,tempest, =~ ,/opt/stack/old/neutron, ]]
+ inc/python:use_library_from_git:249 : return 1
+ lib/neutron-legacy:install_mutnauq:480 : [[ ovn == \o\v\n ]]
+ lib/neutron-legacy:install_mutnauq:481 : install_ovn
+ lib/neutron_plugins/ovn_agent:install_ovn:363 : echo 'Installing OVN and dependent packages'
Installing OVN and dependent packages
+ lib/neutron_plugins/ovn_agent:install_ovn:366 : ovn_sanity_check
+ lib/neutron_plugins/ovn_agent:ovn_sanity_check:350 : is_service_enabled q-agt neutron-agt
+ functions-common:is_service_enabled:2095 : return 0
+ lib/neutron_plugins/ovn_agent:ovn_sanity_check:351 : die 351 'The q-agt/neutron-agt service must be disabled with OVN.'
+ functions-common:die:264 : local exitcode=0
[Call Trace]
./stack.sh:926:stack_install_service
/opt/stack/old/devstack/lib/stack:32:install_neutron
/opt/stack/old/devstack/lib/neutron:704:install_mutnauq
/opt/stack/old/devstack/lib/neutron-legacy:481:install_ovn
/opt/stack/old/devstack/lib/neutron_plugins/ovn_agent:366:ovn_sanity_check
/opt/stack/old/devstack/lib/neutron_plugins/ovn_agent:351:die
[ERROR] /opt/stack/old/devstack/lib/neutron_plugins/ovn_agent:351 The q-agt/neutron-agt service must be disabled with OVN.
Error on exit

Here's one example:

https://zuul.opendev.org/t/openstack/build/a511fc140ad444bb8df77c670815663f

Looking through open changes, this is the oldest one that seems to fail:

https://review.opendev.org/c/openstack/neutron/+/866635
Started at 2022-12-14 06:37:25

I am guessing this is the culprit:

https://review.opendev.org/c/openstack/grenade/+/862475

I'll create a revert of that since it's EOD here.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/867764

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/867764
Committed: https://opendev.org/openstack/neutron/commit/7c449f1833a94a544cf0b9cfcf63f3e7e46fae26
Submitter: "Zuul (22348)"
Branch: master

commit 7c449f1833a94a544cf0b9cfcf63f3e7e46fae26
Author: Sławek Kapłoński <email address hidden>
Date: Thu Dec 15 09:08:15 2022 +0100

    Enable ML2/OVS backend in the -ovs- grenade jobs explicitly

    As with [1] all grenade jobs by default are using ML2/OVN backend now,
    we need to explicitly configure ML2/OVS backend to be used in all ovs
    related grenade jobs. This patch is doing exactly that.

    [1] https://review.opendev.org/c/openstack/grenade/+/862475

    Closes-Bug: #1999705
    Change-Id: I0770ab54064c49c86b7fefcc0e7eba35c6282ccc

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.