Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch

[Jakob Meng]

Jobs tripleo-ci-centos-8-9-multinode-mixed-os and its periodic sibling fail on Tempest test groups [1], [2], [3]:

* tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON
* tempest.scenario.test_network_basic_ops.TestNetworkBasicOps

Debugging from @ykarel and @bogdando [4]:

  [12.09.22 13:37] <ykarel> caused by recent update for ovs/ovn in c9 wallaby
  [12.09.22 13:37] <ykarel> as there is different version for these in C8
  [12.09.22 13:37] <ykarel> controller version - 21.12.3-20.21.0-61.4 mismatch with northd version - 22.06.1-20.23.0-63.4

and [5]:

  [12.09.22 13:56] <ykarel> amoralej, jcapitao[m] ovs/ovn update in c9 wallaby broke tripleo mixed upgrade job
  [12.09.22 13:57] <ykarel> https://review.rdoproject.org/r/c/rdoinfo/+/44409
  [12.09.22 14:01] <ykarel> it's failing because of controller version - 21.12.3-20.21.0-61.4 mismatch with northd version - 22.06.1-20.23.0-63.4

Logs:
[1] https://6ae015e4c924496dfda3-74c26bb67ca36871e200d07f3210a3f0.ssl.cf1.rackcdn.com/850000/2/check/tripleo-ci-centos-8-9-multinode-mixed-os/c2aaf84/logs/undercloud/var/log/tempest/stestr_results.html
[2] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6cd/855187/3/check/tripleo-ci-centos-8-9-multinode-mixed-os/6cd1757/logs/undercloud/var/log/tempest/stestr_results.html
[3] https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-9-multinode-mixed-os/5f64adf/logs/undercloud/var/log/tempest/stestr_results.html
[4] https://meetings.opendev.org/irclogs/%23tripleo/%23tripleo.2022-09-12.log.html
[5] https://meetings.opendev.org/irclogs/%23rdo/%23rdo.2022-09-12.log.html

Build history:
* https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os
* https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-9-multinode-mixed-os

[Marios Andreou (marios-b)]

just discussed in irc/#rdo ... we either revert https://review.rdoproject.org/r/c/rdoinfo/+/44409 or we also bump the c8/wallaby versions too.

We will mark the job non voting to clear the gate blocker.
amoralej will try and bump c8. If there are no problems there then we can go with that in the next day/two. Otherwise we can revisit the revert for rdoinfo/+/44409.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/857142

[Marios Andreou (marios-b)]

As written in comment #1 above:

To clear the gate, "Mark tripleo-ci-centos-8-9-multinode-mixed-os non voting" https://review.opendev.org/c/openstack/tripleo-ci/+/857142

For the integration lines patch to remove the jobs from criteria is at https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44885

[Marios Andreou (marios-b)]

@Ronelle - as discussed in irc i sanity checked if this hits both 8 and 9 lines. As expected it appears in both

9 example there https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-8-multinode-mixed-os/4bcfcdc/logs/undercloud/var/log/tempest/stestr_results.html.gz

8 example there https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-9-multinode-mixed-os/5f64adf/logs/undercloud/var/log/tempest/stestr_results.html

so https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44885 is needeed as is

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/857142
Committed: https://opendev.org/openstack/tripleo-ci/commit/a860a16e9313a7c0e8862af69ec4d94942a30721
Submitter: "Zuul (22348)"
Branch: master

commit a860a16e9313a7c0e8862af69ec4d94942a30721
Author: Marios Andreou <email address hidden>
Date: Mon Sep 12 15:58:40 2022 +0300

    Mark tripleo-ci-centos-8-9-multinode-mixed-os non voting

    This is temporary as described in related-bug (see comment #1)

    Related-Bug: 1989341
    Change-Id: I6587573bbfccdb4d83b1bae3364fc51ef4615bbb

[Marios Andreou (marios-b)]

adding some missing context

Patch from amoralej to bump c8 openvswitch:
https://review.rdoproject.org/r/c/rdoinfo/+/44884 Update rdo-openvswitch to 2.17 in RDO Wallaby for CS8

Tested there by ykarel https://review.opendev.org/c/openstack/tripleo-heat-templates/+/857182/ [DNM] Test c8s wallaby with rdo-openvswitch-2.17

[yatin (yatinkarel)]

https://review.rdoproject.org/r/c/rdoinfo/+/44884 is now merged, and updated rdo-openvswitch is now available in c8 deps repo. non-voting patch can be reverted.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/857778

[Marios Andreou (marios-b)]

thanks ykarel and amoralej o/

revert patches up

* Revert "Mark tripleo-ci-centos-8-9-multinode-mixed-os non voting" https://review.opendev.org/c/openstack/tripleo-ci/+/857778

* Revert "Remove periodic-tripleo-ci-centos-8-9-multinode-mixed-os criteria https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44759

and started seeing green builds again https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os

moving fix-released

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/857778
Committed: https://opendev.org/openstack/tripleo-ci/commit/fde3cd71a8e2938c60ff5af59ba3872031abb736
Submitter: "Zuul (22348)"
Branch: master

commit fde3cd71a8e2938c60ff5af59ba3872031abb736
Author: Marios Andreou <email address hidden>
Date: Thu Sep 15 09:22:04 2022 +0000

    Revert "Mark tripleo-ci-centos-8-9-multinode-mixed-os non voting"

    Also fixes gate issue in deprecated layouts using the 'old'
    queue syntax per the change at [1]

    This reverts commit a860a16e9313a7c0e8862af69ec4d94942a30721 and
    fixes deprecated queue syntax

    [1] https://lists.openstack.org/pipermail/openstack-discuss/2022-September/030505.html

    Reason for revert: See discussion in Related-Bug job is now green again

    Change-Id: Ic9fa83f612604dc15401b6f5694326459487d8fe
    Related-Bug: 1989341

[Marios Andreou (marios-b)]

hitting this same bug downstream now

filed https://bugzilla.redhat.com/show_bug.cgi?id=2131275