[stable/stein][stable/rocky}[stable/queens] CI jobs failing

[Bernard Cafarelli]

Backport of I320ac2306e0f25ff933d8271203e192486062d61 showed that gates in stein and older branches are not in good shape currently:
[stable/stein] https://review.opendev.org/c/openstack/neutron/+/852752
  neutron-tempest-plugin-api-stein
  neutron-tempest-plugin-scenario-linuxbridge-stein
  openstack-tox-py35
  neutron-grenade
  neutron-functional-python27
  neutron-grenade-multinode
  neutron-grenade-dvr-multinode
  grenade-py3
[stable/rocky] https://review.opendev.org/c/openstack/neutron/+/852753
  all jobs failed with RETRY_LIMIT
  File "/tmp/ansible_zuul_console_payload_jmrk60u_/ansible_zuul_console_payload.zip/ansible/modules/zuul_console.py", line 188
    conn.send(f'{ZUUL_CONSOLE_PROTO_VERSION}\n'.encode('utf-8'))
                                              ^
SyntaxError: invalid syntax

 this may have been already fixed recently in rocky (general openstack CI issue)
[stable/queens] https://review.opendev.org/c/openstack/neutron/+/852754
  similar sea of red RETRY_LIMIT and same error as rocky

With recent "Debian Buster with OpenStack Rocky will receive LTS support" mail from https://lists.openstack.org/pipermail/openstack-discuss/2022-August/029910.html we should at least get stein and then rocky back in shape

[Takashi Kajinami (kajinamit)]

Are we still facing the same error ?

This issue was already fixed by https://review.opendev.org/c/zuul/zuul/+/853073 and the current infra nodes are running the version with that fix.
I initially faced the same issue with centos 7 jobs in Puppet repos but confirmed these jobs were fixed after the that zuul fix was pulled.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/853473

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/853433

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/853474

[Bernard Cafarelli (bcafarel)]

Yes, rocky/queens should give better results now, I filled the bug so we have a track of current CI status - the fresh DNM patches should give better results on actual remaining issues

[Bernard Cafarelli (bcafarel)]

Indeed the "sea of red" disappeared, here are list of jobs failing with DNM patches linked here:
* stein
neutron-tempest-plugin-api-stein
neutron-tempest-plugin-scenario-linuxbridge-stein
neutron-grenade
neutron-grenade-multinode
neutron-grenade-dvr-multinode
grenade-py3

* rocky
neutron-tempest-plugin-api-rocky
neutron-tempest-plugin-scenario-linuxbridge-rocky
neutron-tempest-plugin-scenario-openvswitch-rocky
neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-rocky
neutron-grenade
neutron-functional
neutron-tempest-dvr
neutron-tempest-linuxbridge
neutron-tempest-iptables_hybrid

* queens
neutron-tempest-plugin-api-queens
neutron-tempest-plugin-scenario-linuxbridge-queens
neutron-grenade
neutron-functional
neutron-rally-neutron
neutron-tempest-dvr
neutron-tempest-linuxbridge

[Lajos Katona (lajos-katona)]

I checked stein as the freshest one, and some basic analysis:
* grenade looks like timeouting
* neutron-tempest-plugin-scenario-linuxbridge-stein fails with loading logapi (see [1])
* neutron-tempest-plugin-api-stein fails with flavor creation/search, and not founding ntp_image_256M falvor

[1]: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_312/853473/1/check/neutron-tempest-plugin-scenario-linuxbridge-stein/3120131/controller/logs/screen-q-agt.txt

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/853794

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/853794
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/1472917053c202086a2e71366f29152cfe125c63
Submitter: "Zuul (22348)"
Branch: master

commit 1472917053c202086a2e71366f29152cfe125c63
Author: elajkat <email address hidden>
Date: Fri Aug 19 09:57:19 2022 +0200

    Stein: fix failing jobs

    * neutron-tempest-plugin-api-stein can be fixed by adding
    ADVANCED_INSTANCE_TYPE and ADVANCED_INSTANCE_USER to job definition
    * neutron-tempest-plugin-scenario-linuxbridge-stein can be fixed by
    disabling logging service plugin.

    Change-Id: I808f523eee8bac3529879294d720ab975518d321
    Related-Bug: #1986682

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/853608
Committed: https://opendev.org/openstack/neutron/commit/2debca0340494c74bfd620337685dfdfe5a0b5ac
Submitter: "Zuul (22348)"
Branch: stable/stein

commit 2debca0340494c74bfd620337685dfdfe5a0b5ac
Author: elajkat <email address hidden>
Date: Thu Aug 18 10:54:58 2022 +0200

    Stein only: Make CI green again

    This patch depends on 2 other patches:
    * One in neutron-tempest-plugin to make
    neutron-tempest-plugin-scenario-linuxbridge-stein &
    neutron-tempest-plugin-api-stein green.
    * And one in devstack to make it possible for the linuxbridge job to
    switch off logging service plugin.

    - skip grenade, as QA team doesn't support it on older branches
    (see [1])

    [1]: https://docs.openstack.org/grenade/latest/stable_branch_testing_policy.html

    Related-Bug: #1986682
    Depends-On: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/853794
    Depends-On: https://review.opendev.org/c/openstack/devstack/+/854144
    Change-Id: I2246b102227e97463979d26b2b802c232929d69e

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Bernard Cafarelli <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/853473
Reason: Fixed with https://review.opendev.org/c/openstack/neutron/+/853608

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/853433
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/853474
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Slawek Kaplonski (slaweq)]

All mentioned branches are EOL now, so I'm marking it as Won't fix due to EOL.