stable/train: neutron-grenade job consistently fails in reqirements repo

[Takashi Kajinami]

Currently the neutron-grenade job in stable/train branch of requirements repo consistently fails.

Example:
https://zuul.opendev.org/t/openstack/build/1a4f23e400a3491b88b161e50878753a

Looking at the job-output.txt, it seems installation is being stuck at some point
but I could not find out the actual cause because of no logs captured.

[Takashi Kajinami (kajinamit)]

The last successful run passed on June 22nd, but the job result expired so I can no longer review details...

https://zuul.openstack.org/builds?job_name=neutron-grenade&branch=stable%2Ftrain&skip=0

[Mamatisa Nurmatov (isabek)]

Related patch: https://review.opendev.org/c/openstack/requirements/+/850828

[Bernard Cafarelli (bcafarel)]

2022-07-25 06:10:25.683174 | primary | Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
2022-07-25 08:17:30.104018 | primary | + export 'PROJECTS=openstack/grenade '

This looks similar to UT test issues we recently had in other stable branches - close to 2 hours to run the step I pasted (in updating packages)

[Lajos Katona (lajos-katona)]

I copy here my comment on the make non-voting patch (https://review.opendev.org/c/openstack/requirements/+/850828 ):
hmmm, seems like many runs fail with:
2022-07-14 01:43:48.646797 | primary | Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
2022-07-14 04:27:45.905120 | primary | /home/zuul/workspace/devstack-gate/functions.sh: line 1009: 12039 Killed timeout -s 9 ${REMAINING_TIME}m bash -c "source $WORKSPACE/devstack-gate/functions.sh && $cmd"
2022-07-14 04:27:45.905716 | primary | ERROR: the main setup script run by this job failed - exit code: 137

from https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_05f/849767/1/check/neutron-grenade/05f09e6/job-output.txt

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/851330

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/851330
Committed: https://opendev.org/openstack/neutron/commit/01c88ae08c1e5dc3f4b57c7531d367ab7f4094dd
Submitter: "Zuul (22348)"
Branch: stable/train

commit 01c88ae08c1e5dc3f4b57c7531d367ab7f4094dd
Author: Takashi Kajinami <email address hidden>
Date: Thu Jul 28 13:09:09 2022 +0900

    [Train-only] Make neutron-granade job non-voting

    Currently the neutron-grenade job in stable/train is consistently
    failing because of the job script being stuck.

    This change temporarily makes the job non-voting to unblock CI.

    Change-Id: I34aabd97fc51ee6393ab3c6f1c6f462db8efbdb5
    Related-Bug: #1982720

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/851506

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/851512

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/851506
Committed: https://opendev.org/openstack/neutron/commit/5bd813e328f5b7f2f057c41d6e188b1bfc11ee55
Submitter: "Zuul (22348)"
Branch: stable/train

commit 5bd813e328f5b7f2f057c41d6e188b1bfc11ee55
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jul 29 12:40:15 2022 +0200

    [stable/train] Convert neutron-grenade job to the zuul v3 job

    Other grenade jobs, like neutron-grenade-multinode are already converted
    to the native zuul v3 definitions but neutron-grenade job wasn't and it
    seems that as it is using devstack-gate project to provision vm, for
    some reason it stucks now and job fails with timeout.
    To avoid that lets convert that job to the native Zuul v3 syntax.

    This patch is needed only for stable/train as in newer branches we don't
    have that single node job anymore.

    This patch also switches neutron-grenade job to be voting in the check queue
    and to be run in the gate queue. It was made non-voting and no gating with
    the patch https://review.opendev.org/c/openstack/neutron/+/851330 to
    workaround the issue which this patch fixes.

    Closes-bug: #1982720
    Change-Id: Iaa42e0d3020c67d22583ed70e624c401b192dc93

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/851512
Committed: https://opendev.org/openstack/neutron/commit/a4ed825cc69c76b55875adbbd4e2eab24d07a1e3
Submitter: "Zuul (22348)"
Branch: stable/train

commit a4ed825cc69c76b55875adbbd4e2eab24d07a1e3
Author: Bernard Cafarelli <email address hidden>
Date: Fri Jul 29 15:07:23 2022 +0200

    [stable/train] Convert neutron-tempest-postgres-full to zuul v3

    Similar to neutron-grenade, this legacy periodic job is now failing,
    timeout in setup steps [0]

    Switch it to tempest-integrated-networking (as was done in later
    branches)

    Also remove leftover playbooks from neutron-tempest-with-os-ken-master
    job (removed in stable branches)

    Related-Bug: #1982720

    [0] https://zuul.openstack.org/builds?job_name=neutron-tempest-postgres-full&branch=stable%2Ftrain

    Change-Id: I8936df335dcddff5d7f8111815e2062a42e6bf6d

[Brian Haley (brian-haley)]

Can this be considered fixed now?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.