[stable/train] neutron-tempest-plugin scenario jobs fail "sudo: guestmount: command not found"

[Bernard Cafarelli]

These jobs started failing a few days ago on stable/train:

neutron-tempest-plugin-scenario-linuxbridge-train
neutron-tempest-plugin-scenario-openvswitch-train
neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-train
neutron-tempest-plugin-designate-scenario-train

So basically all neutron-tempest-plugin scenario jobs

Here is a sample backport:
https://review.opendev.org/c/openstack/neutron/+/811974
with failed test run:
https://zuul.opendev.org/t/openstack/build/56b44cb5072648fc8e5ad67df22f8c03

+ /opt/stack/neutron-tempest-plugin/tools/customize_ubuntu_image:mount_image:108 : sudo -E guestmount -i --add /tmp/tmp.lUZGwTOkSx/ubuntu-20.04-minimal-cloudimg-amd64.img --pid-file /tmp/tmp.lUZGwTOkSx/pid /tmp/tmp.lUZGwTOkSx/mount
sudo: guestmount: command not found
+ /opt/stack/neutron-tempest-plugin/tools/customize_ubuntu_image:mount_image:1 : cleanup
+ /opt/stack/neutron-tempest-plugin/tools/customize_ubuntu_image:cleanup:166 : error=1

Maybe something changed in the base image recently? Though newer jobs look OK

[Bernard Cafarelli (bcafarel)]

I just saw a DNM patch I was running on stable/stein has a similar error, but only for neutron-tempest-plugin-designate-scenario-stein job?
https://review.opendev.org/c/openstack/neutron/+/813841
https://zuul.opendev.org/t/openstack/build/aa1997e5d51e4144931c3335a33f8d11

The neutron scenario jobs are still running but they passed the deployment step without a problem

[Bernard Cafarelli (bcafarel)]

I think this appeared with the switch to Ubuntu minimal:
https://opendev.org/openstack/neutron-tempest-plugin/commit/7e5923ac25769073e34a5500504bb8851978cb0e

guestmount does not seem to be installed by default for bionic
With pinned versions all around, I guess the fix is to switch back to 18.04 server image for train and stein (and older?)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/815518

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/815518
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/27f057c2fdf19f4c08ef5e3f142761a1b3cab8f2
Submitter: "Zuul (22348)"
Branch: master

commit 27f057c2fdf19f4c08ef5e3f142761a1b3cab8f2
Author: Bernard Cafarelli <email address hidden>
Date: Tue Oct 26 17:45:39 2021 +0200

    [stable/{train,stein}] Use old guest image for these branches

    Customization of Ubuntu minimal guest images fails with missing
    guestmount binary, switch back to the previous guest image in these EM
    branches

    Change-Id: I98e53ac95b1c407149993ee93fe9eb36a23f0615
    Closes-Bug: #1948804

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin 1.8.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/838053
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/935e0a3812a348d547104a223db01ccccf8be549
Submitter: "Zuul (22348)"
Branch: master

commit 935e0a3812a348d547104a223db01ccccf8be549
Author: Ghanshyam Mann <email address hidden>
Date: Thu May 26 12:26:34 2022 -0500

    Pin neutron-tempest-plugin for ussuri/victoria jobs

    As stable/victoria is in EM phase, and qa team also pinning
    tempest 26.1.0 for stable/victoria
    - https://review.opendev.org/c/openstack/devstack/+/843295

    To have compatible neutron-tempest-plugin version with Tempest
    26.1.0, we need to pin neutron-tempest-plugin 1.6.0 version. And
    remove the vicrotia job from master gate.

    Also same version we need to pin for stable/ussuri also as stable/ussuri
    is also pinned with 26.1.0. in the ussuri
    - https://review.opendev.org/c/openstack/devstack/+/838051

    with 1.8.0, we are facing some incomaptbile code with Tempest 26.1.0
    - https://zuul.opendev.org/t/openstack/build/c4d49a80620b44c7ba708e5afebd2b1c

    Also, both EM branch facing below error (bug#1948804)
     sudo: guestmount: command not found
    and applying the same fix from stable/train
    - https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/815518

    Depends-On: https://review.opendev.org/c/openstack/devstack/+/838051
    Depends-On: https://review.opendev.org/c/openstack/devstack/+/843295

    Related-Bug: #1948804

    Change-Id: I7d8c1b4cf5c483f1eb4411649c56d6cd8cd3bc66

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/869142
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/c6faa5191af97eb12e75d9921a1d6c16d0dd8bc9
Submitter: "Zuul (22348)"
Branch: master

commit c6faa5191af97eb12e75d9921a1d6c16d0dd8bc9
Author: yatinkarel <email address hidden>
Date: Wed Jan 4 11:13:26 2023 +0530

    [ovn] Use ubuntu minimal as advanced image

    Current kernel in Ubuntu focal minimal image
    is 5.4.0-1082-kvm and that includes the fix for
    missing multicast feature[1], let's use the
    minimal image in ovn jobs too like other scenario
    jobs[2].

    Cannot use minimal image in releases before victoria
    as those are pinned to neutron-tempest-plugin-1.6.0
    and the required patch [3] is only available since
    1.8.0. victoria jobs already have advanced image
    specific overrides, added overrides in ussuri ovn job.

    [1] https://bugs.launchpad.net/cloud-images/+bug/1946672
    [2] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/813195
    [3] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/813195

    Related-Bug: #1948804
    Change-Id: Ibe180fc6d1785682033886faf8ba03323bb1a3e8

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/873351

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/873351
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/481f19fb786d2f7a8420f6f061791b9c9f91482a
Submitter: "Zuul (22348)"
Branch: master

commit 481f19fb786d2f7a8420f6f061791b9c9f91482a
Author: yatinkarel <email address hidden>
Date: Fri Feb 10 11:01:09 2023 +0530

    Do not use ubuntu minimal image in ussuri scenario jobs

    It was missed for ussuri scenario jobs in [1] as
    with [2] which was merged around the same time these
    jobs stopped running in CI.

    The issue appeard once the job execution was fixed
    with [3].

    [1] https://review.opendev.org/838053
    [2] https://review.opendev.org/836912
    [3] https://review.opendev.org/869151

    Related-Bug: #1948804
    Closes-Bug: #2006763
    Change-Id: If17f9cc26b3bac42badd435bf1c1fae03a617b5a