voting lvm centos-7 job failing in stable/rocky

[Goutham Pacha Ravi]

Description
===========

This is a gate bug tracker.

The Devstack Manila LVM job (manila-tempest-minimal-dsvm-lvm-centos-7 [1]) is failing to setup devstack. The complete devstack log is attached to this bug report, but the failure is below:

    2020-07-22 13:00:41.463 | Collecting uwsgi
    2020-07-22 13:00:41.536 | Downloading https://mirror.bhs1.ovh.opendev.org/pypifiles/packages/c7/75/45234f7b441c59b1eefd31ba3d1041a7e3c89602af24488e2a22e11e7259/uWSGI-2.0.19.1.tar.gz (803kB)
    2020-07-22 13:00:41.799 | Saved ./uWSGI-2.0.19.1.tar.gz
    2020-07-22 13:00:42.038 | Successfully downloaded uwsgi
    2020-07-22 13:00:42.285 | You are using pip version 9.0.3, however version 20.1.1 is available.
    2020-07-22 13:00:42.285 | You should consider upgrading via the 'pip install --upgrade pip' command.
    2020-07-22 13:00:42.331 | + lib/apache:install_apache_uwsgi:121 : local uwsgi
    2020-07-22 13:00:42.336 | ++ lib/apache:install_apache_uwsgi:122 : ls 'uwsgi*'
    2020-07-22 13:00:42.339 | ls: cannot access uwsgi*: No such file or directory
    2020-07-22 13:00:42.345 | + lib/apache:install_apache_uwsgi:122 : uwsgi=
    2020-07-22 13:00:42.349 | + lib/apache:install_apache_uwsgi:1 : exit_trap
    2020-07-22 13:00:42.353 | + ./stack.sh:exit_trap:521 : local r=2
    2020-07-22 13:00:42.357 | ++ ./stack.sh:exit_trap:522 : jobs -p
    2020-07-22 13:00:42.362 | + ./stack.sh:exit_trap:522 : jobs=
    2020-07-22 13:00:42.367 | + ./stack.sh:exit_trap:525 : [[ -n '' ]]
    2020-07-22 13:00:42.370 | + ./stack.sh:exit_trap:531 : '[' -f '' ']'
    2020-07-22 13:00:42.374 | + ./stack.sh:exit_trap:536 : kill_spinner
    2020-07-22 13:00:42.379 | + ./stack.sh:kill_spinner:431 : '[' '!' -z '' ']'
    2020-07-22 13:00:42.383 | + ./stack.sh:exit_trap:538 : [[ 2 -ne 0 ]]
    2020-07-22 13:00:42.386 | + ./stack.sh:exit_trap:539 : echo 'Error on exit'
    2020-07-22 13:00:42.386 | Error on exit
    2020-07-22 13:00:42.389 | + ./stack.sh:exit_trap:541 : type -p generate-subunit
    2020-07-22 13:00:42.393 | + ./stack.sh:exit_trap:542 : generate-subunit 1595422580 262 fail
    2020-07-22 13:00:42.528 | + ./stack.sh:exit_trap:544 : [[ -z /opt/stack/new ]]

RCA:
---

A new uWSGI release broke devstack [2]. A fix was back ported to this stable branch, but was insufficient for CentOS based nodes [3]. A better fix has been submitted against the stable/queens branch of devstack [4]; but this test job fails on stable/rocky even with that fix [5][6], due to a package install error, it appears that we're pulling in a newer version of setuptools in this job, and that needs investigation.

[1] https://opendev.org/openstack/manila/src/commit/892f1d60ae8dd0831779ba9d57ccf1e5b1e89caa/.zuul.yaml#L29
[2] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015432.html
[3] https://review.opendev.org/#/c/735615/ (Fix for uwsgi and several other gate issues on stable/rocky)
[4] https://review.opendev.org/#/c/736189/ (Fix for uwsgi on centos for stable/queens)
[5] https://review.opendev.org/#/c/741030/ (above fix, cherry-picked to stable/rocky)
[6] https://review.opendev.org/#/c/689635/ (patch illustrating package failure, devstacklog attached in comment #1)

[Goutham Pacha Ravi (gouthamr)]

The file attached contains the package failure after applying the uWSGI install fix proposed in [5]: The failure log is also snipped here: http://paste.openstack.org/show/796255/

[Goutham Pacha Ravi (gouthamr)]

This issue was discussed in the manila community irc meeting on 23rd July 2020: http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-07-23-15.00.log.html#l-94

[Goutham Pacha Ravi (gouthamr)]

Extended maintenance branches, rocky and queens run the voting LVM driver job
in CentOS because the "nfs-utils" package on Ubuntu 16.04 lacked two bug-fixes [7][8]
required to test IPv6 exports. The kernel package has been updated on Ubuntu 18.04 [9]
and the latest package has these fixes.

We switched this job to bionic beaver (Ubuntu 18.04 LTS) during the stein release [10]

As discussed on the community meeting, we'll clone the job and run it on Xenial by disabling IPv6 testing at the moment, until all the issues for the CentOS job have been fixed.

[7] http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commit;h=4663c6481c294838260840d234fec7dfd3186451
[8] http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commit;h=188354e57dd8476e66ce30d647180a106da29b88
[9] https://launchpad.net/nfs-utils/+packages
[10] https://review.opendev.org/#/c/608761/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/742724

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/742724
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=9a1345b2eaa2fefb13f70c54bbcbdef6115e44ce
Submitter: Zuul
Branch: stable/rocky

commit 9a1345b2eaa2fefb13f70c54bbcbdef6115e44ce
Author: Goutham Pacha Ravi <email address hidden>
Date: Thu Jul 23 11:22:24 2020 -0700

    [extended-maintenance-branches-only] Replace LVM job

    the voting LVM job (manila-tempest-minimal-dsvm-lvm-centos-7)
    has been failing for a while due to the issue identified
    in [1]. The fix for the issues didn't suffice for CentOS,
    and we use CentOS because of missing IPv6 related fixes
    in Xenial's nfs-utils package. to avoid impacting helpful
    backports to these branches, this commit introduces a
    new voting job called "manila-tempest-dsvm-lvm-em-only"
    that deploys manila with the LVM driver on Xenial and
    runs IPv4 scenario tests.

    [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015432.html
    Related-Bug: #1888717
    Change-Id: I4a9ae0744757d1c9bfe4f2b213eb04364992af0d
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/742987

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/742987
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=4ce9180d6c427139c7d5d9594e03947d131c03a2
Submitter: Zuul
Branch: stable/queens

commit 4ce9180d6c427139c7d5d9594e03947d131c03a2
Author: Goutham Pacha Ravi <email address hidden>
Date: Thu Jul 23 11:22:24 2020 -0700

    [extended-maintenance-branches-only] Replace LVM job

    the voting LVM job (manila-tempest-minimal-dsvm-lvm-centos-7)
    has been failing for a while due to the issue identified
    in [1]. The fix for the issues didn't suffice for CentOS,
    and we use CentOS because of missing IPv6 related fixes
    in Xenial's nfs-utils package. to avoid impacting helpful
    backports to these branches, this commit introduces a
    new voting job called "manila-tempest-dsvm-lvm-em-only"
    that deploys manila with the LVM driver on Xenial and
    runs IPv4 scenario tests.

    This isn't a perfect cherry-pick since the base job
    was named differently on this branch.

    [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015432.html
    Related-Bug: #1888717
    Change-Id: I4a9ae0744757d1c9bfe4f2b213eb04364992af0d
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit 9a1345b2eaa2fefb13f70c54bbcbdef6115e44ce)
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[Goutham Pacha Ravi (gouthamr)]

All fixes have merged. Closing

[Vida Haririan (vhariria)]

This bug was discussed at https://meetings.opendev.org/meetings/manila/2022/manila.2022-06-09-15.00.log.html.

Updated the bug status as all pending fixes were merged.