Ubuntu Focal: manila-tempest-plugin-lvm | zfsonlinux | cephfs-nfs | generic | glusterfs-nfs jobs failing

[Ghanshyam Mann]

While migrating the Victoria gate on Ubuntu Focal, below jobs are failing consistently:

- manila-tempest-plugin-lvm
- manila-tempest-plugin-zfsonlinux
- manila-tempest-plugin-cephfs-nfs
- manila-tempest-plugin-generic
- manila-tempest-plugin-glusterfs-nfs

Failure can be seen in PS -https://review.opendev.org/#/c/743134/1 or by running these jobs on Focal nodeset

[Goutham Pacha Ravi (gouthamr)]

Thanks for reporting this bug, Ghanshyam - certainly appreciate the change to manila-tempest-plugin to ensure that the gate is not broken: https://review.opendev.org/#/c/743134/

[Vida Haririan (vhariria)]

Additional comments http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-09-24-15.00.log.html

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/762212

[Goutham Pacha Ravi (gouthamr)]

Partial Fix: https://review.opendev.org/762007
Switches the ZFSOnLinux job to Focal Fossa.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/762218

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/762219

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/762220

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/762468

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762212
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=e3fea14788471cca01c8219e1ef638007e64c97f
Submitter: Zuul
Branch: master

commit e3fea14788471cca01c8219e1ef638007e64c97f
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Nov 10 13:31:38 2020 -0800

    Retry unmount operation on the ZFSOnLinux driver

    When a share is mounted on the same host as the manila-share
    process, zfs prevents us from destroying the underlying
    dataset until the share has been cleanly unmounted from
    the host. Kernel mounts can take a few seconds to get
    unmounted fully especially when there are a lot of
    linux namespaces that the mountpoint has been shared to.

    Add a retry on these operations to harden the deletion
    process and prevent spurious failures.

    Change-Id: I4aba76b72df274d0a8cb90fe0ab8799523c260ef
    Closes-Bug: #1903773
    Related-Bug: #1896672
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762468
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=8a691d8631b9fdb18996566e0706e05776fa0ec4
Submitter: Zuul
Branch: master

commit 8a691d8631b9fdb18996566e0706e05776fa0ec4
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed Nov 11 23:25:32 2020 -0800

    Retry unmount operation on the LVM driver

    When a share is mounted on the same host as the manila-share
    process, the kernel prevents us from destroying the
    mount directory until the share has been cleanly unmounted
    from the host. Kernel mounts can take a few seconds to get
    unmounted fully especially when there are a lot of
    linux namespaces that the mountpoint has been shared to.

    Add a retry on these operations to harden the deletion
    process and prevent spurious failures.

    Change-Id: I3c1a2ec19d6bc18638db0875519ce60f2c89f33a
    Closes-Bug: #1903773
    Related-Bug: #1896672
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762218
Committed: https://git.openstack.org/cgit/openstack/manila-tempest-plugin/commit/?id=5d7eaf0e57bf8e0a970f6c7b283bc88eacc672c4
Submitter: Zuul
Branch: master

commit 5d7eaf0e57bf8e0a970f6c7b283bc88eacc672c4
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Nov 10 15:24:47 2020 -0800

    [ci] Switch generic driver job to focal fossa

    There is one failing test. See bug #1903922 - this failure
    is observed on older versions of Ubuntu as well, and doesn't
    seem to be an artifact of changing the underlying OS.

    Change-Id: Ifb3cb87512c3d8f7cf3af44f7012b967d502bd8e
    Partial-Bug: #1896672
    Related-Bug: #1903922
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762007
Committed: https://git.openstack.org/cgit/openstack/manila-tempest-plugin/commit/?id=dbb8b2d55d795653cc41b670e4ddba0f467fdf92
Submitter: Zuul
Branch: master

commit dbb8b2d55d795653cc41b670e4ddba0f467fdf92
Author: Goutham Pacha Ravi <email address hidden>
Date: Mon Nov 9 14:22:34 2020 -0800

    [ci] Switch zfsonlinux job to focal fossa

    Depends-On: https://review.opendev.org/762212/
    Partial-Bug: #1896672
    Change-Id: I4f56ddacff864116a5348c299d965270f4d64489
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/763009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762220
Committed: https://git.openstack.org/cgit/openstack/manila-tempest-plugin/commit/?id=1a2622d23ab107bcf4bc07cfa6ef5cc0d52e3527
Submitter: Zuul
Branch: master

commit 1a2622d23ab107bcf4bc07cfa6ef5cc0d52e3527
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Nov 10 15:27:22 2020 -0800

    [ci] Switch glusterfs-nfs job to focal fossa

    Depends-On: https://review.opendev.org/762613
    Change-Id: Ia25a49bb35dae62a045eaad9a86b5192141fb32b
    Partial-Bug: #1896672
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762219
Committed: https://git.openstack.org/cgit/openstack/manila-tempest-plugin/commit/?id=1214b82277f3a1e438231de7094453ce8d9a8cd9
Submitter: Zuul
Branch: master

commit 1214b82277f3a1e438231de7094453ce8d9a8cd9
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue Nov 10 15:26:44 2020 -0800

    [ci] Switch lvm driver job to focal fossa

    Depends-On: https://review.opendev.org/762468/
    Change-Id: I15fe5a95037a31d313450a5a4c1ca394f0c259b6
    Partial-Bug: #1896672
    Related-Bug: #1903773
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/763060

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/763061

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/763061
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=bd19642dfd69af3c63dd4d3711d25c353eed4ab6
Submitter: Zuul
Branch: stable/victoria

commit bd19642dfd69af3c63dd4d3711d25c353eed4ab6
Author: Goutham Pacha Ravi <email address hidden>
Date: Wed Nov 11 23:25:32 2020 -0800

    Retry unmount operation on the LVM driver

    When a share is mounted on the same host as the manila-share
    process, the kernel prevents us from destroying the
    mount directory until the share has been cleanly unmounted
    from the host. Kernel mounts can take a few seconds to get
    unmounted fully especially when there are a lot of
    linux namespaces that the mountpoint has been shared to.

    Add a retry on these operations to harden the deletion
    process and prevent spurious failures.

    Change-Id: I3c1a2ec19d6bc18638db0875519ce60f2c89f33a
    Closes-Bug: #1903773
    Related-Bug: #1896672
    Signed-off-by: Goutham Pacha Ravi <email address hidden>
    (cherry picked from commit 8a691d8631b9fdb18996566e0706e05776fa0ec4)

[Goutham Pacha Ravi (gouthamr)]

https://review.opendev.org/763009 removes the restriction from the final job in the series: Ceph-NFS - with this, all tempest test jobs for stable/victoria and trunk (as of Jan 12 2020) now run on focal fossa

[Goutham Pacha Ravi (gouthamr)]

2021* :)