tripleo

Deployment fails when nfs mount points exist in /var/lib/nova

Bug #1869020 reported by Oliver Walsh on 2020-03-25

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Medium	Oliver Walsh	tripleo ussuri-3 "tripleo ussuri-3"

Bug Description

As we bind mount /var/lib/nova in a number of containers with the shared:z options, libpod will attempt a recursive relabel on every container restart and fail with errno 95 (Operation not supported) on any NFS exports mounted within it.

Similar to https://bugs.launchpad.net/tripleo/+bug/1835503 where it was addressed when NovaNFSEnabled:True, however NFS mounts can still exist in /var/lib/nova even when NovaNfsEnabled: False e.g when a cinder volume backend is NFS or if trilio vault is mounting NFS exports.

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-25: Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/715015

Changed in tripleo:
assignee:	nobody → Oliver Walsh (owalsh)
status:	New → In Progress

Bogdan Dobrelya (bogdando) on 2020-03-26

Changed in tripleo:
milestone:	none → ussuri-3
importance:	Undecided → High
tags:	added: queens-backport-potential stein-backport-potential train-backport-potential
Changed in tripleo:
importance:	High → Medium

Oliver Walsh (owalsh) on 2020-03-30

tags:

removed: queens-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-30: Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/715015
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=45dd4e18a5e1609260236047fdf4dfb3d6858bd5
Submitter: Zuul
Branch: master

commit 45dd4e18a5e1609260236047fdf4dfb3d6858bd5
Author: Oliver Walsh <email address hidden>
Date: Wed Mar 25 15:15:13 2020 +0000

Tolerate NFS exports in /var/lib/nova when selinux relabelling

    When the :z bind mount option is used, podman peforms a recursive relabel of
    the mount point which is failing with "Operation not supported" if there are
    any NFS exports mounted within. While it's possible for NFS to support true
    selinux labelling, in practice is rarely does.

    As we are already walking the tree to set ownership/permission, take ownership
    of the relabelling logic too and skip relabelling on subtrees where we hit this
    error.

Change-Id: Id5503ed274bd5dc0c5365cc994de7e5cdcbc2fb6
Closes-bug: #1869020

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-31: Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/716280

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-01: Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/716280
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=eea3ff0e1556ba69e120b6ac4ae57a57a050fbc1
Submitter: Zuul
Branch: stable/train

commit eea3ff0e1556ba69e120b6ac4ae57a57a050fbc1
Author: Oliver Walsh <email address hidden>
Date: Wed Mar 25 15:15:13 2020 +0000

Tolerate NFS exports in /var/lib/nova when selinux relabelling

    As we are already walking the tree to set ownership/permission, take ownership
    of the relabelling logic too and skip relabelling on subtrees where we hit this
    error.

    Change-Id: Id5503ed274bd5dc0c5365cc994de7e5cdcbc2fb6
    Closes-bug: #1869020
    (cherry picked from commit 45dd4e18a5e1609260236047fdf4dfb3d6858bd5)

tags:

added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-02: Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/716924

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-02: Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/716924
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=cb889805334a7cd7325b2a9a1efe2bd00bd48c31
Submitter: Zuul
Branch: master

commit cb889805334a7cd7325b2a9a1efe2bd00bd48c31
Author: Oliver Walsh <email address hidden>
Date: Thu Apr 2 11:28:30 2020 +0100

Fix selinux denial on centos8/rhel8 when relabelling /var/lib/nova

Id5503ed274bd5dc0c5365cc994de7e5cdcbc2fb6 is failing with permission
denied on rhel8 due to a selinux denial.

Change-Id: If7a565cdb14282261125d4e32488bb9c5ebc504e
Related-bug: #1869020

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-02: Related fix proposed to tripleo-heat-templates (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/717120

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-03: Related fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/717120
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7ac642644115532eb23c7ca0786e7a1ea14379f1
Submitter: Zuul
Branch: stable/train

commit 7ac642644115532eb23c7ca0786e7a1ea14379f1
Author: Oliver Walsh <email address hidden>
Date: Thu Apr 2 11:28:30 2020 +0100

Fix selinux denial on centos8/rhel8 when relabelling /var/lib/nova

Id5503ed274bd5dc0c5365cc994de7e5cdcbc2fb6 is failing with permission
denied on rhel8 due to a selinux denial.

    Change-Id: If7a565cdb14282261125d4e32488bb9c5ebc504e
    Related-bug: #1869020
    (cherry picked from commit cb889805334a7cd7325b2a9a1efe2bd00bd48c31)