[openstack-pike][containers][nova] restarting nova_compute container makes guest disks on nfs share read-only

[Luca Miccini]

see: https://bugzilla.redhat.com/show_bug.cgi?id=1594261

Because of https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/docker/services/nova-compute.yaml#L122 at each start nova_compute will traverse recursively /var/lib/nova/ changing the ownership of the root disks (and console.log) of the vms to nova:nova instead of qemu, hence guests will mark their fs read-only.

This can happen either when restarting the nova_compute container locally or during a scale-out.

*before*

# ls -la
total 70652
drwxr-xr-x. 2 42436 42436 4096 Jun 23 17:33 .
drwxr-xr-x. 6 42436 42436 4096 Jun 23 17:32 ..
-rw-------. 1 root root 0 Jun 23 17:33 console.log
-rw-r--r--. 1 qemu qemu 71630848 Jun 23 17:36 disk
-rw-r--r--. 1 qemu qemu 475136 Jun 23 17:33 disk.config
-rw-r--r--. 1 42436 42436 79 Jun 23 17:32 disk.info

*after*

# ls -la
total 70716
drwxr-xr-x. 2 42436 42436 4096 Jun 23 17:33 .
drwxr-xr-x. 6 42436 42436 4096 Jun 23 17:32 ..
-rw-------. 1 42436 42436 0 Jun 23 17:33 console.log
-rw-r--r--. 1 42436 42436 71696384 Jun 23 17:36 disk
-rw-r--r--. 1 42436 42436 475136 Jun 23 17:33 disk.config
-rw-r--r--. 1 42436 42436 79 Jun 23 17:32 disk.info

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/577855

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/577907

[OpenStack Infra (hudson-openstack)]

Change abandoned by Oliver Walsh (<email address hidden>) on branch: master
Review: https://review.openstack.org/577907
Reason: not required

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577855
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=58624abf5ff97cb1cb016ceae621ef04ac672145
Submitter: Zuul
Branch: master

commit 58624abf5ff97cb1cb016ceae621ef04ac672145
Author: Oliver Walsh <email address hidden>
Date: Mon Jun 25 16:28:17 2018 +0100

    Improve nova statedir ownership logic

    The nova_compute container kolla config is currently set to recursively change
    the ownership of /var/lib/nova to nova:nova on startup.

    This is necessary when upgrading from an non-container deployment to a
    containerized deployment as the nova uids are not consistent across the host
    and container image.

    If the nova instancedir is an NFS mount then open filehandles are
    broken and every VM using that NFS export fails with I/O errors.

    This change re-implements the nova statedir ownership logic to target only the
    files/directories controlled by nova.

    Requires dist-git change: https://review.rdoproject.org/r/14734

    Change-Id: I57d421feb6356d28002e77fb9bfa50a397758cbf
    Closes-bug: 1778465

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/585762

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b4 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585762
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=aff9312637ad47873c6dca738def289ac4aa6ff3
Submitter: Zuul
Branch: stable/queens

commit aff9312637ad47873c6dca738def289ac4aa6ff3
Author: Oliver Walsh <email address hidden>
Date: Mon Jun 25 16:28:17 2018 +0100

    Improve nova statedir ownership logic

    The nova_compute container kolla config is currently set to recursively change
    the ownership of /var/lib/nova to nova:nova on startup.

    This is necessary when upgrading from an non-container deployment to a
    containerized deployment as the nova uids are not consistent across the host
    and container image.

    If the nova instancedir is an NFS mount then open filehandles are
    broken and every VM using that NFS export fails with I/O errors.

    This change re-implements the nova statedir ownership logic to target only the
    files/directories controlled by nova.

    Requires dist-git change: https://review.rdoproject.org/r/15018

    Change-Id: I57d421feb6356d28002e77fb9bfa50a397758cbf
    Closes-bug: 1778465
    (cherry picked from commit 58624abf5ff97cb1cb016ceae621ef04ac672145)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/587066

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.5 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/587066
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=880556f89711b8d5457cd71e63b667d9fe80f0a3
Submitter: Zuul
Branch: stable/pike

commit 880556f89711b8d5457cd71e63b667d9fe80f0a3
Author: Oliver Walsh <email address hidden>
Date: Mon Jun 25 16:28:17 2018 +0100

    Improve nova statedir ownership logic

    The nova_compute container kolla config is currently set to recursively change
    the ownership of /var/lib/nova to nova:nova on startup.

    This is necessary when upgrading from an non-container deployment to a
    containerized deployment as the nova uids are not consistent across the host
    and container image.

    If the nova instancedir is an NFS mount then open filehandles are
    broken and every VM using that NFS export fails with I/O errors.

    This change re-implements the nova statedir ownership logic to target only the
    files/directories controlled by nova.

    Requires dist-git change: https://review.rdoproject.org/r/15067

    Change-Id: I57d421feb6356d28002e77fb9bfa50a397758cbf
    Closes-bug: 1778465
    (cherry picked from commit 58624abf5ff97cb1cb016ceae621ef04ac672145)
    (cherry picked from commit aff9312637ad47873c6dca738def289ac4aa6ff3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.18 release.