[openstack-pike][containers][nova] restarting nova_compute container makes guest disks on nfs share read-only

Bug #1778465 reported by Luca Miccini on 2018-06-25
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Oliver Walsh

Bug Description

see: https://bugzilla.redhat.com/show_bug.cgi?id=1594261

Because of https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/docker/services/nova-compute.yaml#L122 at each start nova_compute will traverse recursively /var/lib/nova/ changing the ownership of the root disks (and console.log) of the vms to nova:nova instead of qemu, hence guests will mark their fs read-only.

This can happen either when restarting the nova_compute container locally or during a scale-out.

*before*

# ls -la
total 70652
drwxr-xr-x. 2 42436 42436 4096 Jun 23 17:33 .
drwxr-xr-x. 6 42436 42436 4096 Jun 23 17:32 ..
-rw-------. 1 root root 0 Jun 23 17:33 console.log
-rw-r--r--. 1 qemu qemu 71630848 Jun 23 17:36 disk
-rw-r--r--. 1 qemu qemu 475136 Jun 23 17:33 disk.config
-rw-r--r--. 1 42436 42436 79 Jun 23 17:32 disk.info

*after*

# ls -la
total 70716
drwxr-xr-x. 2 42436 42436 4096 Jun 23 17:33 .
drwxr-xr-x. 6 42436 42436 4096 Jun 23 17:32 ..
-rw-------. 1 42436 42436 0 Jun 23 17:33 console.log
-rw-r--r--. 1 42436 42436 71696384 Jun 23 17:36 disk
-rw-r--r--. 1 42436 42436 475136 Jun 23 17:33 disk.config
-rw-r--r--. 1 42436 42436 79 Jun 23 17:32 disk.info

Luca Miccini (lmiccini2) on 2018-06-25
description: updated
description: updated
Changed in tripleo:
status: New → Triaged
importance: Undecided → High
milestone: none → rocky-3
Oliver Walsh (owalsh) on 2018-06-25
tags: added: pike-backport-potential
tags: added: queens-backport-potential
Changed in tripleo:
assignee: nobody → Oliver Walsh (owalsh)

Fix proposed to branch: master
Review: https://review.openstack.org/577855

Changed in tripleo:
status: Triaged → In Progress

Change abandoned by Oliver Walsh (<email address hidden>) on branch: master
Review: https://review.openstack.org/577907
Reason: not required

Reviewed: https://review.openstack.org/577855
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=58624abf5ff97cb1cb016ceae621ef04ac672145
Submitter: Zuul
Branch: master

commit 58624abf5ff97cb1cb016ceae621ef04ac672145
Author: Oliver Walsh <email address hidden>
Date: Mon Jun 25 16:28:17 2018 +0100

    Improve nova statedir ownership logic

    The nova_compute container kolla config is currently set to recursively change
    the ownership of /var/lib/nova to nova:nova on startup.

    This is necessary when upgrading from an non-container deployment to a
    containerized deployment as the nova uids are not consistent across the host
    and container image.

    If the nova instancedir is an NFS mount then open filehandles are
    broken and every VM using that NFS export fails with I/O errors.

    This change re-implements the nova statedir ownership logic to target only the
    files/directories controlled by nova.

    Requires dist-git change: https://review.rdoproject.org/r/14734

    Change-Id: I57d421feb6356d28002e77fb9bfa50a397758cbf
    Closes-bug: 1778465

Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b4 development milestone.

Reviewed: https://review.openstack.org/585762
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=aff9312637ad47873c6dca738def289ac4aa6ff3
Submitter: Zuul
Branch: stable/queens

commit aff9312637ad47873c6dca738def289ac4aa6ff3
Author: Oliver Walsh <email address hidden>
Date: Mon Jun 25 16:28:17 2018 +0100

    Improve nova statedir ownership logic

    The nova_compute container kolla config is currently set to recursively change
    the ownership of /var/lib/nova to nova:nova on startup.

    This is necessary when upgrading from an non-container deployment to a
    containerized deployment as the nova uids are not consistent across the host
    and container image.

    If the nova instancedir is an NFS mount then open filehandles are
    broken and every VM using that NFS export fails with I/O errors.

    This change re-implements the nova statedir ownership logic to target only the
    files/directories controlled by nova.

    Requires dist-git change: https://review.rdoproject.org/r/15018

    Change-Id: I57d421feb6356d28002e77fb9bfa50a397758cbf
    Closes-bug: 1778465
    (cherry picked from commit 58624abf5ff97cb1cb016ceae621ef04ac672145)

tags: added: in-stable-queens

This issue was fixed in the openstack/tripleo-heat-templates 8.0.5 release.

Reviewed: https://review.openstack.org/587066
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=880556f89711b8d5457cd71e63b667d9fe80f0a3
Submitter: Zuul
Branch: stable/pike

commit 880556f89711b8d5457cd71e63b667d9fe80f0a3
Author: Oliver Walsh <email address hidden>
Date: Mon Jun 25 16:28:17 2018 +0100

    Improve nova statedir ownership logic

    The nova_compute container kolla config is currently set to recursively change
    the ownership of /var/lib/nova to nova:nova on startup.

    This is necessary when upgrading from an non-container deployment to a
    containerized deployment as the nova uids are not consistent across the host
    and container image.

    If the nova instancedir is an NFS mount then open filehandles are
    broken and every VM using that NFS export fails with I/O errors.

    This change re-implements the nova statedir ownership logic to target only the
    files/directories controlled by nova.

    Requires dist-git change: https://review.rdoproject.org/r/15067

    Change-Id: I57d421feb6356d28002e77fb9bfa50a397758cbf
    Closes-bug: 1778465
    (cherry picked from commit 58624abf5ff97cb1cb016ceae621ef04ac672145)
    (cherry picked from commit aff9312637ad47873c6dca738def289ac4aa6ff3)

tags: added: in-stable-pike

This issue was fixed in the openstack/tripleo-heat-templates 7.0.18 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers