Unable to remount NFS volume in Glance container

Bug #1814200 reported by James Denton
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Undecided
Kevin Carter

Bug Description

OS: Ubuntu 18.04 4.15.0-44-generic
OSA: Master

When using Glance w/ NFS, the playbook fails here:

TASK [systemd_mount : Set the state of the mount] ****************************************************************************************************************************
fatal: [infra1_glance_container-8e894228]: FAILED! => {"changed": true, "cmd": "systemctl reload-or-restart $(systemd-escape -p --suffix=\"mount\" \"/var/lib/glance/images\")", "delta": "0:00:00.044078", "end": "2019-02-01 02:21:07.412222", "msg": "non-zero return code", "rc": 1, "start": "2019-02-01 02:21:07.368144", "stderr": "Job for var-lib-glance-images.mount failed.\nSee \"systemctl status var-lib-glance-images.mount\" and \"journalctl -xe\" for details.", "stderr_lines": ["Job for var-lib-glance-images.mount failed.", "See \"systemctl status var-lib-glance-images.mount\" and \"journalctl -xe\" for details."], "stdout": "", "stdout_lines": []}

The service reports the following error:

root@infra1-glance-container-8e894228:~# systemctl status var-lib-glance-images.mount
â—� var-lib-glance-images.mount - Auto mount for /var/lib/glance/images
   Loaded: loaded (/etc/systemd/system/var-lib-glance-images.mount; enabled; vendor preset: enabled)
   Active: active (mounted) since Fri 2019-02-01 02:18:53 UTC; 3min 22s ago
    Where: /var/lib/glance/images
     What: 192.168.2.5:/volume1/images
  Process: 7499 ExecRemount=/bin/mount 192.168.2.5:/volume1/images /var/lib/glance/images -o remount,_netdev,auto -t nfs (code=exited, status=32)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/var-lib-glance-images.mount

Feb 01 02:21:06 infra1-glance-container-8e894228 systemd[1]: var-lib-glance-images.mount: Failed to reset devices.list: Operation not permitted
Feb 01 02:21:07 infra1-glance-container-8e894228 systemd[1]: Reloading Auto mount for /var/lib/glance/images.
Feb 01 02:21:07 infra1-glance-container-8e894228 mount[7499]: mount.nfs: access denied by server while mounting 192.168.2.5:/volume1/images
Feb 01 02:21:07 infra1-glance-container-8e894228 systemd[1]: var-lib-glance-images.mount: Mount process exited, code=exited status=32
Feb 01 02:21:07 infra1-glance-container-8e894228 systemd[1]: Reload failed for Auto mount for /var/lib/glance/images.

Any attempt to manually remount the volume fails:

root@infra1-glance-container-8e894228:~# /bin/mount 192.168.2.5:/volume1/images /var/lib/glance/images -o remount,_netdev,auto -t nfs
mount.nfs: access denied by server while mounting 192.168.2.5:/volume1/images

The following error on baremetal infra coincided with the failures:

Feb 1 03:27:02 infra1 kernel: [133398.114778] audit: type=1400 audit(1548991622.320:578): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-openstack" name="/var/lib/glance/images/" pid=26172 comm="mount.nfs" flags="rw, remount"

Turns out that adding the following line to /etc/apparmor.d/lxc/lxc-openstack fixed it:

  mount options=(rw,remount),

I am not an AppArmor guru, so I don't know if this is the best approach, but could be a good starting point.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-lxc_hosts (master)

Fix proposed to branch: master
Review: https://review.openstack.org/636755

Changed in openstack-ansible:
assignee: nobody → Kevin Carter (kevin-carter)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-lxc_hosts (master)

Reviewed: https://review.openstack.org/636755
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=1fca76c8d39860c27d8c541a8fe300b427dfb129
Submitter: Zuul
Branch: master

commit 1fca76c8d39860c27d8c541a8fe300b427dfb129
Author: cloudnull <email address hidden>
Date: Wed Feb 13 14:56:52 2019 -0600

    Allow containers to remount volumes when needed

    This change allows containers to mount and remount volumes as needed.
    Before this change, when users had a mounted volume within a container,
    like in the case of services using NFS or RBD, it was not possible to
    remount a volume within the container runtime. While a user could
    unmount and mount a volume or restart a container, these actions
    results in service interuption where as a remount would simply
    reload the mounted volume without service interuption.

    Change-Id: Iff588cad451320167b92f2d79e4693a1037be966
    Closes-Bug: #1814200
    Signed-off-by: cloudnull <email address hidden>

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-lxc_hosts stein-eol

This issue was fixed in the openstack/openstack-ansible-lxc_hosts stein-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-lxc_hosts train-eol

This issue was fixed in the openstack/openstack-ansible-lxc_hosts train-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-lxc_hosts ussuri-eol

This issue was fixed in the openstack/openstack-ansible-lxc_hosts ussuri-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-lxc_hosts yoga-eom

This issue was fixed in the openstack/openstack-ansible-lxc_hosts yoga-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-lxc_hosts victoria-eom

This issue was fixed in the openstack/openstack-ansible-lxc_hosts victoria-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-lxc_hosts wallaby-eom

This issue was fixed in the openstack/openstack-ansible-lxc_hosts wallaby-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-lxc_hosts xena-eom

This issue was fixed in the openstack/openstack-ansible-lxc_hosts xena-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-lxc_hosts zed-eom

This issue was fixed in the openstack/openstack-ansible-lxc_hosts zed-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.