Unable to remount NFS volume in Glance container

[James Denton]

OS: Ubuntu 18.04 4.15.0-44-generic
OSA: Master

When using Glance w/ NFS, the playbook fails here:

TASK [systemd_mount : Set the state of the mount] ****************************************************************************************************************************
fatal: [infra1_glance_container-8e894228]: FAILED! => {"changed": true, "cmd": "systemctl reload-or-restart $(systemd-escape -p --suffix=\"mount\" \"/var/lib/glance/images\")", "delta": "0:00:00.044078", "end": "2019-02-01 02:21:07.412222", "msg": "non-zero return code", "rc": 1, "start": "2019-02-01 02:21:07.368144", "stderr": "Job for var-lib-glance-images.mount failed.\nSee \"systemctl status var-lib-glance-images.mount\" and \"journalctl -xe\" for details.", "stderr_lines": ["Job for var-lib-glance-images.mount failed.", "See \"systemctl status var-lib-glance-images.mount\" and \"journalctl -xe\" for details."], "stdout": "", "stdout_lines": []}

The service reports the following error:

root@infra1-glance-container-8e894228:~# systemctl status var-lib-glance-images.mount
â—� var-lib-glance-images.mount - Auto mount for /var/lib/glance/images
   Loaded: loaded (/etc/systemd/system/var-lib-glance-images.mount; enabled; vendor preset: enabled)
   Active: active (mounted) since Fri 2019-02-01 02:18:53 UTC; 3min 22s ago
    Where: /var/lib/glance/images
     What: 192.168.2.5:/volume1/images
  Process: 7499 ExecRemount=/bin/mount 192.168.2.5:/volume1/images /var/lib/glance/images -o remount,_netdev,auto -t nfs (code=exited, status=32)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/var-lib-glance-images.mount

Feb 01 02:21:06 infra1-glance-container-8e894228 systemd[1]: var-lib-glance-images.mount: Failed to reset devices.list: Operation not permitted
Feb 01 02:21:07 infra1-glance-container-8e894228 systemd[1]: Reloading Auto mount for /var/lib/glance/images.
Feb 01 02:21:07 infra1-glance-container-8e894228 mount[7499]: mount.nfs: access denied by server while mounting 192.168.2.5:/volume1/images
Feb 01 02:21:07 infra1-glance-container-8e894228 systemd[1]: var-lib-glance-images.mount: Mount process exited, code=exited status=32
Feb 01 02:21:07 infra1-glance-container-8e894228 systemd[1]: Reload failed for Auto mount for /var/lib/glance/images.

Any attempt to manually remount the volume fails:

root@infra1-glance-container-8e894228:~# /bin/mount 192.168.2.5:/volume1/images /var/lib/glance/images -o remount,_netdev,auto -t nfs
mount.nfs: access denied by server while mounting 192.168.2.5:/volume1/images

The following error on baremetal infra coincided with the failures:

Feb 1 03:27:02 infra1 kernel: [133398.114778] audit: type=1400 audit(1548991622.320:578): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-openstack" name="/var/lib/glance/images/" pid=26172 comm="mount.nfs" flags="rw, remount"

Turns out that adding the following line to /etc/apparmor.d/lxc/lxc-openstack fixed it:

  mount options=(rw,remount),

I am not an AppArmor guru, so I don't know if this is the best approach, but could be a good starting point.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/636755

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/636755
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=1fca76c8d39860c27d8c541a8fe300b427dfb129
Submitter: Zuul
Branch: master

commit 1fca76c8d39860c27d8c541a8fe300b427dfb129
Author: cloudnull <email address hidden>
Date: Wed Feb 13 14:56:52 2019 -0600

    Allow containers to remount volumes when needed

    This change allows containers to mount and remount volumes as needed.
    Before this change, when users had a mounted volume within a container,
    like in the case of services using NFS or RBD, it was not possible to
    remount a volume within the container runtime. While a user could
    unmount and mount a volume or restart a container, these actions
    results in service interuption where as a remount would simply
    reload the mounted volume without service interuption.

    Change-Id: Iff588cad451320167b92f2d79e4693a1037be966
    Closes-Bug: #1814200
    Signed-off-by: cloudnull <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts stein-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts yoga-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts xena-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts zed-eom release.