Glance image isn't created in the share in case of nfs backend

[Pranali Deore]

Steps to reproduce:
1. Deploy overcloud with cinder and glance on NFS:

Deployment command:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /home/stack/templates/nodes_data.yaml \
-e /home/stack/nfs.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/rhos12.yaml \
--log-file overcloud_deployment_0.log

parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'retry=1'
  CinderNfsServers: 192.168.24.1:/export/cinder

  GlanceBackend: 'file'
  GlanceNfsEnabled: true
  GlanceNfsShare: '192.168.24.1:/export/glance'

(undercloud) [stack@undercloud-0 ~]$ showmount -e
Export list for undercloud-0.redhat.local:
/export/glance 192.168.24.0/24
/export/cinder 192.168.24.0/24

2.
Try to create glance image and check the dir /export/glance - it's empty.

Note: cinder was created as expected under /export/cinder.

(overcloud) [stack@undercloud-0 ~]$ glance image-list
+--------------------------------------+----------+
| ID | Name |
+--------------------------------------+----------+
| 63358c06-1cfe-4810-88aa-d078f7e237bc | test-img |
+--------------------------------------+----------+

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/490839

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/490839
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f6349079cf78baf8f2d0500cf3818e3689527995
Submitter: Jenkins
Branch: master

commit f6349079cf78baf8f2d0500cf3818e3689527995
Author: Pranali Deore <email address hidden>
Date: Fri Aug 4 17:09:16 2017 +0530

    Mount NFS volume to docker container.

    After creating glance image successfully, share location
    was remaining empty because the NFS volume on controller was
    not mounted to docker container.

    Now, connecting NFS volume to the docker container.

    Change-Id: Ib45f117cbbf2b7b2c0faf024e9a8b049c440d872
    Closes-Bug: 1708629

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc1 release candidate.

[Pranali Deore (pranali-deore)]

Seems like this issue is not yet been fixed :(.
Something might have been changed after this merge or may be I have tested it in wrong way while resolving the firewall issues :(

Anyway,

Now the issue here is, there are some permission issues with the mounted volume dir /var/lib/glance and even after resolving this permission issue, the original issue doesn't get resolved i.e.,
The image would not be created in the share as the NFS endpoint is not getting mounted successfully on the container. .
I have investigated little more on this and noticed that the mounting of NFS endpoint[1] doesn't work inside the container even though it makes mountpoint entry in /etc/fstab.

"----------------------------------------------------------------------------------------------------------------------------
()[glance@overcloud-controller-0 /]$ cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Tue Aug 1 17:23:07 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=1af758b4-34e7-40f6-a12d-9ee83d3d521e / ext4 defaults 1 1
192.168.24.1:/export/glance /var/lib/glance/images nfs4 _netdev,bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0
----------------------------------------------------------------------------------------------------------------------------"

I wonder, why the OC deployment doesn't fail when it fails to mount NFS endpoint on container or it doens't even try to mount? (I couldn't find anything related to it in logs :()
I'hv fould only this much logs related to nfs, mounting endpoint is missing:

Sep 7 15:23:56 localhost puppet-user[10]: (/Stage[main]/Tripleo::Glance::Nfs_mount/File[/var/lib/glance/images]/ensure) created
Sep 7 15:23:56 localhost puppet-user[10]: (/Stage[main]/Tripleo::Glance::Nfs_mount/File_line[NFS for glance in fstab]/ensure) created
Sep 7 15:23:56 localhost journal: Notice: /Stage[main]/Tripleo::Glance::Nfs_mount/File[/var/lib/glance/images]/ensure: created
Sep 7 15:23:56 localhost journal: Notice: /Stage[main]/Tripleo::Glance::Nfs_mount/File_line[NFS for glance in fstab]/ensure: created

I'm new in tripleo and don't have hands-on knowledge and experience on containers but after doing some research on this,
I suspect this mounting NFS endpoint on container might not be happenning due to below reasons:
1. nfs-utils is not installed in the container
2. container is not previledged and it runs with non-root user(glance)
3. additionally I think, automount for NFS volume (entry in fstab) doesn't work on container restart

IMO, any of the following could be used for fixing this issue (Please correct me if I'm wrong)
1. mount NFS endpoint on the host ( adding mount in host-prep-tasks section) and fix the permission issue.
2. Create glance_api container with nfs-utils and run it in previledged mode with root user(in this case, mounted NFS volume will not be needed) and we would need to modify kolla start to mount entries in fstab
3. fall back to non-containerized similar to cinder[2].
4. or could be possible by using the docker volume NFS plugin but I'ven't tried this y...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/502403

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/502403
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ed11f8ebcfbaf1fbbebb4c83e3201e462fee14ee
Submitter: Jenkins
Branch: master

commit ed11f8ebcfbaf1fbbebb4c83e3201e462fee14ee
Author: Pranali Deore <email address hidden>
Date: Mon Sep 11 11:51:57 2017 +0530

    Providing required priviledges to the mounted NFS volume

    Since, user ID on host and container differs, image-create
    with NFS backend was failing with permission error. But even after
    resolving permission error[1] the image was not getting created
    on the nfs share as the NFS endpoint is not mounted successfully on
    the container via puppet. This will be fixed by [2].

    Now, adding two below changes in this patch,
    [1]. chown glance:glance /var/lib/glance.
    [2]. Proposing this solution to mount NFS endpoint on the host instead
         of mounting it on glance container, because mounting in container
         does not work as explained in LP Bug.

    Closes-Bug: 1708629
    Change-Id: Ib60cb0d179e7c117dc26440746154136aa9d163e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/510846

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/510846
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f42b2b8865274b292aef2f61b3c5ef30d071fd87
Submitter: Zuul
Branch: stable/pike

commit f42b2b8865274b292aef2f61b3c5ef30d071fd87
Author: Pranali Deore <email address hidden>
Date: Mon Sep 11 11:51:57 2017 +0530

    Providing required priviledges to the mounted NFS volume

    Since, user ID on host and container differs, image-create
    with NFS backend was failing with permission error. But even after
    resolving permission error[1] the image was not getting created
    on the nfs share as the NFS endpoint is not mounted successfully on
    the container via puppet. This will be fixed by [2].

    Now, adding two below changes in this patch,
    [1]. chown glance:glance /var/lib/glance.
    [2]. Proposing this solution to mount NFS endpoint on the host instead
         of mounting it on glance container, because mounting in container
         does not work as explained in LP Bug.

    Closes-Bug: 1708629
    Change-Id: Ib60cb0d179e7c117dc26440746154136aa9d163e
    (cherry picked from commit
    ed11f8ebcfbaf1fbbebb4c83e3201e462fee14ee)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.4 release.