Glance image isn't created in the share in case of nfs backend

Bug #1708629 reported by Pranali Deore on 2017-08-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Low
Pranali Deore

Bug Description

Steps to reproduce:
1. Deploy overcloud with cinder and glance on NFS:

Deployment command:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /home/stack/templates/nodes_data.yaml \
-e /home/stack/nfs.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/rhos12.yaml \
--log-file overcloud_deployment_0.log

parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'retry=1'
  CinderNfsServers: 192.168.24.1:/export/cinder

  GlanceBackend: 'file'
  GlanceNfsEnabled: true
  GlanceNfsShare: '192.168.24.1:/export/glance'

(undercloud) [stack@undercloud-0 ~]$ showmount -e
Export list for undercloud-0.redhat.local:
/export/glance 192.168.24.0/24
/export/cinder 192.168.24.0/24

2.
Try to create glance image and check the dir /export/glance - it's empty.

Note: cinder was created as expected under /export/cinder.

(overcloud) [stack@undercloud-0 ~]$ glance image-list
+--------------------------------------+----------+
| ID | Name |
+--------------------------------------+----------+
| 63358c06-1cfe-4810-88aa-d078f7e237bc | test-img |
+--------------------------------------+----------+

Changed in tripleo:
assignee: nobody → Pranali Deore (pranali-deore)

Fix proposed to branch: master
Review: https://review.openstack.org/490839

Changed in tripleo:
status: New → In Progress
Changed in tripleo:
importance: Undecided → Low
milestone: none → pike-rc1

Reviewed: https://review.openstack.org/490839
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f6349079cf78baf8f2d0500cf3818e3689527995
Submitter: Jenkins
Branch: master

commit f6349079cf78baf8f2d0500cf3818e3689527995
Author: Pranali Deore <email address hidden>
Date: Fri Aug 4 17:09:16 2017 +0530

    Mount NFS volume to docker container.

    After creating glance image successfully, share location
    was remaining empty because the NFS volume on controller was
    not mounted to docker container.

    Now, connecting NFS volume to the docker container.

    Change-Id: Ib45f117cbbf2b7b2c0faf024e9a8b049c440d872
    Closes-Bug: 1708629

Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc1 release candidate.

Pranali Deore (pranali-deore) wrote :
Download full text (3.3 KiB)

Seems like this issue is not yet been fixed :(.
Something might have been changed after this merge or may be I have tested it in wrong way while resolving the firewall issues :(

Anyway,

Now the issue here is, there are some permission issues with the mounted volume dir /var/lib/glance and even after resolving this permission issue, the original issue doesn't get resolved i.e.,
The image would not be created in the share as the NFS endpoint is not getting mounted successfully on the container. .
I have investigated little more on this and noticed that the mounting of NFS endpoint[1] doesn't work inside the container even though it makes mountpoint entry in /etc/fstab.

"----------------------------------------------------------------------------------------------------------------------------
()[glance@overcloud-controller-0 /]$ cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Tue Aug 1 17:23:07 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=1af758b4-34e7-40f6-a12d-9ee83d3d521e / ext4 defaults 1 1
192.168.24.1:/export/glance /var/lib/glance/images nfs4 _netdev,bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0
----------------------------------------------------------------------------------------------------------------------------"

I wonder, why the OC deployment doesn't fail when it fails to mount NFS endpoint on container or it doens't even try to mount? (I couldn't find anything related to it in logs :()
I'hv fould only this much logs related to nfs, mounting endpoint is missing:

Sep 7 15:23:56 localhost puppet-user[10]: (/Stage[main]/Tripleo::Glance::Nfs_mount/File[/var/lib/glance/images]/ensure) created
Sep 7 15:23:56 localhost puppet-user[10]: (/Stage[main]/Tripleo::Glance::Nfs_mount/File_line[NFS for glance in fstab]/ensure) created
Sep 7 15:23:56 localhost journal: Notice: /Stage[main]/Tripleo::Glance::Nfs_mount/File[/var/lib/glance/images]/ensure: created
Sep 7 15:23:56 localhost journal: Notice: /Stage[main]/Tripleo::Glance::Nfs_mount/File_line[NFS for glance in fstab]/ensure: created

I'm new in tripleo and don't have hands-on knowledge and experience on containers but after doing some research on this,
I suspect this mounting NFS endpoint on container might not be happenning due to below reasons:
1. nfs-utils is not installed in the container
2. container is not previledged and it runs with non-root user(glance)
3. additionally I think, automount for NFS volume (entry in fstab) doesn't work on container restart

IMO, any of the following could be used for fixing this issue (Please correct me if I'm wrong)
1. mount NFS endpoint on the host ( adding mount in host-prep-tasks section) and fix the permission issue.
2. Create glance_api container with nfs-utils and run it in previledged mode with root user(in this case, mounted NFS volume will not be needed) and we would need to modify kolla start to mount entries in fstab
3. fall back to non-containerized similar to cinder[2].
4. or could be possible by using the docker volume NFS plugin but I'ven't tried this y...

Read more...

Changed in tripleo:
status: Fix Released → In Progress

Reviewed: https://review.openstack.org/502403
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ed11f8ebcfbaf1fbbebb4c83e3201e462fee14ee
Submitter: Jenkins
Branch: master

commit ed11f8ebcfbaf1fbbebb4c83e3201e462fee14ee
Author: Pranali Deore <email address hidden>
Date: Mon Sep 11 11:51:57 2017 +0530

    Providing required priviledges to the mounted NFS volume

    Since, user ID on host and container differs, image-create
    with NFS backend was failing with permission error. But even after
    resolving permission error[1] the image was not getting created
    on the nfs share as the NFS endpoint is not mounted successfully on
    the container via puppet. This will be fixed by [2].

    Now, adding two below changes in this patch,
    [1]. chown glance:glance /var/lib/glance.
    [2]. Proposing this solution to mount NFS endpoint on the host instead
         of mounting it on glance container, because mounting in container
         does not work as explained in LP Bug.

    Closes-Bug: 1708629
    Change-Id: Ib60cb0d179e7c117dc26440746154136aa9d163e

Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b1 development milestone.

Reviewed: https://review.openstack.org/510846
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f42b2b8865274b292aef2f61b3c5ef30d071fd87
Submitter: Zuul
Branch: stable/pike

commit f42b2b8865274b292aef2f61b3c5ef30d071fd87
Author: Pranali Deore <email address hidden>
Date: Mon Sep 11 11:51:57 2017 +0530

    Providing required priviledges to the mounted NFS volume

    Since, user ID on host and container differs, image-create
    with NFS backend was failing with permission error. But even after
    resolving permission error[1] the image was not getting created
    on the nfs share as the NFS endpoint is not mounted successfully on
    the container via puppet. This will be fixed by [2].

    Now, adding two below changes in this patch,
    [1]. chown glance:glance /var/lib/glance.
    [2]. Proposing this solution to mount NFS endpoint on the host instead
         of mounting it on glance container, because mounting in container
         does not work as explained in LP Bug.

    Closes-Bug: 1708629
    Change-Id: Ib60cb0d179e7c117dc26440746154136aa9d163e
    (cherry picked from commit
    ed11f8ebcfbaf1fbbebb4c83e3201e462fee14ee)

tags: added: in-stable-pike

This issue was fixed in the openstack/tripleo-heat-templates 7.0.4 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers