insufficient permissions on glance images for direct copy

[Pavel Gluschak]

I'm running Havana multinode. Instances and images are located on SAN attached shared disk (GPFS). Glance images need to be copied by "cp" instead of "curl" to nova's "_base" directory. Here is my configs:

** /etc/glance/glance-api.conf
filesystem_store_datadir = /gpfs/images/
show_multiple_locations = True
filesystem_store_metadata_file = /etc/glance/gpfs.json

** /etc/glance/gpfs.json
{
    "id": "b2b3229e-f22f-4af1-a809-fcf72afe8577",
    "mountpoint": "/gpfs"
}

** /etc/nova/nova.conf
allowed_direct_url_schemes=file
filesystems=gpfs
[image_file_url:gpfs]
id=b2b3229e-f22f-4af1-a809-fcf72afe8577
mountpoint=/gpfs

** Nova log on compute node
2013-12-25 17:29:15.512 10058 INFO nova.virt.libvirt.driver [req-af8fc341-bc26-4217-ba47-51a63d39a934 3cb68bbdc8bf499d82dee70392fe1c62 d1944f8305224f00b7f1faf72937f448] [instance: 4c18cdd0-c852-406
7-bef7-de0b3e6db82b] Creating image
2013-12-25 17:29:16.109 10058 ERROR nova.image.glance [req-af8fc341-bc26-4217-ba47-51a63d39a934 3cb68bbdc8bf499d82dee70392fe1c62 d1944f8305224f00b7f1faf72937f448] Unexpected error while running com
mand.
Command: cp /gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315 /var/lib/nova/instances/_base/bbc9f62419d817181cf3f8f72530133bc0a1172e.part
Exit code: 1
Stdout: ''
Stderr: "cp: cannot open `/gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315' for reading: Permission denied\n"
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Traceback (most recent call last):
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 338, in download
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance xfer_mod.download(context, o, dst_path, loc_meta)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance File "/usr/lib/python2.6/site-packages/nova/image/download/file.py", line 164, in download
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance lv_utils.copy_image(source_file, dst_file)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 462, in copy_image
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance execute('cp', src, dest)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 50, in execute
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance return utils.execute(*args, **kwargs)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance File "/usr/lib/python2.6/site-packages/nova/utils.py", line 177, in execute
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance return processutils.execute(*cmd, **kwargs)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 178, in execute
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance cmd=' '.join(cmd))
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance ProcessExecutionError: Unexpected error while running command.
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Command: cp /gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315 /var/lib/nova/instances/_base/bbc9f62419d817181cf3f8f72530133bc0a1172e.part
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Exit code: 1
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Stdout: ''
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Stderr: "cp: cannot open `/gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315' for reading: Permission denied\n"

** File permissions on image
-rw-r-----. 1 glance glance 10718478336 Dec 23 19:21 /gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315

I assume that compute service was trying to copy image on behalf on "nova" user, that's why this operation was failed with "Permission denied".

[Michael Still (mikal)]

Yes, the nova components are running as the nova user. However, apart from becoming root they don't really have a mechanism to become the glance user. If you want to use filesystem stores like this I would recommend changing the group ownership of these files to one that contains both nova and glance.

[Pavel Gluschak (scsnow)]

Whenever I add new images into glance I have to change permissions on that image files as well to get this works. That's ugly.
We may want to reroute this ticket to glance team to see what we could enhance here..

[Pavel Gluschak (scsnow)]

I created user glance on compute nodes:

glance:x:161:161:OpenStack Glance Daemons:/var/lib/glance:/sbin/nologin

Then I've added nova user to glance group:
$ groups nova
nova : nova nobody qemu glance

And I'm still getting same permission error in the compute log:
2013-12-30 16:39:55.117 10058 TRACE nova.image.glance Command: cp /gpfs/images/f7164998-3fb7-4175-ab08-88ba90f666af /var/lib/nova/instances/_base/ab788f9cda6df158f429306ee2b467e54e6dd604.part
2013-12-30 16:39:55.117 10058 TRACE nova.image.glance Exit code: 1
2013-12-30 16:39:55.117 10058 TRACE nova.image.glance Stdout: ''
2013-12-30 16:39:55.117 10058 TRACE nova.image.glance Stderr: "cp: cannot open `/gpfs/images/f7164998-3fb7-4175-ab08-88ba90f666af' for reading: Permission denied\n"

I logged in as nova user and tried to execute failing command - it works!
Seems like that command is executed on behalf of another user, not nova...

[Zhi Yan Liu (lzy-dev)]

My initial solution is, also talked with John Bresnahan, to add a new option to filesystem store driver which allow admin/operator set a permission code, and fs driver will apply that to the image file when glance adding the new image object.

[John Bresnahan (jbresnah)]

The new option should also be able to set group permissions. In this way the nova user and the glance user could be the exclusive members of the group that owns the files created and permissions could be set to 0640.

SELinux experts should probably weigh in on this issue as well.

[Zhi Yan Liu (lzy-dev)]

@John, yes I think new option value should be able to just provide a complete permission code, and allow admin/operator set any value to it, including group permission field. And invalid code will be logged as an exception when glance try to apply it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/106983

[Zhi Yan Liu (lzy-dev)]

https://review.openstack.org/#/c/106983

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/119529

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/119529
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=706f9112711ad61cda08a3c075d3fbe87bf2ed9d
Submitter: Jenkins
Branch: master

commit 706f9112711ad61cda08a3c075d3fbe87bf2ed9d
Author: Zhi Yan Liu <email address hidden>
Date: Sat Sep 6 11:57:37 2014 +0800

    Allowing operator to configure a permission for image file in fs store

    In this way the user other service used, e.g. Nova, who consumes the
    image could be the exclusive member of the group that owns the files
    created.

    Closes-bug: 1264302
    Related-Id: I4d543d205b0805fe00dcab1b0872c0a5e0f97a5f
    Change-Id: Iec8396f92ed11531dccb82957da2455ca333430a
    Signed-off-by: Zhi Yan Liu <email address hidden>

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zhi Yan Liu (<email address hidden>) on branch: master
Review: https://review.openstack.org/106983
Reason: This bug has been fixed in glance_store.