Can not upload image to glance

[Hui Kang]

Hi,
uploading image to glance failed due to incorrect permission of /var/lib/glance of the glance-data

410 Gone: Error in store configuration. Adding images to store is disabled. (HTTP N/A)

Permission of /var/lib/glance of the glance-api container shows

drwxr-xr-x 2 root root 4096 Nov 25 20:57 glance

which should be "drwxr-xr-x 3 glance nobody 4096 Nov 6 15:35 glance"

The cause could be the glance-api bootstrap container changes the permission, but the volume is not from glance-data. So when the real glance-api container started and have "volume from", it has a wrong permission.

A temporary workaround is to remove the
  volumes_from:
 - glance_data in glance/tasks/start.yml

- Hui

[Yongfeng Du (dolpherdu)]

I also met this problem with the latest change: https://review.openstack.org/#/c/261843/

When glance-api container starts, it failed to create /var/lib/glance/images/ directory due to "no permission":
======================
2016-01-06 07:47:57.610 1 DEBUG glance_store.backend [-] Attempting to import store file _load_store /var/lib/kolla/venv/local/lib/python2.
7/site-packages/glance_store/backend.py:149
2016-01-06 07:47:57.611 1 INFO glance_store._drivers.filesystem [-] Directory to write image files does not exist (/var/lib/glance/images/)
. Creating.
2016-01-06 07:47:57.612 1 ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images/
2016-01-06 07:47:57.613 1 WARNING glance_store.driver [-] Failed to configure store correctly: Store filesystem could not be configured cor
rectly. Reason: Unable to create datadir: /var/lib/glance/images/ Disabling add method.
2016-01-06 07:47:57.613 1 DEBUG glance_store.capabilities [-] Store glance_store._drivers.filesystem.Store doesn't support updating dynamic
 storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. update_capabiliti
es /var/lib/kolla/venv/local/lib/python2.7/site-packages/glance_store/capabilities.py:98

[Hui Kang (huikang27)]

A temporary solution is to remove the volumes of /var/lib/glance when starting glance-api container

- "glance:/var/lib/glance/"

[Hui Kang (huikang27)]

After further investigation, I found that the root cause is changing ownership failed for bind mounted directory.

In docker/glance-base/Dockerfile.j2, it showns the owner file to the user glance.

    && chown -R glance: /etc/glance /var/log/glance /var/lib/glance /home/glance

However, when I exam the owernship in the glance-api container I found the following

glance@vosklab11:/$ ls -al /etc/ | grep glance
drwxr-xr-x 4 glance glance 4096 Jan 6 17:31 glance

glance@vosklab11:/$ ls -al /var/lib/ | grep glance
drwxrwxrwx 3 root root 4096 Jan 6 17:05 glance

As you see from above, /var/lib/glance is a bind mount directory and chown command does not take effect.

Further, if you check the ownership of /var/lib/glance in the glance-registry, it shows

root@vosklab11:~# docker exec -it glance_registry bash
glance@vosklab11:/$
glance@vosklab11:/$
glance@vosklab11:/$ ls -al /var/lib/ | grep glance
drwxr-xr-x 2 glance glance 4096 Jan 6 17:28 glance

This ownership is correct, because glance-registry does not have bind mount of /var/lib/glance

[Michal Rostecki (mrostecki)]

The cause of this bug is the fact that Docker doesn't support mounts by user. Every time you're running a container with bind mount, the mounted directory is owned by root.

chmod of /var/lib/glance is done only on bootstrap currently. It should be done on every run of container.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264640

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michal Rostecki (<email address hidden>) on branch: master
Review: https://review.openstack.org/264640

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264391
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=e0fc4cb3bd944ff9b7363dadded85e14de027405
Submitter: Jenkins
Branch: master

commit e0fc4cb3bd944ff9b7363dadded85e14de027405
Author: SamYaple <email address hidden>
Date: Wed Jan 6 22:43:44 2016 +0000

    Fix missing mountpoint in glance bootstrap

    Change-Id: I637eef710c0c4964a440e21e3ac15f8fca4b1830
    Closes-Bug: #1519965

[OpenStack Infra (hudson-openstack)]

Change abandoned by Hui Kang (<email address hidden>) on branch: master
Review: https://review.openstack.org/264305
Reason: As fixed by Sam's patch set