500 error returned while uploading image using multi filesystem store and 'filesystem_store_metadata_file' option enabled

[Abhijeet Malawade]

When we try to upload image with multi filesystem store enabled and 'filesystem_store_metadata_file'
(containing list of all directories configured for multi filesystem store) is provided
then glance throws 'HTTPInternalServerError (HTTP 500)' error.

- Glance Configuration:

1. /etc/glance/glance-api.conf

[DEFAULT]
show_multiple_locations = True
filesystem_store_metadata_file = /etc/glance/metadata.json

[glance_store]
filesystem_store_datadirs = /var/lib/glance/images1/:1
filesystem_store_datadirs = /var/lib/glance/images2/:2

2. /etc/glance/metadata.json

[
    {
        "id": "f0781415-cf81-47cd-8860-b83f9c2a415c",
        "mountpoint": "/var/lib/glance/images1/"
    },
    {
        "id": "5d2dd1db-8684-46bb-880f-b94a1942cfd2",
        "mountpoint": "/var/lib/glance/images2/"
    }
]

3. 'df -ha' command result:

openstack@openstack:/opt/stack$ df -ha
Filesystem Size Used Avail Use% Mounted on
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
tmpfs 799M 600K 798M 1% /run
/dev/sda1 236M 41M 183M 19% /boot
nfsd 0 0 0 - /proc/fs/nfsd
10.69.4.173:/export/images2 443G 23G 398G 6% /var/lib/glance/images2
10.69.4.172:/export/images1 447G 8.7G 415G 3% /var/lib/glance/images1
openstack@openstack:/opt/stack$

- Steps to reproduce:

1. Create image:
glance --os-image-api-version 2 image-create --name Test123 --disk-format raw --container-format ami

2. Upload image data:

openstack@openstack-150:~$ glance --os-image-api-version 2 image-upload 47d39050-cc7e-498a-a800-4faf80a72c93 < /home/openstack/workbook/test.py

HTTPInternalServerError (HTTP 500)
openstack@openstack-150:~$

- glance-api.log :

2014-12-11 22:16:59.586 3495 ERROR glance_store.backend [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] The storage driver <glance_store._drivers.filesystem.Store
 object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-
8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type
2014-12-11 22:16:59.591 3495 ERROR glance.api.v2.image_data [95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058 48f94106d3b24ca2a0a9e2951c505bf9 - - -] Failed to upload image data due to internal error
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data Traceback (most recent call last):
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/v2/image_data.py", line 74, in upload
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data image.set_data(data, size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/domain/proxy.py", line 160, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.base.set_data(data, size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 252, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.notifier.error('image.upload', msg)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/notifier.py", line 201, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/api/policy.py", line 176, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return self.image.set_data(*args, **kwargs)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/quota/__init__.py", line 296, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data self.image.set_data(data, size=size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/opt/stack/glance/glance/location.py", line 364, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data context=self.context)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 357, in add_to_backend
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return store_add_to_backend(image_id, data, size, store, context)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 338, in store_add_to_backend
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data raise exceptions.BackendException(msg)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data BackendException: The storage driver <glance_store._drivers.filesystem.Store object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint': u'/opt/stack/data/glance/images/', u'id': u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint': u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type

When we enable 'filesystem_store_metadata_file' file option in glance-api.conf then it adds this json data as metadata for all glance images.
This data is then used by NOVA to directly copy image using 'cp' command. (when glance image directories are shared between nova-compute and glance)
For direct download in nova.conf we need to enable 'allowed_direct_url_schemes' option.
Also nova needs 'id' and 'moutpoint' keys to be present in glance-image metadata for direct image copy.

If there is only one glance image directory (ie- one nfs mountpoint) then we do not get this '500 internal server error'.
But if there are multiple image data directories configured and in metadata JSON file we give list of dicts containing all
mountpoints then we get above error.

IMO we need to validate glance metadata JSON file (for 'id' and 'mountpoint' keys) before adding it as image-metadata and if there are multiple mountpoints (directories) configured for glance then we need to add appropriate metadata to glance-image instead of adding all image metadata which is provided in JSON file.

[Abhijeet Malawade (abhijeet-malawade)]

Fix proposed to master branch : https://review.openstack.org/#/c/141311/

[Pranali Deore (pranali-deore)]

We can update 'location' to image using PATCH api as well.
So, validation of metadata also needs to be done in glance project.

[Zhi Yan Liu (lzy-dev)]

Copied my comment about the metadata validation point from change #141311 for other folks who interested on this point:

More thinking on metadata checking stuff widely. Currently we have two points need to make sure metadata is valid: a) new added image should has a proper metadata with its location, this happened on image adding operation, and metadata provided by configured 'filesystem_store_metadata_file' content; b) new added image location should has a correct formatted metadata, this happened on location adding operation by PATCH api, and metadata provided by end user directly. I consider we need to do the same validation for there two points which you provided in this patch, now we only use 'check_location_metadata()' [0][1] to make sure metadata only contains required dict and unicode string instead of json schema like you did here. In feature, I think we need to expose a new interface for store driver like 'check_location_metadata', and calling it in store internal to cover point #a, and calling it at [0] to cover point #b.

[0] https://github.com/openstack/glance/blob/master/glance/location.py#L93
[1] https://github.com/openstack/glance_store/blob/master/glance_store/backend.py#L297

[Zhi Yan Liu (lzy-dev)]

And I think the improvement of above metadata validation concern probably should be covered by a separated bug, due to this bug is talking about 500 error when uploading image instead of adding/updating image location directly.

[Pranali Deore (pranali-deore)]

IMO, using PATCH api it is not possible to pass 'file', 'filesystem' in location field because of the recently merged patch https://review.openstack.org/#/c/145640/. So, here in case of glace we don't need to validate the 'id' and 'mountpoint' in metadata.

[Zhi Yan Liu (lzy-dev)]

RE @Pranali: True, actually I'm talking about a common issue here instead of file store case only however. Anyway, I see the bug was reported for file store, as I said I just want to share my more thoughts around metadata checking stuff.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141311
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=6b9cfa2951beda340d5d63069d63ddd2bd7a3467
Submitter: Jenkins
Branch: master

commit 6b9cfa2951beda340d5d63069d63ddd2bd7a3467
Author: Abhijeet Malawade <email address hidden>
Date: Mon Dec 8 05:50:33 2014 -0800

    Validate metadata JSON file

    If metadata is not valid dictionary then glance-api
    throws 'internal server' error.

    Validating metadata JSON file using jsonschema library
    at configuration stage and caching metadata and using it
    directly when new image being added.
    Metadata JSON file requires 'id' and 'mountpoint' keys
    to be 'string'. If value of 'id' and 'mountpoint' are not
    string then it will raise ValidationError error.
    After getting exception during json data validation,
    'BadStoreConfiguration' exception will be raised.

    If metadata file is list of dicts containing multiple mountpoints
    then returned image metadata as dict containing exact mountpoint
    where image is stored.

    If image path do not starts with any of the 'mountpoint' provided
    in metadata JSON file then error is logged and empty
    dictionary is returned to glance api.

    DocImpact:
    Metadata JSON schema file should be valid object. This JSON object
    should contain keys 'id', 'mountpoint' and value of both keys
    should be string.

    Example of valid metadata JSON:

    1. If there is only one mountpoint-
    {
      "id": "f0781415-cf81-47cd-8860-b83f9c2a415c",
      "mountpoint": "/var/lib/glance/images/"
    }

    2. If there are more than one mountpoints-
    [
      {
        "id": "f0781415-cf81-47cd-8860-b83f9c2a415c",
        "mountpoint": "/var/lib/glance/images1/"
      },
      {
        "id": "asd81415-cf81-47cd-8860-b83f9c2a415c",
        "mountpoint": "/var/lib/glance/images2/"
      }
    ]

    Change-Id: I76b0a0de54e909fb0ec90bd7f9972da76a2501eb
    Closes-Bug: 1401778

[Ian Cordasco (icordasc)]

Is this still a bug in glance? It doesn't seem like it. This should have been fixed in glance_store (which it appears it has been).