With multiple backends enabled, adding a location does not default to default store

Bug #1802587 reported by Brian Rosmaita
36
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Critical
Abhishek Kekane

Bug Description

I have multiple backends set up in an S-1 devstack and make this call:

data.json:
[
    {
        "op": "add",
        "path": "/locations/-",
        "value": {"url": "https://review.openstack.org/static/title.svg?e=6b20e64d6472edfb0e38b30442683e8a",
                  "metadata": {},
                  "validation_data": {"checksum": "24c3834b2a43a823c181a82b9727d687",
                                      "os_hash_algo": "sha512",
                                      "os_hash_value": "a8ddd76c5b222c772584bdc6099887cdbd9d7fbc48bc5acaa07dcd8a3ba36a32b22f502ea4789a792a66580ef316d929d0506ec3df900d44d9bd51f97ea90ae2"
                                     }
                 }
    }
]

demo! curl -X PATCH \
  -H "x-auth-token: $TK" \
  -H "Content-type: application/openstack-images-v2.1-json-patch" \
  -d @data.json \
  "$OS_IMAGE_URL/v2/images/$IMG"

What I expect: the default store will be used.
What happens:
<html>
 <head>
  <title>400 Bad Request</title>
 </head>
 <body>
  <h1>400 Bad Request</h1>
  Invalid location<br /><br />
 </body>

Here's the error that's occurring. (Line numbers may be off slightly from my hacking.)

Traceback (most recent call last):
  File "/opt/stack/glance/glance/location.py", line 108, in _check_location_uri
    uri, backend, context=context)
  File "/opt/stack/glance_store/glance_store/multi_backend.py", line 486, in get_size_from_uri_and_backend
    uri, backend, conf=CONF)
  File "/opt/stack/glance_store/glance_store/location.py", line 109, in get_location_from_uri_and_backend
    raise exceptions.UnknownScheme(scheme=backend)
UnknownScheme: Unknown scheme 'None' found in URI

The call to store_api.get_size_from_uri_and_backend() is being made with:
uri: https://review.openstack.org/static/title.svg?e=6b20e64d6472edfb0e38b30442683e8a
backend: None

The call is happening here:
https://github.com/openstack/glance/blob/97dac0f3800ad9768e5c118656c6bbbf55fc866d/glance/location.py#L106

Maybe before the call is made, if backend == None, set backend to the default backend?
That won't actually set the location metadata, but it will get it past this call.

The workaround is to add the backend to the location metadata as part of the PATCH.
I have an http store with id == 'Wilma'.
This patch works as expected:

[
    {
        "op": "add",
        "path": "/locations/-",
        "value": {"url": "https://review.openstack.org/static/title.svg?e=6b20e64d6472edfb0e38b30442683e8a",
                  "metadata": {"backend": "Wilma"},
                  "validation_data": {"checksum": "24c3834b2a43a823c181a82b9727d687",
                                      "os_hash_algo": "sha512",
                                      "os_hash_value": "a8ddd76c5b222c772584bdc6099887cdbd9d7fbc48bc5acaa07dcd8a3ba36a32b22f502ea4789a792a66580ef316d929d0506ec3df900d44d9bd51f97ea90ae2"
                                     }
                 }
    }
]

(Note: you have to do this same workaround to get this to work in the glanceclient on Iain's current patch when multiple backends are enabled in Glance.)

Maybe this behavior is fine, that is, don't guess what store to use when multiple backends are enabled, but instead reject the request and make the user state it explicitly. But in that case we need a much better error message -- "Invalid location" is pretty vague.

Revision history for this message
Abhishek Kekane (abhishek-kekane) wrote :
Changed in glance:
assignee: nobody → Abhishek Kekane (abhishek-kekane)
Changed in glance:
status: New → In Progress
importance: Undecided → High
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

I was just discussing this with Iain in IRC:
http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2018-11-16.log.html#t2018-11-16T20:17:06

I am beginning to think that if multiple backends are enabled, we should force the user to specify explicitly which backend is being referred to.

Revision history for this message
Abhishek Kekane (abhishek-kekane) wrote :

Hi Brian,

Thank you, I agree with your suggestion. As per the discussion/suggestion from Ian and you. This needs 3 patches to fix this issue;

1. glance_store, add a new method to get backend from location URI if backend is not specified in location metadata.

2. glance, call above method if backend is not specified in location metadata.

3. python-glanceclient, user can pass --backend to location-add call.

Erno Kuvaja (jokke)
Changed in glance:
importance: High → Critical
Revision history for this message
iain MacDonnell (imacdonn) wrote :

I proposed changes for (1) and (2). (3) can be done already, but it involves the user composing JSON. A --baskend option could be added.

https://review.openstack.org/618932
https://review.openstack.org/618933

Revision history for this message
Victor Coutellier (alistarle) wrote :

I notice that with current implementation of nova snapshot, snapshoting VM with direct_snapshot enabled backend (exemple ceph backend) will always fail, as nova will always hit the "400 invalid location error", should this increase criticity of this bug ?

BTW a spec is about to be merged about the ability of specifying glance backend for nova snapshot : https://review.opendev.org/#/c/641210/5

Revision history for this message
Walt Boring (walter-boring) wrote :

I have run into this as well.
I have setup glance backends as described here:
https://docs.openstack.org/glance/latest/admin/multistores.html

...
[DEFAULT]
...
enabled_backends = fast:rbd, reliable:file

[glance_store]
default_backend = fast
show_image_direct_url = True

[fast]
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8
show_image_direct_url = True
rados_connect_timeout = 0
store_description = "Fast rbd backend"

[reliable]
filesystem_store_datadir = /opt/stack/data/glance/reliable
store_description = "Reliable filesystem backend"

vagrant@ass:~: $ openstack image create --disk-format qcow2 --container-format bare --public --file ./xenial-server-cloudimg-amd64-disk1.img xenial
HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.

Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/domain/proxy.py", line 99, in save
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi result = self.base.save(base_item, from_state=from_state)
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 86, in save
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi self._set_acls(image)
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 72, in _set_acls
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi context=self.context
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 496, in set_a>
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi location_uri, backend, conf=CONF)
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/location.py", line 109, in get_locati>
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi raise exceptions.UnknownScheme(scheme=backend)
Jul 30 18:48:59 ass <email address hidden>[6863]: ERROR glance.common.wsgi glance_store.exceptions.UnknownScheme: Unknown scheme 'None' found in URI

Revision history for this message
Abhishek Kekane (abhishek-kekane) wrote :

Could you try the same using glance image-create command (i.e. using glanceclient). Openstack client is not compatible or up-to-date with current glance changes.

Revision history for this message
Walt Boring (walter-boring) wrote :

I get the same error.

vagrant@ass:~: $ glance image-create --name xenial --visibility public --disk-format qcow2 --container-format bare --file xenial-server-cloudimg-amd64-disk1.img
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2019-07-30T21:22:46Z |
| disk_format | qcow2 |
| id | c90e04bf-b670-44f6-975a-2f7609c08887 |
| min_disk | 0 |
| min_ram | 0 |
| name | xenial |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| owner | 38e0f566b6bf445e89a8404c0466fc90 |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2019-07-30T21:22:46Z |
| virtual_size | Not available |
| visibility | public |
+------------------+--------------------------------------+
HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.

Jul 30 23:22:51 ass <email address hidden>[6863]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 86, in save
Jul 30 23:22:51 ass <email address hidden>[6863]: ERROR glance.common.wsgi self._set_acls(image)
Jul 30 23:22:51 ass <email address hidden>[6863]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 72, in _set_acls
Jul 30 23:22:51 ass <email address hidden>[6863]: ERROR glance.common.wsgi context=self.context
Jul 30 23:22:51 ass <email address hidden>[6863]: ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 496, in set_a>
Jul 30 23:22:51 ass <email address hidden>[6863]: ERROR glance.common.wsgi location_uri, backend, conf=CONF)
Jul 30 23:22:51 ass <email address hidden>[6863]: ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/location.py", line 109, in get_locati>
Jul 30 23:22:51 ass <email address hidden>[6863]: ERROR glance.common.wsgi raise exceptions.UnknownScheme(scheme=backend)
Jul 30 23:22:51 ass <email address hidden>[6863]: ERROR glance.common.wsgi glance_store.exceptions.UnknownScheme: Unknown scheme 'None' found in URI

Revision history for this message
Walt Boring (walter-boring) wrote :

I also tried specifying either of my 2 defined backends 'fast' or 'reliable'
and got the same failure either way.

Revision history for this message
Abhishek Kekane (abhishek-kekane) wrote :

Hi Walt,

Could you please try after applying patch https://review.opendev.org/#/c/658962/

The reason for failure is we have release glance_store version 1.0.0 which uses 'store' keyword instead of 'backend'. If you apply this patch and run the command again after restarting glance-api service you will get the expected result.

Revision history for this message
Walt Boring (walter-boring) wrote :

Thank you, that worked. I tried using both my backends and glance-api seems happy with that patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.opendev.org/617229
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=4e070fd6e0346948898b751516234ba62121e2a8
Submitter: Zuul
Branch: master

commit 4e070fd6e0346948898b751516234ba62121e2a8
Author: Abhishek Kekane <email address hidden>
Date: Fri May 24 07:53:16 2019 +0000

    Make location API compatible with multiple store

    In case of Multiple store while adding location to image if backend store is
    not provided it returns 400 BadRequest saying 'Invalid location'.

    Made changes to loaction API to findout the store from location URI
    and add it as a backend store to the location metadata.

    DocImpact
    Closes-Bug: #1802587
    Co-Authored-By: Victor Coutellier <email address hidden>
    Change-Id: If6d0348346d2086a2500b0012a0e81e80cea7395

Changed in glance:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 19.0.0.0rc1

This issue was fixed in the openstack/glance 19.0.0.0rc1 release candidate.

Revision history for this message
Jan Wasilewski (janwasilewski) wrote :

Hi,

isn't planned to be added as well to openstack/glance for stein release? It is a bit important there as old parameters needs to be replaced:

-stores = glance.store.rbd.Store
+enabled_backends = fast:rbd

 # DEPRECATED:
 # The default scheme to use for storing images.
@@ -2134,7 +2135,7 @@
 # cycle.
 #default_store = file

-default_store = rbd
+default_backend = fast

So it would be nice to have it in-place there.

Best regards,
Jan

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.