[MultipleStores] All queries to glance API fails with 503 error if rbd store configured incorrectly

[Abhishek Kekane]

If I misconfigure my glance-api.conf with an incorrect rbd_store_user for one of my store then that store does not work as expected. All requests to glance fails with a 503 error.

Steps to reproduce:
1. Enabled multiple stores by configuring more than one store using enabled_backends config option
2. Provide incorrect rbd_store_user name for one of the rbd store.
3. Restart the glance api service
4. Run glance image-list command

Expected output:
Should list available images

Actual output:
Fails with 503 error

NOTE: Sample glance-api.conf

[default]
enabled_backends = ceph:rbd,fast:file

[glance_store]
default_backend = ceph

[ceph]
rbd_store_pool = images
rbd_store_user = glance-break # incorrect user name
rbd_store_ceph_conf = /etc/ceph/ceph.conf

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/722915
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=98b9091129e78d03dc5b97bd1503063499f166d5
Submitter: Zuul
Branch: master

commit 98b9091129e78d03dc5b97bd1503063499f166d5
Author: Abhishek Kekane <email address hidden>
Date: Fri Apr 24 15:39:26 2020 +0000

    Fix: API returns 503 if one of the store is mis-configured

    If user mis-configures one of the multiple stores defined in glance-api
    configuratio file, then service returns 503 for all API requests.

    Made provision to exclude faulty store and resume service normally
    to function wiht other configured stores.

    Closes-Bug: #1875281
    Change-Id: I685a0ecbfba7cf7dbce3fd2eb84f97bd8ffbfcf3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/726309

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726309
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=17b66ab6a50f8dda28fc2f4c02489bb5ba4241e3
Submitter: Zuul
Branch: stable/ussuri

commit 17b66ab6a50f8dda28fc2f4c02489bb5ba4241e3
Author: Abhishek Kekane <email address hidden>
Date: Fri Apr 24 15:39:26 2020 +0000

    Fix: API returns 503 if one of the store is mis-configured

    If user mis-configures one of the multiple stores defined in glance-api
    configuratio file, then service returns 503 for all API requests.

    Made provision to exclude faulty store and resume service normally
    to function wiht other configured stores.

    Closes-Bug: #1875281
    Change-Id: I685a0ecbfba7cf7dbce3fd2eb84f97bd8ffbfcf3
    (cherry picked from commit 98b9091129e78d03dc5b97bd1503063499f166d5)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/726735

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726735
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=a6227665ef8ec654ca342362996b1cc40357a290
Submitter: Zuul
Branch: stable/train

commit a6227665ef8ec654ca342362996b1cc40357a290
Author: Abhishek Kekane <email address hidden>
Date: Fri Apr 24 15:39:26 2020 +0000

    Fix: API returns 503 if one of the store is mis-configured

    If user mis-configures one of the multiple stores defined in glance-api
    configuratio file, then service returns 503 for all API requests.

    Made provision to exclude faulty store and resume service normally
    to function wiht other configured stores.

    Closes-Bug: #1875281
    Change-Id: I685a0ecbfba7cf7dbce3fd2eb84f97bd8ffbfcf3
    (cherry picked from commit 98b9091129e78d03dc5b97bd1503063499f166d5)
    (cherry picked from commit 17b66ab6a50f8dda28fc2f4c02489bb5ba4241e3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 2.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store train-eol release.