Incomplete support for multiple Volume Types with Cinder storage backend

[Przemyslaw Hausman]

With Cinder backing Glance and multiple Volume Types present in Cinder, it is not possible to decide which Volume Type should be used when creating new image.

The expectation is that the following image creation commands end up with two images created, backed by two different volume types, `volume-type-1` and `volume-type-2`:

```
glance image-create --store volume-type-1 ...
glance image-create --store volume-type-2 ...
```

Instead, both images are backed by Cinder volumes of type `volume-type-1`.

Glance's `cinder-volume-types` config option is configured with comma separated list of volume types: 'volume-type-1,volume-type-2'. The output of `glance stores-info` correctly shows two cinder stores:

[{"id": "local", "description": "Local filesystem store"}, {"id": "volume-type-1", "default": "true"}, {"id": "volume-type-2"}]

When examining Glance config file rendered by the Glance charm, I can see both stanzas for two storage backends but each stanza is missing `cinder_volume_type` configuration:

```
[DEFAULT]
...

enabled_backends = local:file, volume-type-1:cinder, volume-type-2:cinder

[glance_store]

default_backend = volume-type-1

...

[volume-type-1]
cinder_catalog_info = volumev3:cinderv3:internalURL

volume-type-2]
cinder_catalog_info = volumev3:cinderv3:internalURL
```

It looks like there's a code populating cinder_volume_type: https://opendev.org/openstack/charm-glance/src/branch/stable/yoga/hooks/glance_contexts.py#L371. I'm not sure why it is not rendered in my environment though.

Glance channel: yoga/stable
Cinder channel: yoga/stable
Ubuntu Jammy 22.04

[Alex Kavanagh (ajkavanagh)]

After some oob conversation with Przemyslaw, we worked out the bug is at:

https://opendev.org/openstack/charm-glance/src/commit/3fadd2374f99bb7d4e212cbe6cdb8116d5eecfbd/hooks/glance_contexts.py#L389

Essentially, this code reads as:

                for volume_type in volume_types:
                    if 'volume_type' not in ctxt['enabled_backend_configs']:
                        ctxt['enabled_backend_configs'][volume_type] = {}
                    ctxt['enabled_backend_configs'][volume_type].update(
                        {'cinder_catalog_info': volume_catalog_info})

And the bug is the string 'volume_type' rather than the bare volume_type in the "if 'volume_type' not in ...". It will always blank the ctxt[..][volume_type] as 'volume_type' (as a string key) doesn't exist in the ctxt[..] dictionary.

A relatively simple fix, but just waiting on Przemyslaw to do some live testing to see if the simple fix actually resolves the whole problem.

[Przemyslaw Hausman (phausman)]

We have tried patching the charm in live environment, i.e. by replacing 'volume_type' with volume_type, but it looks like there's something else missing still.

With patched glance charm, `glance image-create` fails as follows:

```
$ glance image-create --store volume-type-1 --file ./cirros-0.5.1-x86_64-disk.img --container-format bare --disk-format qcow2
HTTP 503 Service Unavailable: Service Unavailable: The server is temporarily unable to service your: request due to maintenance downtime or capacity: problems. Please try again later.: Apache/2.4.52 (Ubuntu) Server at image.example.com Port 9292
```

glance-api.conf seem to be configured correctly:
```
...

[volume-type-1]
cinder_volume_type = volume-type-1
cinder_http_retries = 3
cinder_state_transition_timeout = 30
cinder_catalog_info = volumev3:cinderv3:internalURL

[volume-type-2]
cinder_volume_type = volume-type-2
cinder_http_retries = 3
cinder_state_transition_timeout = 30
cinder_catalog_info = volumev3:cinderv3:internalURL
```

Glance log shows the following DEBUG messages, I'm not sure if this is relevant though:

```
2024-02-12 10:20:31.173 2512343 DEBUG glance_store.multi_backend [-] Attempting to import store file _load_multi_store /usr/lib/python3/dist-packages/glance_store/multi_backend.py:171
2024-02-12 10:20:31.174 2512343 DEBUG glance_store.capabilities [-] Store glance_store._drivers.filesystem.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. update_capabilities /usr/lib/python3/dist-packages/glance_store/capabilities.py:91
2024-02-12 10:20:31.174 2512343 DEBUG glance_store.multi_backend [-] Registering store file with schemes ('file', 'filesystem') create_multi_stores /usr/lib/python3/dist-packages/glance_store/multi_backend.py:304
2024-02-12 10:20:31.175 2512343 DEBUG glance_store.driver [-] Late loading location class glance_store._drivers.filesystem.StoreLocation get_store_location_class /usr/lib/python3/dist-packages/glance_store/driver.py:117
2024-02-12 10:20:31.175 2512343 DEBUG glance_store.location [-] Registering scheme file with {'local': {'store': <glance_store._drivers.filesystem.Store object at 0x7f0e86c797b0>, 'location_class': <class 'glance_store._drivers.filesystem.StoreLocation'>, 'store_entry': 'file'}} register_scheme_backend_map /usr/lib/python3/dist-packages/glance_store/location.py:132
2024-02-12 10:20:31.175 2512343 DEBUG glance_store.location [-] Registering scheme file with {'local': {'store': <glance_store._drivers.filesystem.Store object at 0x7f0e86c797b0>, 'location_class': <class 'glance_store._drivers.filesystem.StoreLocation'>, 'store_entry': 'file'}} register_scheme_backend_map /usr/lib/python3/dist-packages/glance_store/location.py:132
2024-02-12 10:20:31.175 2512343 DEBUG glance_store.location [-] Registering scheme filesystem with {'local': {'store': <glance_store._drivers.filesystem.Store object at 0x7f0e86c797b0>, 'location_class': <class 'glance_store._drivers.filesystem.StoreLocation'>, 'store_entry': 'file'}} register_scheme_backend_map /usr/lib/python3/dist-packages/glance_store/location.py:132
2024-02-12 10:20:31.175 2...

[Przemyslaw Hausman (phausman)]

Subscribing field-critical as we're hitting this on customer deployment.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-glance/+/908842

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance/+/908842
Committed: https://opendev.org/openstack/charm-glance/commit/cfb228fc43ef29764295cefb45407162143cee09
Submitter: "Zuul (22348)"
Branch: master

commit cfb228fc43ef29764295cefb45407162143cee09
Author: James Page <email address hidden>
Date: Mon Feb 12 17:14:55 2024 +0000

    Fix check for internal endpoint enablement

    For cinder volume backend sections, ensure that enabling internal
    endpoint usage for service access does not completely clear the
    existing content of the section.

    Change-Id: I383a0794e924db3c7f4ce47d1e3b88355c2475e4
    Closes-Bug: 2052726

[Przemyslaw Hausman (phausman)]

Hi Alex, can you please take a look my comment in https://bugs.launchpad.net/charm-glance/+bug/2052726/comments/2. It looks like the immediate patch (replacing 'volume_type' with volume_type) does not seem to work.

[Alex Kavanagh (ajkavanagh)]

> Hi Alex, can you please take a look my comment in https://bugs.launchpad.net/charm-glance/+bug/2052726/comments/2. It looks like the immediate patch (replacing 'volume_type' with volume_type) does not seem to work.

How are the cinder volume types configured? i.e. what does cinder.conf contain?

[Vijay Kumar (vijay10193)]

Hi Alex,

The cinder is configured with two backend storage, find the attached cinder.conf.

We have cinder and cinder-volume charms, and cinder-volume is deployed to bare metal and cinder deployed on a container

[SANSEN (mathieusansen)]

Hi, were you able to adapt the code and make it ready to work with two backend storage at the same time?
Regards

[Alex Kavanagh (ajkavanagh)]

Setting back to triaged as the bug fix doesn't seem to be a complete fix, based on new information in #8 and #9.

[Jorge Merlino (jorge-merlino)]

Hi Alex,

I've been debugging this issue and found out that glance-api does not start correctly when the cinder_volume_type parameter is configured unless there also exist these other parameters:

cinder_store_user_name
cinder_store_password
cinder_store_project_name
cinder_store_auth_address

I think this is probably a glance-store bug as looking at
https://docs.openstack.org/glance/latest/configuration/configuring.html
it seems these parameters are optional. In any case, all configuration examples in that page include them.
I'll open a bug in glance-store but I think this can be solved if the charm adds those parameters to the configuration file.

[Jorge Merlino (jorge-merlino)]

This is the glance_store bug report: https://bugs.launchpad.net/glance-store/+bug/2058069

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-glance/+/913837

[Felipe Reyes (freyes)]

@Jorge,

> I think this is probably a glance-store bug as looking at
> https://docs.openstack.org/glance/latest/configuration/configuring.html
> it seems these parameters are optional.

Can you file a bug against upstream cinder about this?

[Jorge Merlino (jorge-merlino)]

Hi all,

Just found out here:

https://docs.openstack.org/glance/latest/configuration/configuring.html#configuring-multiple-cinder-storage-backend

That these parameters I added to the charm are actually mandatory for the case of multiple cinder backends, so they are indeed needed in this case.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance/+/913837
Committed: https://opendev.org/openstack/charm-glance/commit/07da644b2ba6a2371cf7ef5249e196f18a3b8823
Submitter: "Zuul (22348)"
Branch: master

commit 07da644b2ba6a2371cf7ef5249e196f18a3b8823
Author: Jorge Merlino <email address hidden>
Date: Wed Mar 20 15:27:01 2024 -0300

    Add parameters to cinder backend configuration

    Add configuration parameters cinder_store_user_name,
    cinder_store_password, cinder_store_project_name and
    cinder_store_auth_address to cinder volume type backend configuration.
    This avoids a glance_store bug that prevents glance-api from starting.

    Closes-Bug: #2052726
    Change-Id: I4b6ca32ed426f77a326711c2457046dbda594e77

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-glance/+/916442

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-glance/+/916443

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance/+/916443
Committed: https://opendev.org/openstack/charm-glance/commit/828343670a3cafd2ae07fc6eacf74c3681e1b781
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 828343670a3cafd2ae07fc6eacf74c3681e1b781
Author: Jorge Merlino <email address hidden>
Date: Wed Mar 20 15:27:01 2024 -0300

    Add parameters to cinder backend configuration

    Add configuration parameters cinder_store_user_name,
    cinder_store_password, cinder_store_project_name and
    cinder_store_auth_address to cinder volume type backend configuration.
    This avoids a glance_store bug that prevents glance-api from starting.

    Closes-Bug: #2052726
    Change-Id: I4b6ca32ed426f77a326711c2457046dbda594e77
    (cherry picked from commit 07da644b2ba6a2371cf7ef5249e196f18a3b8823)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance/+/916442
Committed: https://opendev.org/openstack/charm-glance/commit/0ab6adc259e8d5e874787c68b05ceb33cd675b7b
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 0ab6adc259e8d5e874787c68b05ceb33cd675b7b
Author: James Page <email address hidden>
Date: Mon Feb 12 17:14:55 2024 +0000

    Fix check for internal endpoint enablement

    For cinder volume backend sections, ensure that enabling internal
    endpoint usage for service access does not completely clear the
    existing content of the section.

    Change-Id: I383a0794e924db3c7f4ce47d1e3b88355c2475e4
    Closes-Bug: 2052726
    (cherry picked from commit cfb228fc43ef29764295cefb45407162143cee09)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-glance/+/918177

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-glance/+/918178

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance/+/918177
Committed: https://opendev.org/openstack/charm-glance/commit/87209391f8a41242700626cba9c4aa2925b4951a
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 87209391f8a41242700626cba9c4aa2925b4951a
Author: Jorge Merlino <email address hidden>
Date: Wed Mar 20 15:27:01 2024 -0300

    Add parameters to cinder backend configuration

    Add configuration parameters cinder_store_user_name,
    cinder_store_password, cinder_store_project_name and
    cinder_store_auth_address to cinder volume type backend configuration.
    This avoids a glance_store bug that prevents glance-api from starting.

    Closes-Bug: #2052726
    Change-Id: I4b6ca32ed426f77a326711c2457046dbda594e77
    (cherry picked from commit 07da644b2ba6a2371cf7ef5249e196f18a3b8823)