image-volume ambiguous configuration leads to Quota exceeded for resources: ['volumes']

[Aliaksandr Vasiuk]

Hello,

Have two openstack clusters with cinder backed by pure storage arrays.
1st cloud:
Ubuntu Focal, OpenStack Ussuri, cinder charm revision 607 from ussuri/stable, cinder packages version 16.4.1-0ubuntu1, Juju "2.9.42".
2nd cloud:
Ubuntu Focal, OpenStack Ussuri, cinder charm revision 607 from ussuri/stable, cinder packages version 16.4.2-0ubuntu2.2, Juju "2.9.42".

The image-volume cache is enabled with "juju config cinder-volume image-volume-cache-enabled=true".
Every time OpenStack volume is created from an image, I see in cinder logs:
```
Failed to create new image-volume cache entry. Error: Quota exceeded for resources: ['volumes']
```
I create volumes with:
```
openstack volume create --image d906b8bd-1f7f-4e46-b01b-b147e97be33f --size 6 --type pure-c-array-iscsi foo-bar-volume
# Or a VM with bootable volume
openstack server create foo-bar-vm --flavor m1.medium --image ubuntu-20.04 --network network --boot-from-volume 6
```

I checked that cinder charm generates cinder.conf with:
```
cinder_internal_tenant_project_id = services
cinder_internal_tenant_user_id = cinder
```

However, in the documentation [1] it is recommended to use IDs.

For some reason, maybe because of some internal Keystone kitchen, we have two "services" projects on both clouds. And one of the projects has a "cinder" user and another one has none. And I checked a number of other Focal Ussuri and even newly deployed Jammy Yoga clouds, they all have two "services" projects generated by Juju.

I tried to update `cinder.conf` manually and put explicitly the ID of the "cinder" user and the ID of the project that has this user configured. And restarted the cinder-volume service. After that the "Quota exceeded for resources" vanished and the cache started working on both clouds, volumes are created in a split second.

As it looks to me the situation with two "services" projects might be quite uneasy to resolve. So the most preferable fix would be to specify the exact IDs for "internal_tenant".

[1] https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html

[Aliaksandr Vasiuk (valexby)]

Hi,

Checked a bit the charm code yesterday. I think we can fix it by using admin_tenant_id in cinder.conf template.
Basically replace:
```
cinder_internal_tenant_project_id = {{ admin_tenant_name }}
```
with
```
cinder_internal_tenant_project_id = {{ admin_tenant_id }}
```
I double-checked that we store both the name and the id in the charm relation with keystone, and we should be able to use it when rendering the config.
I'm assigning the bug to myself, and hope to come up with a resolution soon.

Best Regards,
Alex.

[Aliaksandr Vasiuk (valexby)]

Hi,

I was able to test the above change on `jammy-antelope` bundle, works great. I reproduced the bug without the change and fixed the bug with the change. Will propose the change request tomorrow.

Best Regards,
Alex.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/892764

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-cinder/+/900300

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-cinder/+/900321

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-cinder/+/900327

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/892764
Committed: https://opendev.org/openstack/charm-cinder/commit/ff442694d27e5481b9994a3dd4412a7a004d1c00
Submitter: "Zuul (22348)"
Branch: master

commit ff442694d27e5481b9994a3dd4412a7a004d1c00
Author: Erlon R. Cruz <email address hidden>
Date: Fri Oct 20 17:37:28 2023 -0300

    Use ids for cinder internal tenant and user

    admin_tenant_id and admin_user_id are more explicit than
    admin_tenant_name and admin_user as names could duplicate
    and cause issues.

    Includes sync from Charm-helpers PR #874 below

    Charm-helpers-pr: https://github.com/juju/charm-helpers/pull/874

    Closes-Bug: #2030755
    Change-Id: Idbc2f3d12dcf325b4a53a3dda1ecfa75a199295a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/900327
Committed: https://opendev.org/openstack/charm-cinder/commit/80115a2cb219c72b36adfb60b94c3cd56dbaf086
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 80115a2cb219c72b36adfb60b94c3cd56dbaf086
Author: Erlon R. Cruz <email address hidden>
Date: Fri Oct 20 17:37:28 2023 -0300

    Use admin_tenant_id as cinder internal tenant

    admin_tenant_id is more explicit than admin_tenant_name
    as project names might duplicate

    Charm-helpers-pr: https://github.com/juju/charm-helpers/pull/861

    Depends-on: I6807128492cb184de3ef5f454d0ea745246b242e

    Closes-Bug: #2030755
    Change-Id: Idbc2f3d12dcf325b4a53a3dda1ecfa75a199295a
    (cherry picked from commit ff442694d27e5481b9994a3dd4412a7a004d1c00)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/900300
Committed: https://opendev.org/openstack/charm-cinder/commit/e979c91908b0490ec78d9bc0f29254fdc38d1bb1
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit e979c91908b0490ec78d9bc0f29254fdc38d1bb1
Author: Erlon R. Cruz <email address hidden>
Date: Fri Oct 20 17:37:28 2023 -0300

    Use admin_tenant_id as cinder internal tenant

    admin_tenant_id is more explicit than admin_tenant_name
    as project names might duplicate

    Charm-helpers-pr: https://github.com/juju/charm-helpers/pull/861

    Depends-on: I6807128492cb184de3ef5f454d0ea745246b242e

    Closes-Bug: #2030755
    Change-Id: Idbc2f3d12dcf325b4a53a3dda1ecfa75a199295a
    (cherry picked from commit ff442694d27e5481b9994a3dd4412a7a004d1c00)
    (cherry picked from commit 80115a2cb219c72b36adfb60b94c3cd56dbaf086)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/900321
Committed: https://opendev.org/openstack/charm-cinder/commit/73d596bb9e65361a76a38e83bcee45c6453df09a
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 73d596bb9e65361a76a38e83bcee45c6453df09a
Author: Erlon R. Cruz <email address hidden>
Date: Fri Oct 20 17:37:28 2023 -0300

    Use admin_tenant_id as cinder internal tenant

    admin_tenant_id is more explicit than admin_tenant_name
    as project names might duplicate

    Charm-helpers-pr: https://github.com/juju/charm-helpers/pull/861

    Depends-on: I6807128492cb184de3ef5f454d0ea745246b242e

    Closes-Bug: #2030755
    Change-Id: Idbc2f3d12dcf325b4a53a3dda1ecfa75a199295a
    (cherry picked from commit ff442694d27e5481b9994a3dd4412a7a004d1c00)
    (cherry picked from commit 80115a2cb219c72b36adfb60b94c3cd56dbaf086)
    (cherry picked from commit e979c91908b0490ec78d9bc0f29254fdc38d1bb1)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-cinder/+/915461

[Edward Hope-Morley (hopem)]

The keystone patches are here: https://review.opendev.org/q/I500fd131cbd6cd5c2b38fdbe81b8b48e50a3e3f7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-keystone/+/900230
Committed: https://opendev.org/openstack/charm-keystone/commit/eb11ca2bdae311ac76209736b32def8ffb93abcd
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit eb11ca2bdae311ac76209736b32def8ffb93abcd
Author: Erlon R. Cruz <email address hidden>
Date: Fri Oct 20 17:55:10 2023 -0300

    Adds service_user_id into relation data

    This is necessary to avoid collisions between
    same usernames used service users.

    Depends-on: I4fbfa8fba84b11c4e30e4db9a0c358db1e8c94f1
    Closes-Bug: #2030755
    Change-Id: I500fd131cbd6cd5c2b38fdbe81b8b48e50a3e3f7
    (cherry picked from commit 86a323abfe852f4756bf198a0654e09cbce13782)
    (cherry picked from commit 47942040316733dd16dadb3881dbfc3833cc11f2)
    (cherry picked from commit bdf00de6065b22b91e831b7fe883fefdafe118c3)
    (cherry picked from commit 3eebc3ad6a84e6f23b6b9b07fd7ec19ccbe49ce0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/915461
Committed: https://opendev.org/openstack/charm-cinder/commit/d36cf9835723dc5aa4ec91f448a27a8d336e2463
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit d36cf9835723dc5aa4ec91f448a27a8d336e2463
Author: Erlon R. Cruz <email address hidden>
Date: Fri Oct 20 17:37:28 2023 -0300

    Use admin_tenant_id as cinder internal tenant

    admin_tenant_id is more explicit than admin_tenant_name
    as project names might duplicate

    This commit is related to code rdata.get -> _resolve.
    Related commits are below but it is long.
    So I changed _resolve to rdata.get here.

    3faf38d3d8fa738d0582b4537e77eaef1d5c332a IndentityServiceContext app data fixes
    4137a3d5d676ed6f332504b4edfcdf7f06c2f0bc
    openstack: identity-service forwards compat (#730)

    Charm-helpers-pr: https://github.com/juju/charm-helpers/pull/894

    Depends-on: I6807128492cb184de3ef5f454d0ea745246b242e

    Closes-Bug: #2030755
    Change-Id: Idbc2f3d12dcf325b4a53a3dda1ecfa75a199295a
    (cherry picked from commit ff442694d27e5481b9994a3dd4412a7a004d1c00)
    (cherry picked from commit 80115a2cb219c72b36adfb60b94c3cd56dbaf086)
    (cherry picked from commit e979c91908b0490ec78d9bc0f29254fdc38d1bb1)
    (cherry picked from commit 73d596bb9e65361a76a38e83bcee45c6453df09a)