Bug #1939389 “Missing [nova] section in cinder.conf “Extend an A...” : Bugs : OpenStack Cinder Charm

Alex Kavanagh (ajkavanagh) on 2021-08-10

Changed in charm-cinder:
status:	New → Triaged
importance:	Undecided → High

Revision history for this message

Alex Kavanagh (ajkavanagh) wrote on 2021-08-10:

#1

So it looks like the [nova] section was added a while back, and the config keys that are used were removed from cinder in the train release[1]. The commit that adds the [nova] section [2].

Were going to have to add a [nova] section to the cinder.conf (as the bug reporter indicates) with the relevant details. This should be backported to the Train release.

1. https://docs.openstack.org/releasenotes/puppet-cinder/train.html#relnotes-15-4-0-stable-train
2. https://review.opendev.org/c/openstack/cinder/+/460640

Revision history for this message

Samuel Allan (samuelallan) wrote on 2022-04-05:

#2

Could you provide instructions to reproduce the issue?

I tried the following on focal-xena to create a volume, attach it to an instance, then resize it:

```
openstack volume create --size 1 vol1
openstack server add volume juju-7a66ea-k8s-on-openstack-0 vol1
cinder reset-state --state available vol1
openstack volume set vol1 --size 2
```

This completed successfully without errors (at least, the commands returned successfully, no errors in /var/log/cinder/cinder-*.log).

Revision history for this message

Samuel Allan (samuelallan) wrote on 2022-04-05:

#3

(This doesn't negate the fact that we still need to add support for the [nova] section, but I'm trying to figure out how to test it.)

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-04-05:

#4

I've added a snippet from the other bug filed at the same time:
https://bugs.launchpad.net/ubuntu/+source/python-os-brick/+bug/1939390
> Steps to reproduce:
>
> openstack volume create --type my-iscsi-volume-type --size 10 test-resize
> openstack server add volume test-server test-resize
> openstack volume set --size 20 test-resize

I suppose the key to reproduce it is either online resizing or using iSCSI backend. But if it's the former, it would be easier to test it with Ceph RBD backend.

description:

updated

Revision history for this message

Gabriel Samfira (gabriel-samfira) wrote on 2022-04-05:

#5

Hi Samuel,

The volume needs to be "in-use". I see you are resetting it's state to "available". When in-use, cinder will attempt to notify nova to refresh the size of the disk. See:

https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L2939-L2943

On iSCSI backends the volume needs to first be refreshed on the host before the VM can be made aware of the new size. I suspect there may be some operation that needs to be done on RBD backends as well, but I cannot say for sure.

To reproduce, I recommend you use any iSCSI backend. We used a Dell Unity, but any iSCSI backend will do.

Revision history for this message

Samuel Allan (samuelallan) wrote on 2022-04-05:

#6

Thanks Nobuto, Gabriel!

> The volume needs to be "in-use". I see you are resetting it's state to "available".

Ah that makes more sense. When I was testing, openstack refused to change the volume size while in-use: "Volume is in in-use state, it must be available before size can be extended". Will need to try a different backend.

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-04-06:

#7

https://docs.openstack.org/cinder/latest/cli/cli-manage-volumes.html#extend-attached-volume
`--os-volume-api-version 3.42` or later was the key to allow resize for the record.

description:

updated

Revision history for this message

Samuel Allan (samuelallan) wrote on 2022-04-06 (last edit on 2022-04-06):

#8

So I tested with cinder-lvm, which using an iSCSI backend, and couldn't reproduce the issue. I then dug into the code and discovered that cinder appears to try to autodetect the keystone auth parameters if it doesn't find data in the [nova] section. (ref. code here: https://opendev.org/openstack/cinder/src/commit/85387cae8b3641d6d193c6550edbfb0de1d8384e/cinder/compute/nova.py#L98-L106 ).

So it may be difficult to reproduce the exact issue that Gabriel reported without knowing more about the openstack configuration (eg. regions, etc.).

However, I think this is missing the point, which is that the [nova] section is indeed missing and not configurable from juju for cinder.conf (and absolutely appears to be what is causing the bug). I'm not sure how we want this section to be configurable though. There are a lot of options:

[nova]
# locally declared in cinder:compute/nova.py
region_name =
interface =
token_auth_url =

# declared in keystoneauth1:loading/session.py
cafile =
certfile =
keyfile =
insecure =
timeout =
collect-timing =
split-loggers =

# declared in keystoneauth1:loading/conf.py
auth_type =
auth_section =

---

Which do we want to expose as-is to the charm config, vs exposing a higher level config option, vs other methods?

Revision history for this message

Gabriel Samfira (gabriel-samfira) wrote on 2022-04-06:

#9

The deployment is single region, with OpenStack Victoria. The config, apart from the [nova] section I added, is the one generated by the juju charms.

I created a template for victoria, which is a copy o the Queens config, with the following section added:

[nova]
{% include "parts/keystone-authtoken" %}

Ant the contents of that parts file is:

# Authentication type to load (string value)
auth_type = password

# Authentication URL (string value)
auth_url = {{ auth_protocol }}://{{ auth_host }}:{{ auth_port }}

# Username (string value)
username = {{ admin_user }}

# User's password (string value)
password = {{ admin_password }}

# Project name to scope to (string value)
project_name = {{ admin_tenant_name }}

{% if admin_domain_name -%}
project_domain_name = {{ admin_domain_name }}
user_domain_name = {{ admin_domain_name }}
{% else -%}
project_domain_name = default
user_domain_name = default
{% endif -%}

{% if ca_cert_path -%}
# PEM encoded Certificate Authority to use when verifying
# HTTPs connections. (string value)
cafile = {{ ca_cert_path }}
{% endif -%}

region_name = {{ region }}

After a charm upgrade, the error was gone, and volumes were properly refreshed. Not sure if there is anything that is missing.

Samuel Allan (samuelallan) on 2022-04-11

description:

updated

Revision history for this message

Samuel Allan (samuelallan) wrote on 2022-04-11:

#10

Update: have successfully reproduced this bug on Victoria and Wallaby. However I couldn't reproduce it on Xena.

Revision history for this message

Nobuto Murata (nobuto) wrote on 2022-04-12:

#11

Samuel has reproduced the issue on Xena finally so we will take a look when we have some cycles.

Revision history for this message

Samuel Allan (samuelallan) wrote on 2022-04-13:

#12

Download full text (12.8 KiB)

I've been trying to figure out a solution here.
We need to somehow get `project_domain_id` or `project_domain` into the context here:

2022-04-12 23:43:07.956 151261 WARNING cinder.compute.nova [req-a23f3da3-3004-4798-b248-3348fe1309ef ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -] XXX context = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6LQxoGaMiTVufoGBn1nA7drrEM9k4cf4CGNtZyM-iw68RLEO2f9OPZ_0_6WzD7BLwPYSO9XpKUBoVVDchWqcefKctRcsb_wStkK4B4eJd64pTxEC6lSH5iExWwUpH712j_XNdjo8lwMY_Cr2xjwAWBtqZo9fb7WiqLp7P1vSxN27E', 'request_id': 'req-a23f3da3-3004-4798-b248-3348fe1309ef', 'global_request_id': 'req-a718a415-0414-4a86-b4bd-87c8f5c832e4', 'resource_uuid': None, 'roles': ['Admin', 'reader', 'member', 'load-balancer_admin', 'admin'], 'user_identity': 'ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -', 'is_admin_project': True, 'user_id': 'ff32a8ed05614f7a8f615be6da92fa18', 'project_id': '91468a8dcfa94fa98db004ff4c175a9b', 'project_name': 'admin', 'domain_id': None, 'read_deleted': 'no', 'remote_address': '192.168.151.129', 'timestamp': '2022-04-12T23:43:07.378961+00:00', 'quota_class': None, 'service_catalog': [{'type': 'identity', 'name': 'keystone', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.156:35357/v3', 'publicURL': 'https://192.168.151.156:5000/v3', 'internalURL': 'https://192.168.151.156:5000/v3'}]}, {'type': 'compute', 'name': 'nova', 'endpoints': [{'region': 'RegionOne', 'publicURL': 'https://192.168.151.108:8774/v2.1', 'internalURL': 'https://192.168.151.108:8774/v2.1', 'adminURL': 'https://192.168.151.108:8774/v2.1'}]}, {'type': 'object-store', 'name': 'swift', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.138:443/swift', 'internalURL': 'https://192.168.151.138:443/swift/v1', 'publicURL': 'https://192.168.151.138:443/swift/v1'}]}, {'type': 'key-manager', 'name': 'barbican', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.109:9311', 'publicURL': 'https://192.168.151.109:9311', 'adminURL': 'https://192.168.151.109:9312'}]}, {'type': 'image', 'name': 'glance', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.110:9292', 'publicURL': 'https://192.168.151.110:9292', 'adminURL': 'https://192.168.151.110:9292'}]}]}

I could trace the context as far back as in oslo messaging rpc message received:

2022-04-12 23:43:07.506 151261 WARNING oslo_messaging.rpc.server [-] ZZZ message.ctxt = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6LQxoGaMiTVufoGBn1nA7drrEM9k4cf4CGNtZyM-iw68RLEO2f9OPZ_0_6WzD7BLwPYSO9XpKUBoVVDchWqcefKctRcsb_wStkK4B4eJd64pTxEC6lSH5iExWwUpH712j_XNdjo8lwMY_Cr2xjwAWBtqZo9fb7W...

I've been trying to figure out a solution here.
We need to somehow get `project_domain_id` or `project_domain` into the context here:

2022-04-12 23:43:07.956 151261 WARNING cinder.compute.nova [req-a23f3da3-3004-4798-b248-3348fe1309ef ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -] XXX context = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6LQxoGaMiTVufoGBn1nA7drrEM9k4cf4CGNtZyM-iw68RLEO2f9OPZ_0_6WzD7BLwPYSO9XpKUBoVVDchWqcefKctRcsb_wStkK4B4eJd64pTxEC6lSH5iExWwUpH712j_XNdjo8lwMY_Cr2xjwAWBtqZo9fb7WiqLp7P1vSxN27E', 'request_id': 'req-a23f3da3-3004-4798-b248-3348fe1309ef', 'global_request_id': 'req-a718a415-0414-4a86-b4bd-87c8f5c832e4', 'resource_uuid': None, 'roles': ['Admin', 'reader', 'member', 'load-balancer_admin', 'admin'], 'user_identity': 'ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -', 'is_admin_project': True, 'user_id': 'ff32a8ed05614f7a8f615be6da92fa18', 'project_id': '91468a8dcfa94fa98db004ff4c175a9b', 'project_name': 'admin', 'domain_id': None, 'read_deleted': 'no', 'remote_address': '192.168.151.129', 'timestamp': '2022-04-12T23:43:07.378961+00:00', 'quota_class': None, 'service_catalog': [{'type': 'identity', 'name': 'keystone', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.156:35357/v3', 'publicURL': 'https://192.168.151.156:5000/v3', 'internalURL': 'https://192.168.151.156:5000/v3'}]}, {'type': 'compute', 'name': 'nova', 'endpoints': [{'region': 'RegionOne', 'publicURL': 'https://192.168.151.108:8774/v2.1', 'internalURL': 'https://192.168.151.108:8774/v2.1', 'adminURL': 'https://192.168.151.108:8774/v2.1'}]}, {'type': 'object-store', 'name': 'swift', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.138:443/swift', 'internalURL': 'https://192.168.151.138:443/swift/v1', 'publicURL': 'https://192.168.151.138:443/swift/v1'}]}, {'type': 'key-manager', 'name': 'barbican', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.109:9311', 'publicURL': 'https://192.168.151.109:9311', 'adminURL': 'https://192.168.151.109:9312'}]}, {'type': 'image', 'name': 'glance', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.110:9292', 'publicURL': 'https://192.168.151.110:9292', 'adminURL': 'https://192.168.151.110:9292'}]}]}

I could trace the context as far back as in oslo messaging rpc message received:

2022-04-12 23:43:07.506 151261 WARNING oslo_messaging.rpc.server [-] ZZZ message.ctxt = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6LQxoGaMiTVufoGBn1nA7drrEM9k4cf4CGNtZyM-iw68RLEO2f9OPZ_0_6WzD7BLwPYSO9XpKUBoVVDchWqcefKctRcsb_wStkK4B4eJd64pTxEC6lSH5iExWwUpH712j_XNdjo8lwMY_Cr2xjwAWBtqZo9fb7WiqLp7P1vSxN27E', 'request_id': 'req-a23f3da3-3004-4798-b248-3348fe1309ef', 'global_request_id': 'req-a718a415-0414-4a86-b4bd-87c8f5c832e4', 'resource_uuid': None, 'roles': ['Admin', 'reader', 'member', 'load-balancer_admin', 'admin'], 'user_identity': 'ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -', 'is_admin_project': True, 'user_id': 'ff32a8ed05614f7a8f615be6da92fa18', 'project_id': '91468a8dcfa94fa98db004ff4c175a9b', 'project_name': 'admin', 'domain_id': None, 'read_deleted': 'no', 'remote_address': '192.168.151.129', 'timestamp': '2022-04-12T23:43:07.378961+00:00', 'quota_class': None, 'service_catalog': [{'type': 'identity', 'name': 'keystone', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.156:35357/v3', 'publicURL': 'https://192.168.151.156:5000/v3', 'internalURL': 'https://192.168.151.156:5000/v3'}]}, {'type': 'compute', 'name': 'nova', 'endpoints': [{'region': 'RegionOne', 'publicURL': 'https://192.168.151.108:8774/v2.1', 'internalURL': 'https://192.168.151.108:8774/v2.1', 'adminURL': 'https://192.168.151.108:8774/v2.1'}]}, {'type': 'object-store', 'name': 'swift', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.138:443/swift', 'internalURL': 'https://192.168.151.138:443/swift/v1', 'publicURL': 'https://192.168.151.138:443/swift/v1'}]}, {'type': 'key-manager', 'name': 'barbican', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.109:9311', 'publicURL': 'https://192.168.151.109:9311', 'adminURL': 'https://192.168.151.109:9312'}]}, {'type': 'image', 'name': 'glance', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.110:9292', 'publicURL': 'https://192.168.151.110:9292', 'adminURL': 'https://192.168.151.110:9292'}]}], 'client_timeout': None}

Context is built somewhere, external to cinder, and sent via amqp message?
Need to figure out how this context is built.

Sometimes in context there is `'project_domain': None`,
and sometimes it's `'project_domain': 'c2a7d9d1e315405ca054dda569aefa93'` (where the value is the correct domain id).

eg.:

2022-04-12 23:43:00.430 151261 WARNING oslo_messaging.rpc.server [-] ZZZ message.ctxt = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': 'c2a7d9d1e315405ca054dda569aefa93', 'project_domain': 'c2a7d9d1e315405ca054dda569aefa93', 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6E4AmOEgJHlkHFibNFAlt3TeIbfBdLIO-jiyDVQVfjCchNoGVsb1LpOchEaTWPqGcNE1HlGkGakNkgfACpHPoWfyivaemK9Ke7JNTQDDJVSHwciLtcNN_WgWeeKU9E1BOjNzVaX6OdVwJpnnZ0PXjwg_G4OESery2cdCQ1HgRKt5c', 'request_id': 'req-f9c43b35-5153-42b0-bb30-ed878428b5c5', 'global_request_id': 'req-f79b421a-998b-464c-929a-3d93177e06b4', 'resource_uuid': None, 'roles': ['Admin', 'reader', 'member', 'load-balancer_admin'], 'user_identity': 'ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - c2a7d9d1e315405ca054dda569aefa93 c2a7d9d1e315405ca054dda569aefa93', 'is_admin_project': True, 'user_id': 'ff32a8ed05614f7a8f615be6da92fa18', 'project_id': '91468a8dcfa94fa98db004ff4c175a9b', 'project_name': 'admin', 'domain_id': None, 'read_deleted': 'no', 'remote_address': '192.168.151.129', 'timestamp': '2022-04-12T23:43:00.333440', 'quota_class': None, 'service_catalog': [{'type': 'identity', 'name': 'keystone', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.156:35357/v3', 'publicURL': 'https://192.168.151.156:5000/v3', 'internalURL': 'https://192.168.151.156:5000/v3'}]}, {'type': 'compute', 'name': 'nova', 'endpoints': [{'region': 'RegionOne', 'publicURL': 'https://192.168.151.108:8774/v2.1', 'internalURL': 'https://192.168.151.108:8774/v2.1', 'adminURL': 'https://192.168.151.108:8774/v2.1'}]}, {'type': 'object-store', 'name': 'swift', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.138:443/swift', 'internalURL': 'https://192.168.151.138:443/swift/v1', 'publicURL': 'https://192.168.151.138:443/swift/v1'}]}, {'type': 'key-manager', 'name': 'barbican', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.109:9311', 'publicURL': 'https://192.168.151.109:9311', 'adminURL': 'https://192.168.151.109:9312'}]}, {'type': 'image', 'name': 'glance', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.110:9292', 'publicURL': 'https://192.168.151.110:9292', 'adminURL': 'https://192.168.151.110:9292'}]}], 'client_timeout': None}

So what's the difference between the two?
Why does the context sometimes get populated correctly with the domain information (project_domain, user_domain), and other times not?
It seems to be roughly 50% of the time, which may suggest some kind of data race / multithreading error,
but then again it's not populating the project_domain 100% of the time for resizing the lvm volume.

On the oslo_messaging side, the `ctxt` appears to be a standard python dictionary.
By the time it gets to cinder/compute/nova.py, it's a cinder.context.RequestContext, which is a subclass of the oslo RequestContext ( https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py ).

Manually adding a "project_domain_id" key with the correct value to the ctxt in oslo_messaging in /usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py fixes the issue.

So the question is: where should project_domain_id be coming from?
The nova section in cinder.conf appears to be a red herring;
that doesn't officially have a project_domain_id section.
I'm not sure about the section that Gabriel added to the file.
Other similar keystone-authtoken part files begin with the [keystone_authtoken] section header.

https://opendev.org/openstack/cinder/src/commit/1eb7384cb6542784a904891666cca4abbe42841d/cinder/api/middleware/auth.py#L112-L113 looks promising,
but I added logging there and it was never called.

Potentially interesting sections of cinder.conf:

```
[keystone_authtoken]
auth_type = password
auth_uri = https://192.168.151.156:5000/v3
auth_url = https://192.168.151.156:35357/v3
# should this be admin_domain?  I tried changing this but it didn't appear to help
project_domain_name = service_domain
user_domain_name = service_domain
project_name = services
```

According to https://docs.openstack.org/cinder/ussuri/_static/cinder.conf.sample , there is support for a [service_user].project_domain_id option,
but setting that also didn't help.

Example traceback extended back further to oslo messaging, to figure out where the context comes from.

File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
    result = func(ctxt, **new_args)
  File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 2941, in extend_volume
    nova_api.extend_volume(context, instance_uuids, volume.id)
  File "/usr/lib/python3/dist-packages/cinder/compute/nova.py", line 216, in extend_volume
    result = self._send_events(context, events, api_version=api_version)
  File "/usr/lib/python3/dist-packages/cinder/compute/nova.py", line 151, in _send_events
    response = nova.server_external_events.create(events)
  File "/usr/lib/python3/dist-packages/novaclient/v2/server_external_events.py", line 38, in create
    return self._create('/os-server-external-events', body, 'events',
  File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 401, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3/dist-packages/novaclient/client.py", line 69, in request
    resp, body = super(SessionClient, self).request(url,
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 554, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request
    auth_headers = self.get_auth_headers(auth)
  File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers
    return auth.get_headers(self, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 106, in get_headers
    token = self.get_token(session)
  File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
    return self.get_access(session).auth_token
  File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
    resp = session.post(token_url, json=body, headers=headers,
  File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-de6f6d63-40a6-4418-bea3-f924622b1595)

Revision history for this message

Gabriel Samfira (gabriel-samfira) wrote on 2022-04-13:

#13

In the above mentioned template, there is a "project_domain_name" and "user_domain_name" set if "admin_domain_name" is set. With the project_domain_name set, the keystone client used by the nova client should have the proper context.

I would suggest you do a:

charm pull cinder

add the template mentioned above, then do an upgrade of the charm using the locally modified version.

After hooks run, you should have the proper nova section with the needed settings, and the error should no longer manifest, as the nova client will have proper credentials.

Revision history for this message

Gabriel Samfira (gabriel-samfira) wrote on 2022-04-13:

#14

Adding a bit of context.

All OpenStack projects use the same oslo-config boilerplate to provide credentials for the various clients a service may use. In this case, Cinder needs to talk to Nova, so a nova client needs to be instantiated. The sections for the various clients are identical. The only difference is the group name (eg: [nova], [neutron], etc). These individual sections are so alike, that in more recent versions of OpenStack, you can chose to either customize them (you may want a different Region for one of the clients), or use the "auth_section" option.

example:

[nova]
auth_section = "keystone_auth"

[keystone_auth]
.... The template contents from previous comment .....

In the cinder code, this is used here (for nova at least):

https://github.com/openstack/cinder/blob/master/cinder/compute/nova.py#L115-L129

A keystone session is created with the credentials from [nova] and that session is passed into the nova client.

In general, if a service needs to talk to another service, for whatever reason, a proper [<service_name>] section needs to exist in the config with the proper keystone credentials (identical most times to all other services).

Revision history for this message

Samuel Allan (samuelallan) wrote on 2022-04-13:

#15

Ah thanks for the context Gabriel! That is something I had no idea about, and haven't read that in the documentation anywhere.

I'm still a little hazy on exactly what modifications you made; could you share a diff?

Something that bugs me though, is that it's not missing any auth params except for the project domain id. Why is it not getting the project domain id for this case only? It expects to receive it in the context, so it can use it here: https://opendev.org/openstack/cinder/src/commit/1eb7384cb6542784a904891666cca4abbe42841d/cinder/compute/nova.py#L107-L110

Revision history for this message

Gabriel Samfira (gabriel-samfira) wrote on 2022-04-14:

#16

My pleasure! :)

That particular bit of code, should not be hit, unless you don't have "auth_type" set in the config. The cinder user *should* be privileged I think.

Here is a quick diff of the stuff I posted in comment #9:

https://github.com/gabriel-samfira/charm-cinder/commit/82505257b129db91b1aabeac744e02c900852287

Give this a shot in your env, see if it fixes it. I wrote this a while ago. If the charm template context changed, you may need to adapt it. A relation to keystone should ensure that context exists.

Revision history for this message

Samuel Allan (samuelallan) wrote on 2022-04-27:

#17

Update: I managed to successfully reproduce the issue and Gabriel's fix by cherry picking https://github.com/gabriel-samfira/charm-cinder/commit/82505257b129db91b1aabeac744e02c900852287 on stable/xena. But now I have two questions:

- With the fix there is no more traceback or crash, but I'm not sure I can confirm it's actually successfully notified the instance. If I run `lsblk` on the openstack instance after live resizing the instance, it doesn't reflect the updated size. It does show the updated size if I reboot the openstack instance though.

- Is this how we want to fix the issue? It seems that if it does support the `auth_section = "SECTION_NAME"` format, this would result in less duplication. However I couldn't get it to work that way, and I can't find docs about how this is supposed to work. Also https://github.com/openstack/cinder/blob/master/cinder/compute/nova.py#L115-L129 directly accesses the nova group from config; I don't see if/where it resolves through an `auth_section` key.

Revision history for this message

Gabriel Samfira (gabriel-samfira) wrote on 2022-04-27:

#18

Hi Samuel,

Would you mind sharing the output of lsblk and fdisk -l before and after live resizing the disk?

On the VM, after resize, could you rescan the disk by doing a:

echo 1>/sys/class/block/sdx/device/rescan

where sdx is youd block device.

In regards to the second point, I would simply use the same keystone auth "part" in that patch. Other charms have similar chunks of template that get added to the configs. It may be more verbose, but Feel free to use whichever approach seems better to you.

OpenStack Infra (hudson-openstack) on 2022-05-22

Changed in charm-cinder:
status:	Triaged → In Progress

Revision history for this message

Samuel Allan (samuelallan) wrote on 2022-05-22:

#19

Opened https://review.opendev.org/c/openstack/charm-cinder/+/842902

Samuel Allan (samuelallan) on 2022-06-06

Changed in charm-cinder:
assignee:	nobody → Samuel Walladge (swalladge)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-06-30: Fix merged to charm-cinder (master)

#20

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/842902
Committed: https://opendev.org/openstack/charm-cinder/commit/024de3783fc459fc25596ec6084dd065e945df76
Submitter: "Zuul (22348)"
Branch: master

commit 024de3783fc459fc25596ec6084dd065e945df76
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

This issue has been observed on victoria and newer,
hence the addition of cinder.conf for victoria.

func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/776

Closes-Bug: #1939389

Co-authored-by: Samuel Walladge <email address hidden>

Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1

Changed in charm-cinder:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-07-07: Related fix proposed to charm-cinder (master)

#21

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/848914

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-07-07: Fix proposed to charm-cinder (stable/yoga)

#22

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-cinder/+/848915

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-07-07: Related fix proposed to charm-cinder (stable/yoga)

#23

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-cinder/+/848916

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-25: Fix proposed to charm-cinder (stable/xena)

#24

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-cinder/+/854593

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-25: Fix proposed to charm-cinder (stable/wallaby)

#25

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-cinder/+/854594

Nobuto Murata (nobuto) on 2022-08-26

summary:	- Missing [nova] section in cinder.conf + Missing [nova] section in cinder.conf "Extend an Attached Volume" + doesn't work with Victoria onwards
description:	updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-09-05: Related fix merged to charm-cinder (master)

#26

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/848914
Committed: https://opendev.org/openstack/charm-cinder/commit/ba21da64f6a547bd44956d71933030d529956bcd
Submitter: "Zuul (22348)"
Branch: master

commit ba21da64f6a547bd44956d71933030d529956bcd
Author: Nobuto Murata <email address hidden>
Date: Thu Jul 7 09:37:28 2022 +0900

Re-enable AppArmor enforcement

    Ensure tests are run with aa-profile-mode=enforce. So we should be able
    to notice any future behavioral changes in the upstream or additional
    rules required for nova-compute AppArmor profile in the functional
    tests.

func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/870

    Depends-Onː Id2db3a70b8d1287bda006f1bbc5442038f7070f1
    Related-Bug: #1979812
    Related-Bug: #1939389

Change-Id: If658d9e8ee11248ef09e774f3e21fe0f801dc066

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-09-08: Fix merged to charm-cinder (stable/yoga)

#27

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/848915
Committed: https://opendev.org/openstack/charm-cinder/commit/ed0f80b7650bd03266897c0f17aa7d84ba9711fd
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit ed0f80b7650bd03266897c0f17aa7d84ba9711fd
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

This issue has been observed on victoria and newer,
hence the addition of cinder.conf for victoria.

Closes-Bug: #1939389

Co-authored-by: Samuel Walladge <email address hidden>

Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1
(cherry picked from commit 024de3783fc459fc25596ec6084dd065e945df76)

tags:

added: in-stable-yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-09-12: Related fix merged to charm-cinder (stable/yoga)

#28

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/848916
Committed: https://opendev.org/openstack/charm-cinder/commit/27c1c0a053545b18b1802d943b296c39c6aa4706
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 27c1c0a053545b18b1802d943b296c39c6aa4706
Author: Nobuto Murata <email address hidden>
Date: Thu Jul 7 09:37:28 2022 +0900

Re-enable AppArmor enforcement

    Ensure tests are run with aa-profile-mode=enforce. So we should be able
    to notice any future behavioral changes in the upstream or additional
    rules required for nova-compute AppArmor profile in the functional
    tests.

func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/871

    Depends-Onː Id2db3a70b8d1287bda006f1bbc5442038f7070f1
    Related-Bug: #1979812
    Related-Bug: #1939389

Change-Id: If658d9e8ee11248ef09e774f3e21fe0f801dc066
(cherry picked from commit ba21da64f6a547bd44956d71933030d529956bcd)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-09-13: Related fix proposed to charm-cinder (stable/xena)

#29

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-cinder/+/857343

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-09-13: Related fix proposed to charm-cinder (stable/wallaby)

#30

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-cinder/+/857348

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-10: Fix merged to charm-cinder (stable/xena)

#31

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/854593
Committed: https://opendev.org/openstack/charm-cinder/commit/c77025825b21e3707d19416e9c136b01872144fc
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c77025825b21e3707d19416e9c136b01872144fc
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

This issue has been observed on victoria and newer,
hence the addition of cinder.conf for victoria.

Closes-Bug: #1939389

Co-authored-by: Samuel Walladge <email address hidden>

Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1

tags:

added: in-stable-xena

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-10: Related fix merged to charm-cinder (stable/xena)

#32

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/857343
Committed: https://opendev.org/openstack/charm-cinder/commit/850bcaa45213edfff2cc43ddb2d8dba76b9bf9f5
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 850bcaa45213edfff2cc43ddb2d8dba76b9bf9f5
Author: Nobuto Murata <email address hidden>
Date: Thu Jul 7 09:37:28 2022 +0900

Re-enable AppArmor enforcement

    Ensure tests are run with aa-profile-mode=enforce. So we should be able
    to notice any future behavioral changes in the upstream or additional
    rules required for nova-compute AppArmor profile in the functional
    tests.

func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/906

Related-Bug: #1979812
Related-Bug: #1939389

Change-Id: If658d9e8ee11248ef09e774f3e21fe0f801dc066

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-10: Fix merged to charm-cinder (stable/wallaby)

#33

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/854594
Committed: https://opendev.org/openstack/charm-cinder/commit/2b71f3b1d7bdd8a0902cd6439bc2f66e797293fc
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 2b71f3b1d7bdd8a0902cd6439bc2f66e797293fc
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

This issue has been observed on victoria and newer,
hence the addition of cinder.conf for victoria.

Closes-Bug: #1939389

Co-authored-by: Samuel Walladge <email address hidden>

Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1
(cherry picked from commit 024de3783fc459fc25596ec6084dd065e945df76)

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-10: Related fix merged to charm-cinder (stable/wallaby)

#34

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/857348
Committed: https://opendev.org/openstack/charm-cinder/commit/aa96cb0efd86136a8f6eaaa935b7526a47e81a8f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit aa96cb0efd86136a8f6eaaa935b7526a47e81a8f
Author: Nobuto Murata <email address hidden>
Date: Thu Jul 7 09:37:28 2022 +0900

Re-enable AppArmor enforcement

    Ensure tests are run with aa-profile-mode=enforce. So we should be able
    to notice any future behavioral changes in the upstream or additional
    rules required for nova-compute AppArmor profile in the functional
    tests.

func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/907

Related-Bug: #1979812
Related-Bug: #1939389

Change-Id: If658d9e8ee11248ef09e774f3e21fe0f801dc066
(cherry picked from commit ba21da64f6a547bd44956d71933030d529956bcd)

Revision history for this message

Marcin Wilk (wilkmarcin) wrote on 2022-11-21:

#35

Download full text (3.8 KiB)

Reproduced the same problem on Ussuri when trying to extend already attached volume as a regular user (a user assigned 'member' role only). Steps to recreate:

1. OpenStack Ussuri, charm: cinder| ussuri/edge | rev. 568
2. create a volume and attach it to a VM
3. as a user with the 'member' role assigned to the project try to extend the volume:
`cinder extend <vol id> <new size>
4. the command completes with the rc 0 (all good), `openstack volume show` shows new size but cinder-volume.log contains following error and the volume size from the VM perspective remains unchanged:
#### actual 403 error message from nova #####
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova [req-13d1093e-9a7f-4f03-b505-c985d9d45691 754c0e6eae22488982f393b69e9800a3 d6761dacbb0649189a07a4a1a191a8c0 - 6a4439622e71431c8b96073e33c3b7e1 6a4439622e71431c8b96073e33c3b7e1] Failed to notify nova on events: [{'name': 'volume-extended', 'server_uuid': '3d51176f-281c-4ab3-8631-042ccd407d1d', 'tag': '99c14113-f1ed-4e28-a9b9-54c7eb862db8'}].: novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-6a908980-00ae-455e-b8fb-aab951032251)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova Traceback (most recent call last):
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/cinder/compute/nova.py", line 149, in _send_events
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova response = nova.server_external_events.create(events)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/novaclient/v2/server_external_events.py", line 38, in create
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova return self._create('/os-server-external-events', body, 'events',
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova resp, body = self.api.client.post(url, body=body)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova return self.request(url, 'POST', **kwargs)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova File "/usr/lib/python3/dist-packages/novaclient/client.py", line 78, in request
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova raise exceptions.from_response(resp, body, url, method)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-6a908980-00ae-455e-b8fb-aab951032251)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova

#### successful completion - that is wrong ####
2022-11-21 12:53:22.880 1243598 INFO cinder.volume.manager [req-13d1093e-9a7f-4f03-b505-c985d9d45691 754c0e6eae22488982f393b69e9800a3 d6761dacbb0649189a07a4a1a191a8c0 - 6a4439622e71431c8b96073e33c3b7e1 6a4439622e71431c8b96073e33c3b7e1] E...

Reproduced the same problem on Ussuri when trying to extend already attached volume as a regular user (a user assigned 'member' role only). Steps to recreate:

1. OpenStack Ussuri, charm: cinder| ussuri/edge | rev. 568
2. create a volume and attach it to a VM
3. as a user with the 'member' role assigned to the project try to extend the volume:
`cinder extend <vol id> <new size>
4. the command completes with the rc 0 (all good), `openstack volume show` shows new size but cinder-volume.log contains following error and the volume size from the VM perspective remains unchanged:
#### actual 403 error message from nova #####
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova [req-13d1093e-9a7f-4f03-b505-c985d9d45691 754c0e6eae22488982f393b69e9800a3 d6761dacbb0649189a07a4a1a191a8c0 - 6a4439622e71431c8b96073e33c3b7e1 6a4439622e71431c8b96073e33c3b7e1] Failed to notify nova on events: [{'name': 'volume-extended', 'server_uuid': '3d51176f-281c-4ab3-8631-042ccd407d1d', 'tag': '99c14113-f1ed-4e28-a9b9-54c7eb862db8'}].: novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-6a908980-00ae-455e-b8fb-aab951032251)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova Traceback (most recent call last):
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova   File "/usr/lib/python3/dist-packages/cinder/compute/nova.py", line 149, in _send_events
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova     response = nova.server_external_events.create(events)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova   File "/usr/lib/python3/dist-packages/novaclient/v2/server_external_events.py", line 38, in create
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova     return self._create('/os-server-external-events', body, 'events',
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova   File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova     resp, body = self.api.client.post(url, body=body)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova   File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova     return self.request(url, 'POST', **kwargs)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova   File "/usr/lib/python3/dist-packages/novaclient/client.py", line 78, in request
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova     raise exceptions.from_response(resp, body, url, method)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-6a908980-00ae-455e-b8fb-aab951032251)
2022-11-21 12:53:22.806 1243598 ERROR cinder.compute.nova

#### successful completion - that is wrong ####
2022-11-21 12:53:22.880 1243598 INFO cinder.volume.manager [req-13d1093e-9a7f-4f03-b505-c985d9d45691 754c0e6eae22488982f393b69e9800a3 d6761dacbb0649189a07a4a1a191a8c0 - 6a4439622e71431c8b96073e33c3b7e1 6a4439622e71431c8b96073e33c3b7e1] Extend volume completed successfully.

This is because if the [nova] section in the cinder.conf is missing, the api request to the nova-api from the cinder is executed as the user who initiated the cinder cli call (in the case above its 754c0e6eae22488982f393b69e9800a3) who does not have privileges to execute 'volume-extend'.

Adding following [nova] section to the cinder.conf solves the problem:
[nova]
url = <nova api url>
region_name = <region>
auth_url = <keystone api url>
auth_type = password
project_domain_name = <project_domain>
user_domain_name = <nova_user_domain>
project_name = <project_name>
username = nova
password = <password>

Please provide the fix to the Ussuri version of the charm (ussri channel).
Thank you.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-16: Fix proposed to charm-cinder (stable/ussuri)

#36

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-cinder/+/867997

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-16:

#37

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-cinder/+/867998

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-16: Change abandoned on charm-cinder (stable/ussuri)

#38

Change abandoned by "Tiago Pasqualini da Silva <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-cinder/+/867998

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-16: Fix proposed to charm-cinder (master)

#39

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/868001

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-01-31: Fix merged to charm-cinder (master)

#40

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/868001
Committed: https://opendev.org/openstack/charm-cinder/commit/e737668e2fa2b3c59daca5676bd27b0bf4db53fe
Submitter: "Zuul (22348)"
Branch: master

commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

Closes-Bug: #1939389
Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-02: Fix proposed to charm-cinder (stable/zed)

#41

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872571

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-02: Fix proposed to charm-cinder (stable/yoga)

#42

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872572

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-02: Fix proposed to charm-cinder (stable/xena)

#43

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872573

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-02: Fix proposed to charm-cinder (stable/wallaby)

#44

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872574

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-02: Fix proposed to charm-cinder (stable/victoria)

#45

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872575

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-02: Fix proposed to charm-cinder (stable/ussuri)

#46

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-cinder/+/872576

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-17: Fix proposed to charm-cinder (stable/victoria)

#47

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-cinder/+/874334

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-17: Change abandoned on charm-cinder (stable/victoria)

#48

Change abandoned by "Tiago Pasqualini da Silva <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-cinder/+/874334
Reason: wrong cherry-pick

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-05-25: Fix merged to charm-cinder (stable/zed)

#49

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872571
Committed: https://opendev.org/openstack/charm-cinder/commit/00fe7b77e85d54a6e35aaddcbf50257b99cfa2b5
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 00fe7b77e85d54a6e35aaddcbf50257b99cfa2b5
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-05-25: Fix merged to charm-cinder (stable/yoga)

#50

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872572
Committed: https://opendev.org/openstack/charm-cinder/commit/4496922f48c72f4636d93b9268c9dad0c63a7574
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 4496922f48c72f4636d93b9268c9dad0c63a7574
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-05-31: Fix merged to charm-cinder (stable/victoria)

#51

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/874334
Committed: https://opendev.org/openstack/charm-cinder/commit/2bf21135499d2fa22932c24ca06a5e520e0f2529
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 2bf21135499d2fa22932c24ca06a5e520e0f2529
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

This issue has been observed on victoria and newer,
hence the addition of cinder.conf for victoria.

Closes-Bug: #1939389

Co-authored-by: Samuel Walladge <email address hidden>

Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1
(cherry picked from commit 024de3783fc459fc25596ec6084dd065e945df76)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-05-31: Fix merged to charm-cinder (stable/xena)

#52

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872573
Committed: https://opendev.org/openstack/charm-cinder/commit/a71622b7b227b11bbc60c4ee7b391d2a98e52e1a
Submitter: "Zuul (22348)"
Branch: stable/xena

commit a71622b7b227b11bbc60c4ee7b391d2a98e52e1a
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-05-31: Fix merged to charm-cinder (stable/wallaby)

#53

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872574
Committed: https://opendev.org/openstack/charm-cinder/commit/90ff83272b1e3b5f732bdff95289f53bf8028069
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 90ff83272b1e3b5f732bdff95289f53bf8028069
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-06-02: Fix merged to charm-cinder (stable/ussuri)

#54

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/867997
Committed: https://opendev.org/openstack/charm-cinder/commit/8eac344dd68f574787eb74bb56c1e62a040cb787
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 8eac344dd68f574787eb74bb56c1e62a040cb787
Author: Gabriel Adrian Samfira <email address hidden>
Date: Thu Apr 14 18:51:06 2022 +0300

Add nova section

    Cinder sometimes needs to communicate with nova,
    for example to notify VMs when live-resizing a disk.
    Under certain circumstances,
    the autodetected authentication details don't work,
    so we must add a dedicated section in the config
    for the nova client with proper credentials.

This issue has been observed on victoria and newer,
hence the addition of cinder.conf for victoria.

Closes-Bug: #1939389

Co-authored-by: Samuel Walladge <email address hidden>

Change-Id: I6d44223b7c2863c6d9c62b46a41275fd360f92d1
(cherry-picked from commit 2b71f3b1d7bdd8a0902cd6439bc2f66e797293fc)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-06-07: Fix merged to charm-cinder (stable/victoria)

#55

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872575
Committed: https://opendev.org/openstack/charm-cinder/commit/8f1ce44171eaca3bc085b921c8815c6c6a36af4c
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 8f1ce44171eaca3bc085b921c8815c6c6a36af4c
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-06-08: Fix merged to charm-cinder (stable/ussuri)

#56

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/872576
Committed: https://opendev.org/openstack/charm-cinder/commit/755a8c9b7b43e99ae5bda053aff73350d4277a42
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 755a8c9b7b43e99ae5bda053aff73350d4277a42
Author: Tiago Pasqualini <email address hidden>
Date: Fri Dec 16 14:52:20 2022 -0300

Add nova section for Ussuri

    Commit 024de37 added the nova section on cinder.conf, but placed it
    on the Victoria folder. This issue is also observed on Ussuri, so
    this patch moves this from Victoria to Ussuri.

    Closes-Bug: #1939389
    Change-Id: Ia39177d6f47ddda0b1acb27cde02ae5e74a01032
    (cherry picked from commit e737668e2fa2b3c59daca5676bd27b0bf4db53fe)

OpenStack Cinder Charm

Missing [nova] section in cinder.conf "Extend an Attached Volume" doesn't work with Victoria onwards

Bug Description

Other bug subscribers

Remote bug watches

	Status	Importance	Assigned to
OpenStack Cinder Charm	Fix Committed	High	Samuel Allan
Ussuri	Fix Committed	Undecided	Unassigned
Victoria	Fix Committed	Undecided	Unassigned
Wallaby	Fix Committed	Undecided	Unassigned
Xena	Fix Committed	Undecided	Unassigned
Yoga	Fix Committed	Undecided	Unassigned
Zed	Fix Committed	Undecided	Unassigned