server add volume fails

[peter eisch]

Description
===========
<class 'cinderclient.exceptions.Unauthorized'> (HTTP 500)
It seems to start by not finding the instance in nova.

Environment
===========

Environment: (Rocky)
openstack-nova-common-18.3.0-1.el7.noarch
openstack-nova-conductor-18.3.0-1.el7.noarch
openstack-nova-api-18.3.0-1.el7.noarch
openstack-nova-scheduler-18.3.0-1.el7.noarch
openstack-nova-console-18.3.0-1.el7.noarch
openstack-nova-placement-api-18.3.0-1.el7.noarch
python2-novaclient-11.0.1-1.el7.noarch
openstack-nova-novncproxy-18.3.0-1.el7.noarch
python-nova-18.3.0-1.el7.noarch

Hypervisor:
Linux KVM

Storate:
CEPH: 14.2.8 / RBD volumes

Network:
Neutron (OpenVSwitch)

Steps to reproduce
==================

I did: openstack --debug server add volume c331814d-b758-460e-9972-bc1e987b933d 0fc5bec7-9364-458c-a809-f38389890a60

I also did: openstack --debug server add volume allalal peter-test-001

I received:

ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010)

Expected result
===============
Associate the volume and exit cleanly.

Actual result
=============
Nova fails to find existing instance and fails the attempt to add the volume. All other server commands for the instance work as expected (stop, start, migrate, etc.)

Logs & Configs
==============
/etc/nova/nova.conf
[DEFAULT]
cpu_allocation_ratio = 16.0
debug = true
enabled_apis = osapi_compute,metadata
metadata_proxy_shared_secret = <secret>
metadata_workers = 4
my_ip = 10.176.0.42
ram_allocation_ratio = 1.0
scheduler_host_subset_size = 2
transport_url = rabbit://openstack:<passwd>@mq-a01.xxx.cloud,openstack:<passwd>@mq-a02.xxx.cloud,openstack:<passwd>@mq-a03.xxx.cloud
cinder_catalog_info=volumev3:cinderv3:publicURL

[api]
auth_strategy = keystone

[api_database]
connection = mysql+pymysql://atx_nova_api:<passwd>@db-a00.xxx.cloud/atx_nova_api

[filter_scheduler]
enabled_filters = AvailabilityZoneFilter,ComputeCapabilitiesFilter,ComputeFilter,ImagePropertiesFilter,RamFilter,RetryFilter,ServerGroupAffinityFilter,ServerGroupAntiAffinityFilter
host_subset_size = 4
io_ops_weight_multiplier = 10.0
max_instances_per_host = 100
ram_weight_multiplier = 6.0
soft_affinity_weight_multiplier = 10.0
soft_anti_affinity_weight_multiplier = 10.0
weight_classes = nova.scheduler.weights.all_weighers

[database]
connection = mysql+pymysql://atx_nova:<passwd>@db-a00.xxx.cloud/atx_nova

[glance]
api_servers = https://glance-a00.xxx.cloud

[keystone_authtoken]
auth_type = password
www_authenticate_uri = https://keystone-a00.xxx.cloud
auth_url = https://keystone-a00.xxx.cloud
memcached_servers = memcached-a01.xxx.cloud:11211,memcached-a02.xxx.cloud:11211,memcached-a03.xxx.cloud:11211
password = <passwd>
project_domain_name = default
project_name = service
user_domain_name = default
username = atx_nova
service_token_roles_required = true

[neutron]
auth_type = password
auth_url = https://keystone-a00.xxx.cloud
metadata_proxy_shared_secret = <secret>
password = <passwd>
project_domain_name = Default
project_name = service
region_name = atx
service_metadata_proxy = true
#uses keystoneauth1: url = https://neutron-a00.xxx.cloud
user_domain_name = Default
username = atx_neutron

[oslo_concurrency]
lock_path = /var/lib/nova/tmp

[placement]
auth_type = password
auth_url = https://keystone-a00.xxx.cloud
os_region_name = atx
password = <passwd>
project_domain_name = Default
project_name = service
user_domain_name = Default
username = atx_placement

[scheduler]
discover_hosts_in_cells_interval = 300

[oslo_notifications_group]
driver = messaging
topics = notifications

[cache]
backend=oslo_cache.memcache_pool
enable = true

[oslo_messaging_rabbit]
amqp_durable_queues = true
rabbit_ha_queues = true
rabbit_retry_backoff = 2
rabbit_retry_interval = 1

[peisch@jump ~]$ openstack volume list
+--------------------------------------+----------------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+----------------+-----------+------+-------------+
| ... | ... | available | 10 | |
| 0fc5bec7-9364-458c-a809-f38389890a60 | peter-test-001 | available | 10 | |
| ... | ... | available | 10 | |
+--------------------------------------+----------------+-----------+------+-------------+
[peisch@hopslam ~]$ openstack server list
+--------------------------------------+-----------+--------+---------------------+----------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------+--------+---------------------+----------+--------+
| ... | ... | ACTIVE | admin=10.1.2.5 | Centos77 | 2x2x20 |
| c331814d-b758-460e-9972-bc1e987b933d | allalal | ACTIVE | admin=10.1.2.7 | Centos77 | 2x2x20 |
+--------------------------------------+-----------+--------+---------------------+----------+--------+
[peisch@jump ~]$

Openstack Debug Log:
(starting after keystone auth)
Starting new HTTPS connection (1): nova-a00.xxx.cloud
https://nova-a00.xxx.cloud:443 "GET /v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d HTTP/1.1" 200 1646
RESP: [200] Content-Length: 1646 Content-Type: application/json Date: Wed, 13 May 2020 19:39:35 GMT Openstack-Api-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-a98e058c-27df-471a-9553-8f9f52dab060 X-Openstack-Nova-Api-Version: 2.1 X-Openstack-Request-Id: req-a98e058c-27df-471a-9553-8f9f52dab060
RESP BODY: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"admin": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:a3:6a:5c", "version": 4, "addr": "10.176.128.7", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://nova-a00.xxx.cloud/v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d", "rel": "self"}, {"href": "http://nova-a00.xxx.cloud/servers/c331814d-b758-460e-9972-bc1e987b933d", "rel": "bookmark"}], "image": {"id": "41bb501b-907e-4e0f-a9fe-072a5d3fbe58", "links": [{"href": "http://nova-a00.xxx.cloud/images/41bb501b-907e-4e0f-a9fe-072a5d3fbe58", "rel": "bookmark"}]}, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000601", "OS-SRV-USG:launched_at": "2020-04-27T20:44:42.000000", "flavor": {"id": "d15ae531-2911-4c5a-aeb5-d62081def56d", "links": [{"href": "http://nova-a00.xxx.cloud/flavors/d15ae531-2911-4c5a-aeb5-d62081def56d", "rel": "bookmark"}]}, "id": "c331814d-b758-460e-9972-bc1e987b933d", "security_groups": [{"name": "default"}], "user_id": "497a19c5da034431b6c01d5d8f523968", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "ACTIVE", "updated": "2020-04-27T20:45:57Z", "hostId": "d98666c745c89cc71a5fa952c75899cc9bd67808a24a117dc5054f5b", "OS-EXT-SRV-ATTR:host": "compute-a19.xxx.cloud", "OS-SRV-USG:terminated_at": null, "key_name": null, "OS-EXT-SRV-ATTR:hypervisor_hostname": "compute-a19.xxx.cloud", "name": "allalal", "created": "2020-04-24T14:51:33Z", "tenant_id": "1134ba6f01ef43f49961aa6bbe49f5bc", "os-extended-volumes:volumes_attached": [], "metadata": {}}}
GET call to compute for https://nova-a00.xxx.cloud/v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d used request id req-a98e058c-27df-471a-9553-8f9f52dab060
REQ: curl -g -i -X GET https://cinder-a00.xxx.cloud/v3/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60 -H "Accept: application/json" -H "User-Agent: python-cinderclient" -H "X-Auth-Token: {SHA1}99c93352a2ccbc3dd1f8df8997f0469a95f9f367"
Starting new HTTPS connection (1): cinder-a00.xxx.cloud
https://cinder-a00.xxx.cloud:443 "GET /v3/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60 HTTP/1.1" 200 1055
RESP: [200] Content-Length: 1055 Content-Type: application/json Date: Wed, 13 May 2020 19:39:35 GMT Openstack-Api-Version: volume 3.0 Vary: OpenStack-API-Version X-Compute-Request-Id: req-c192e90e-711d-498e-806d-1eeded367306 X-Openstack-Request-Id: req-c192e90e-711d-498e-806d-1eeded367306
RESP BODY: {"volume": {"migration_status": null, "attachments": [], "links": [{"href": "https://cinder-a00.xxx.cloud/v3/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60", "rel": "self"}, {"href": "https://cinder-a00.xxx.cloud/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60", "rel": "bookmark"}], "availability_zone": "nova", "os-vol-host-attr:host": "cinder-a03.xxx.cloud@sizeone#sizeone", "encrypted": false, "updated_at": "2020-04-24T14:52:25.000000", "replication_status": null, "snapshot_id": null, "id": "0fc5bec7-9364-458c-a809-f38389890a60", "size": 10, "user_id": "497a19c5da034431b6c01d5d8f523968", "os-vol-tenant-attr:tenant_id": "1134ba6f01ef43f49961aa6bbe49f5bc", "os-vol-mig-status-attr:migstat": null, "metadata": {}, "status": "available", "description": null, "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": "peter-test-001", "bootable": "false", "created_at": "2020-04-24T14:52:24.000000", "volume_type": "sizeone"}}
GET call to volumev3 for https://cinder-a00.xxx.cloud/v3/1134ba6f01ef43f49961aa6bbe49f5bc/volumes/0fc5bec7-9364-458c-a809-f38389890a60 used request id req-c192e90e-711d-498e-806d-1eeded367306
REQ: curl -g -i -X POST https://nova-a00.xxx.cloud/v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d/os-volume_attachments -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}99c93352a2ccbc3dd1f8df8997f0469a95f9f367" -d '{"volumeAttachment": {"volumeId": "0fc5bec7-9364-458c-a809-f38389890a60"}}'
https://nova-a00.xxx.cloud:443 "POST /v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d/os-volume_attachments HTTP/1.1" 500 210
RESP: [500] Content-Length: 210 Content-Type: application/json; charset=UTF-8 Date: Wed, 13 May 2020 19:39:36 GMT Openstack-Api-Version: compute 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-fb493f88-61c5-4ebd-829b-305b31aaa010 X-Openstack-Nova-Api-Version: 2.1 X-Openstack-Request-Id: req-fb493f88-61c5-4ebd-829b-305b31aaa010
RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'cinderclient.exceptions.Unauthorized'>", "code": 500}}
POST call to compute for https://nova-a00.xxx.cloud/v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d/os-volume_attachments used request id req-fb493f88-61c5-4ebd-829b-305b31aaa010
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 464, in take_action
    parsed_args.device,
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 393, in substitution
    return methods[-1].func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 56, in create_server_volume
    body, "volumeAttachment")
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 361, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010)
clean_up AddServerVolume: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 464, in take_action
    parsed_args.device,
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 393, in substitution
    return methods[-1].func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 56, in create_server_volume
    body, "volumeAttachment")
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 361, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'cinderclient.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-fb493f88-61c5-4ebd-829b-305b31aaa010)

END return value: 1

==> /var/log/nova/nova-api.log <==
2020-05-13 13:59:06.159 14830 DEBUG nova.api.openstack.wsgi [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] Action: 'create', calling method: <bound method VolumeAttachmentController.create of <nova.api.openstack.compute.volumes.VolumeAttachmentController object at 0x7f5113d97050>>, body: {"volumeAttachment": {"volumeId": "0fc5bec7-9364-458c-a809-f38389890a60"}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:615
2020-05-13 13:59:06.163 14830 DEBUG nova.compute.api [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] [instance: c331814d-b758-460e-9972-bc1e987b933d] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2421
2020-05-13 13:59:06.171 14830 DEBUG oslo_concurrency.lockutils [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] Lock "3bc099c6-6bc5-4d39-9259-3c2fca30c75e" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2020-05-13 13:59:06.171 14830 DEBUG oslo_concurrency.lockutils [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] Lock "3bc099c6-6bc5-4d39-9259-3c2fca30c75e" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2020-05-13 13:59:07.223 14830 INFO nova.api.openstack.wsgi [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] HTTP exception thrown: Volume 0fc5bec7-9364-458c-a809-f38389890a60 could not be found.
2020-05-13 13:59:07.223 14830 DEBUG nova.api.openstack.wsgi [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] Returning 404 to user: Volume 0fc5bec7-9364-458c-a809-f38389890a60 could not be found. __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1077
2020-05-13 13:59:07.225 14830 INFO nova.osapi_compute.wsgi.server [req-1299524b-1fb1-46b1-82de-9d3ced5c694f c351f9a7674a4a0cbe63ca633f66f606 1134ba6f01ef43f49961aa6bbe49f5bc - default default] 172.22.0.66,10.176.0.102 "POST /v2.1/servers/c331814d-b758-460e-9972-bc1e987b933d/os-volume_attachments HTTP/1.1" status: 404 len: 506 time: 1.0708430

[Balazs Gibizer (balazs-gibizer)]

It seems you have a proper [neutron] and [placement] section in your nova.conf but you don't have a [cinder] section. You need to configure the similar [cinder] section to allow nova to talk to cinder. I think this is the root case of the error you see:

ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'cinderclient.exceptions.Unauthorized'> (HTTP 500)

I'm setting this bug to Invalid now. Please fix you config and try again. If you still see the same issue then set the bug status back to New and add your new config to the report.

[peter eisch (peterbokunet)]

Thank you for the suggestion. I rolled back through our versions of nova.conf and it seems we've never had a cinder block in it. I did populate it, restarted and the results are the same.

/etc/nova/nova.conf
[DEFAULT]
cpu_allocation_ratio = 16.0
debug = true
enabled_apis = osapi_compute,metadata
metadata_proxy_shared_secret = <secret>
metadata_workers = 4
my_ip = 10.176.0.42
ram_allocation_ratio = 1.0
scheduler_host_subset_size = 2
transport_url = rabbit://openstack:<passwd>@mq-a01.xxx.cloud,openstack:<passwd>@mq-a02.xxx.cloud,openstack:<passwd>@mq-a03.xxx.cloud

[api]
auth_strategy = keystone

[api_database]
connection = mysql+pymysql://atx_nova_api:<passwd>@db-a00.xxx.cloud/atx_nova_api

[filter_scheduler]
enabled_filters = AvailabilityZoneFilter,ComputeCapabilitiesFilter,ComputeFilter,ImagePropertiesFilter,RamFilter,RetryFilter,ServerGroupAffinityFilter,ServerGroupAntiAffinityFilter
host_subset_size = 4
io_ops_weight_multiplier = 10.0
max_instances_per_host = 100
ram_weight_multiplier = 6.0
soft_affinity_weight_multiplier = 10.0
soft_anti_affinity_weight_multiplier = 10.0
weight_classes = nova.scheduler.weights.all_weighers

[database]
connection = mysql+pymysql://atx_nova:<passwd>@db-a00.xxx.cloud/atx_nova

[cinder]
catalog_info=volumev3:cinderv3:publicURL
os_region_name=atx
http_retries=3
auth_type = password
auth_url = https://cinder-s00.xxx.cloud
password = <password>
project_domain_name = Default
project_name = service
user_domain_name = Default
username = atx_cinder

[glance]
api_servers = https://glance-a00.xxx.cloud

[keystone_authtoken]
auth_type = password
www_authenticate_uri = https://keystone-a00.xxx.cloud
auth_url = https://keystone-a00.xxx.cloud
memcached_servers = memcached-a01.xxx.cloud:11211,memcached-a02.xxx.cloud:11211,memcached-a03.xxx.cloud:11211
password = <passwd>
project_domain_name = default
project_name = service
user_domain_name = default
username = atx_nova
service_token_roles_required = true

[neutron]
auth_type = password
auth_url = https://keystone-a00.xxx.cloud
metadata_proxy_shared_secret = <secret>
password = <passwd>
project_domain_name = Default
project_name = service
region_name = atx
service_metadata_proxy = true
#uses keystoneauth1: url = https://neutron-a00.xxx.cloud
user_domain_name = Default
username = atx_neutron

[oslo_concurrency]
lock_path = /var/lib/nova/tmp

[placement]
auth_type = password
auth_url = https://keystone-a00.xxx.cloud
os_region_name = atx
password = <passwd>
project_domain_name = Default
project_name = service
user_domain_name = Default
username = atx_placement

[scheduler]
discover_hosts_in_cells_interval = 300

[oslo_notifications_group]
driver = messaging
topics = notifications

[cache]
backend=oslo_cache.memcache_pool
enable = true

[oslo_messaging_rabbit]
amqp_durable_queues = true
rabbit_ha_queues = true
rabbit_retry_backoff = 2
rabbit_retry_interval = 1

/etc/cinder/cinder.conf (from the cinder node, of course)
[DEFAULT]
api_paste_config = /etc/cinder/api-paste.ini
auth_strategy = keystone
backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
backup_ceph_conf=/etc/ceph/ceph.conf
backup_ceph_user = cinder
backup_ceph_chunk_size = 134217728
backup_ceph_pool = a...

[Balazs Gibizer (balazs-gibizer)]

@Peter: Thanks for the response. Could you please confirm that the configured username and password for cinder are correct by using that user and pass in you CLI config file / env and listing the volumes with the CLI?

[peter eisch (peterbokunet)]

Something is weird and/or I need to learn more:

[peisch@commander cinder-look]$ cat atx-admin_cinder.json
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "atx_cinder",
          "domain": { "id": "default" },
          "password": "329na2Tb$80g"
        }
      }
    }
  }
}
[peisch@commander cinder-look]$ echo $TOKEN

[peisch@commander cinder-look]$ curl -vs -X GET http://10.180.0.142:8776/v3/ -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $TOKEN"; echo
* About to connect() to 10.180.0.142 port 8776 (#0)
* Trying 10.180.0.142...
* Connected to 10.180.0.142 (10.180.0.142) port 8776 (#0)
> GET /v3/ HTTP/1.1
> Host: 10.180.0.142:8776
> User-Agent: python-cinderclient
> Accept: application/json
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Content-Length: 114
< Www-Authenticate: Keystone uri="https://keystone-s00.xxx.cloud/v3"
< X-Openstack-Request-Id: req-6f9ecdac-561c-4c2a-93be-401b42158e5e
< Date: Mon, 18 May 2020 15:41:53 GMT
<
* Connection #0 to host 10.180.0.142 left intact
{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
I was expecting this outcome as it hadn't any auth.

[peisch@commander cinder-look]$ export TOKEN=`curl -si -d @atx-admin_cinder.json -H "Content-type: application/json" https://keystone-a00.xxx.cloud/v3/auth/tokens|awk '/X-Subject-Token/ {print $2}'`
[peisch@commander cinder-look]$ curl -vs -X GET http://10.180.0.142:8776/v3/ -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $TOKEN"; echo
* About to connect() to 10.180.0.142 port 8776 (#0)
* Trying 10.180.0.142...
* Connected to 10.180.0.142 (10.180.0.142) port 8776 (#0)
> GET /v3/ HTTP/1.1
> Host: 10.180.0.142:8776
> User-Agent: python-cinderclient
> Accept: application/json
> X-Auth-Token: gAAAAABewqziIAkl9ZALaVALfwDlUD_ZoisMNpQv-AbvErzM95yd9n91_hEPhqMR-rF9ntjTyjWsHaBJjKlKqJ7KaL00oEBNOQCC6tcjfxFabBAy1dcg3yTyMXpheg-McDd4rfQstEj3RZp7jWD1S6acQLiqLO8f4OWpWe7p5PM5MWzpVG824a0
>
< HTTP/1.1 200 OK
< X-Compute-Request-Id: req-c126150d-f6c7-4397-b3cc-ac561a6e9f16
< Content-Type: application/json
< Content-Length: 389
< Openstack-Api-Version: volume 3.0
< Vary: OpenStack-API-Version
< X-Openstack-Request-Id: req-c126150d-f6c7-4397-b3cc-ac561a6e9f16
< Date: Mon, 18 May 2020 15:42:33 GMT
<
* Connection #0 to host 10.180.0.142 left intact
{"versions": [{"status": "CURRENT", "updated": "2018-07-17T00:00:00Z", "links": [{"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://cinder-s00.xxx.cloud/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.55", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=3"}], "id": "v3.0"}]}

With auth, I get results, but now when I query volumes:

[peisch@commander cinder-look]$ curl -vs -X GET http://10.176.0.142:8776/v3/1134ba6f01ef43f49961aa6bbe49f5bc/volumes -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: $TOKEN"; echo
* About to connect() to 10.176.0.142 port 8776 (#0)
* Trying...

[peter eisch (peterbokunet)]

nova-api.log
  60995 2020-10-09 09:24:53.917 1346 ERROR nova.volume.cinder [req-af525fe2-040a-4a10-aeb9-87530d4bb1e1 5ae0b9cf906e4312aa9f2a0f57471d49 d2ff913f5b6542cda307c9cd6a95a214 - default default] [instance: 2668538c-2d79-48d8-b58f-22a3c76dc62f] Create attachment failed for vo lume 41426c57-e520-4626-8da8-963fdb1e2490. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401)
  60996 2020-10-09 09:24:53.933 1346 ERROR nova.api.openstack.wsgi [req-af525fe2-040a-4a10-aeb9-87530d4bb1e1 5ae0b9cf906e4312aa9f2a0f57471d49 d2ff913f5b6542cda307c9cd6a95a214 - default default] Unexpected exception in API method: Unauthorized: The request you have made requires authentication. (HTTP 401)

cinder-manager.log
2020-04-22 14:08:42.542 2256794 CRITICAL cinder [req-86c017a1-c30c-4428-9ef2-9b352d42e763 - - - - -] Unhandled error: VolumeNotFound: Volume <email address hidden> could not be found.
2020-04-22 14:08:42.542 2256794 ERROR cinder Traceback (most recent call last):
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/bin/cinder-manage", line 10, in <module>
2020-04-22 14:08:42.542 2256794 ERROR cinder sys.exit(main())
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/cmd/manage.py", line 868, in main
2020-04-22 14:08:42.542 2256794 ERROR cinder fn(**fn_kwargs)
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/cmd/manage.py", line 468, in delete
2020-04-22 14:08:42.542 2256794 ERROR cinder volume = objects.Volume.get_by_id(ctxt, volume_id)
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/objects/base.py", line 345, in get_by_id
2020-04-22 14:08:42.542 2256794 ERROR cinder orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 1789, in get_by_id
2020-04-22 14:08:42.542 2256794 ERROR cinder return IMPL.get_by_id(context, model, id, *args, **kwargs)
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 183, in wrapper
2020-04-22 14:08:42.542 2256794 ERROR cinder return f(*args, **kwargs)
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 7089, in get_by_id
2020-04-22 14:08:42.542 2256794 ERROR cinder return _GET_METHODS[model](context, id, *args, **kwargs)
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 183, in wrapper
2020-04-22 14:08:42.542 2256794 ERROR cinder return f(*args, **kwargs)
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2186, in volume_get
2020-04-22 14:08:42.542 2256794 ERROR cinder return _volume_get(context, volume_id)
2020-04-22 14:08:42.542 2256794 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 183, i...

[peter eisch (peterbokunet)]

Same action of attaching a volume in Horizon errors with:

Select a valid choice. 41426c57-e520-4626-8da8-963fdb1e2490 is not one of the available choices.

and this is after I had selected it from the drop-down list in the same pane.

[peter eisch (peterbokunet)]

From the CEPH perspective, the volume in in stp.sizeone and the cinder user has full access.

exported keyring for client.cinder
[client.cinder]
 key = <redacted>
 caps mon = "allow r"
 caps osd = "profile rbd pool=stp.cinder-volumes, profile rbd pool=stp.images, profile rbd pool=stp.glance, profile rbd pool=stp.cinder-backup, profile rbd pool=stp.sizeone"