publicURL endpoint for volumev3 service in regionOne region not found

[Ray Klassen]

This is a new installation based on the Ubuntu instructions for xena openstack. Each component is in a separate container to simulate multiple machines. The OS is Debian, not Ubuntu. So far everything has been successful to the limits of my understanding of openstack (I'm brand new at it and expect that all errors are mine until this one which instructs me to report a bug) So far, I have all the servers running and using horizon and the openstack client I have been able to create networks, volumes, etc. when I try to start an instance with horizon it gives me the same error as the openstack command line client.

The errors complain that the public url for the cinder volume cannot be found. You can see from the tcp stream below that yes it can be found...

I repeat. I'm only posting this as a bug because the error message prompts me to do so. the probability that something in my custom setup is causing this is reasonably high. but here goes.

Command:
nova boot --flavor test-flavour --image 5eedda20-efb0-4c7e-9f1c-806c818043b1 --block-device source=volume,id=d78e4bdc-22fd-4335-beaa-e5f89c0cc168,dest=volume,shutdown=preserve myinstance

Result:
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneauth1.exceptions.catalog.EndpointNotFound'> (HTTP 500) (Request-ID: req-01007635-fc9c-43da-b9b7-0aa8c4e47c67)

relevant section from nova-api.log:

EndpointNotFound: publicURL endpoint for volumev3 service in regionOne region not found
2021-11-16 22:40:28.166 145 INFO nova.api.openstack.wsgi [req-a647bfb4-a523-431c-a012-7f790e307c29 4a1b790b1ad8416996d1e8f5f206f34c 78d0093b66f547d2be2b78e496f9d795 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2021-11-16 22:40:28.167 145 INFO nova.api.openstack.requestlog [req-a647bfb4-a523-431c-a012-7f790e307c29 4a1b790b1ad8416996d1e8f5f206f34c 78d0093b66f547d2be2b78e496f9d795 - default default] 10.0.3.13 "POST /v2.1/servers" status: 500 len: 223 microversion: 2.90 time: 1.257421

Follow TCP stream from Wireshark:

Content-Type: application/json
X-Subject-Token: gAAAAABhlDbCrq4CC3HXZCGtpe57DsdIa-Ad58-802fXX9o79I4wqhZ5r4FNiyPR2E1nD1Wj0fI0RHHOyYTx34r7unVqQmr2k70HAnmN5kJwvFvpJPl62oJo1Ervcoq5gNHNE5TFqBDpNBjf5XSF2SXhkHCncQ6YzRHb3-_RYE_tkTYrO5mCQDs
Vary: X-Auth-Token
x-openstack-request-id: req-d5f72857-dbc9-4f58-a094-f0181b895a3a

{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "e95852d76728426c9830d92c583a60af", "name": "neutron", "password_expires_at": null}, "audit_ids": ["E0h0DC2DStuzCb-xci2g_g"], "expires_at": "2021-11-16T23:54:58.000000Z", "issued_at": "2021-11-16T22:54:58.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "c6f8f9e8a537475d936ab5004f00bd98", "name": "service"}, "is_domain": false, "roles": [{"id": "a1ea9da4765a452d8137490c89bb787a", "name": "member"}, {"id": "b6a93016eab04fd083eeae2b80e4d506", "name": "admin"}, {"id": "85c135a2d9494464ac633ada42b51a87", "name": "reader"}], "catalog": [{"endpoints": [{"id": "2ce0c72136b849d8a33107d17b09f79e", "interface": "public", "region_id": "RegionOne", "url": "http://cinder:8776/v3/c6f8f9e8a537475d936ab5004f00bd98", "region": "RegionOne"}, {"id": "42ccf499e3244dad8e6c86ee87dc66eb", "interface": "admin", "region_id": "RegionOne", "url": "http://cinder:8776/v3/c6f8f9e8a537475d936ab5004f00bd98", "region": "RegionOne"}, {"id": "fc579a720fa9452a9a53501c5dfbea55", "interface": "internal", "region_id": "RegionOne", "url": "http://cinder:8776/v3/c6f8f9e8a537475d936ab5004f00bd98", "region": "RegionOne"}], "id": "07086f0ab79a4df598ad5c6355031af2", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "2e916997f0dd42599f4142d544157b7e", "interface": "public", "region_id": "RegionOne", "url": "http://neutron:9696", "region": "RegionOne"}, {"id": "2f9e329d6cbd4b58a4769c247bdbb1ad", "interface": "internal", "region_id": "RegionOne", "url": "http://neutron:9696", "region": "RegionOne"}, {"id": "a6047bdc5fda477b888e7f8b11c6a235", "interface": "admin", "region_id": "RegionOne", "url": "http://neutron:9696", "region": "RegionOne"}], "id": "2c8378826c124fd9a2bc9f5b38b31e04", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "6c1c39bce0de4dcd8bd250ca7c2d32e5", "interface": "public", "region_id": "RegionOne", "url": "http://placement:8778", "region": "RegionOne"}, {"id": "d55d294866c545c0bac5bad940129f31", "interface": "admin", "region_id": "RegionOne", "url": "http://placement:8778", "region": "RegionOne"}, {"id": "f56b8ba6a893485a9d11307b0a189e83", "interface": "internal", "region_id": "RegionOne", "url": "http://placement:8778", "region": "RegionOne"}], "id": "50774fbc1f384094a452cc7dd8432445", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "1c7c82ae259c4a6ca8df76a5c208088a", "interface": "internal", "region_id": "RegionOne", "url": "http://nova:8774/v2.1", "region": "RegionOne"}, {"id": "898a5b9a52b64d0d87952b06640f81e0", "interface": "public", "region_id": "RegionOne", "url": "http://nova:8774/v2.1", "region": "RegionOne"}, {"id": "fca5d0c3a6da46eb99eb169fb729421c", "interface": "admin", "region_id": "RegionOne", "url": "http://nova:8774/v2.1", "region": "RegionOne"}], "id": "713c34365006450abaa2724a89b19cc6", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "43e80872cbd645c094dc43a0a96ec0d4", "interface": "internal", "region_id": "RegionOne", "url": "http://controller:5000/v3/", "region": "RegionOne"}, {"id": "8bdeb36dee9b482facaab36c8186272b", "interface": "admin", "region_id": "RegionOne", "url": "http://controller:5000/v3/", "region": "RegionOne"}, {"id": "a4996eea2e7b4f9f9b57e29f2ac3ce59", "interface": "public", "region_id": "RegionOne", "url": "http://controller:5000/v3/", "region": "RegionOne"}], "id": "7324d66b42454969a79eab24f2fe3251", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "249cfcadac654720b8a23a6343022653", "interface": "internal", "region_id": "RegionOne", "url": "http://glance:9292", "region": "RegionOne"}, {"id": "82de01540faa489c85317db54f3dfc84", "interface": "admin", "region_id": "RegionOne", "url": "http://glance:9292", "region": "RegionOne"}, {"id": "940f7bc419624cdcbb5c93d9daf98ed5", "interface": "public", "region_id": "RegionOne", "url": "http://glance:9292", "region": "RegionOne"}], "id": "8f4f3dfca25e487ea962719622a83084", "type": "image", "name": "glance"}]}}

[Ray Klassen (raydar10k)]

also: nova, the host executing the instance launch never attempts to connect to cinder or look it up (it's in /etc/hosts file anyway)

[Ray Klassen (raydar10k)]

More exploring has satisfied me that cinder is responding to nova, but nova does not understand the response. The following is from a packet dump between the two

===request for info=====

GET / HTTP/1.1
Host: cinder:8776
User-Agent: uwsgi keystoneauth1/4.4.0 python-requests/2.25.1 CPython/3.9.2
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
X-Auth-Token: gAAAAABhl-uasATQs6wtazUjjfnv5uvv0u4ERNGkJoaVXzo1af6aPje0pGWIR9FK68AXt_Yliyqt_kUOCo1cSIK1LvvMd7L5DrKoFzv0z8Q6TrjyqhZ3_80JHeKC83Sfevawd65UI9AYQEkgh4Hl6rqv7wTvh6A3XHdRQHcUg-fmnmfiAqyyfko
X-Service-Token: gAAAAABhl-hqLH3A8rcGoxoNY-E9Fpe1wQRMj9f8CHXehaZ8KqKik8N4Y2EtXVy35MBcSxMQCze72s5jVooliOwIseLcIM34bRsuZEiUfGOy3yS4E4AtnUJTaRw0HFrEuSpFLSzORTvcXXtgGqX256lXPb7-cwY-iM-pd0dGxA_ZowZhNqS03mY

HTTP/1.1 300 Multiple Choices
Content-Type: application/json
OpenStack-API-Version: volume 3.0
Vary: OpenStack-API-Version
Connection: close

{"versions": [{"id": "v3.0", "status": "CURRENT", "version": "3.66", "min_version": "3.0", "updated": "2021-09-16T00:00:00Z", "links": [{"rel": "describedby", "type": "text/html", "href": "https://docs.openstack.org/"}, {"rel": "self", "href": "http://cinder:8776/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=3"}]}]}

====request for to attach disk=======

POST /v3/78d0093b66f547d2be2b78e496f9d795/attachments HTTP/1.1
Host: cinder:8776
User-Agent: python-cinderclient
Accept-Encoding: gzip, deflate
Accept: application/json
Connection: keep-alive
X-OpenStack-Request-ID: req-9c4abd38-9f76-4317-a014-a7ac74876411
OpenStack-API-Version: volume 3.44
X-Auth-Token: gAAAAABhl-uasATQs6wtazUjjfnv5uvv0u4ERNGkJoaVXzo1af6aPje0pGWIR9FK68AXt_Yliyqt_kUOCo1cSIK1LvvMd7L5DrKoFzv0z8Q6TrjyqhZ3_80JHeKC83Sfevawd65UI9AYQEkgh4Hl6rqv7wTvh6A3XHdRQHcUg-fmnmfiAqyyfko
X-Service-Token: gAAAAABhl-hqLH3A8rcGoxoNY-E9Fpe1wQRMj9f8CHXehaZ8KqKik8N4Y2EtXVy35MBcSxMQCze72s5jVooliOwIseLcIM34bRsuZEiUfGOy3yS4E4AtnUJTaRw0HFrEuSpFLSzORTvcXXtgGqX256lXPb7-cwY-iM-pd0dGxA_ZowZhNqS03mY
Content-Type: application/json
Content-Length: 147

{"attachment": {"volume_uuid": "42caf199-aa77-4987-a1ab-0dd79f5aa952", "connector": null, "instance_uuid": "746e8192-8561-4019-86df-c5179d805dea"}}HTTP/1.1 200 OK
Content-Type: application/json
x-compute-request-id: req-0c90c8a1-9d23-475d-b232-32388c981c8a
OpenStack-API-Version: volume 3.44
Vary: OpenStack-API-Version
x-openstack-request-id: req-0c90c8a1-9d23-475d-b232-32388c981c8a
Connection: close

{"attachment": {"id": "bee7db8c-8b8d-4548-9ce7-1101eef164e1", "status": "reserved", "instance": "746e8192-8561-4019-86df-c5179d805dea", "volume_id": "42caf199-aa77-4987-a1ab-0dd79f5aa952", "attached_at": "", "detached_at": "", "attach_mode": "null", "connection_info": {}}}

====

after all that nova gives the error

publicURL endpoint for volumev3 service in regionOne region not found

Can't be.

[Artom Lifshitz (notartom)]

Could you post the output of `openstack endpoint list`, as well as your nova.conf file? Thanks!

[Ray Klassen (raydar10k)]

Unfortunately that computer has been repurposed. I have no further info for you.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]