Incorrect endpoints for Nova API: ERROR (UnsupportedVersion): Server doesn't support microversions

[Timur Nurlygayanov]

Note: this issue was found on MOS 9.0 version #350 (downstream), this bug is actual for Mitaka release and next releases as well.

Steps To Reproduce:
1. Deploy cluster with 1 controller and 1 compute node.
2. Login via ssh to OpenStack controller node
3. Execute the following commands:
nova version-list
nova --os-compute-api-version 2 list
nova --os-compute-api-version 2.1 list
nova --os-compute-api-version 2.25 list
nova --service-type computev21 --os-compute-api-version 2.1 list
nova --service-type computev3 --os-compute-api-version 2.1 list

Expected Result:
Nova CLI allow to use microversions from console

Observed Result:
We can see different errors for these CLI commands:

///////////////////////////////////////////////

root@node-1:~# nova version-list
Client supported API versions:
Minimum version 2.1
Maximum version 2.25

Server supported API versions:
+------+-----------+----------------------+-------------+---------+
| Id | Status | Updated | Min Version | Version |
+------+-----------+----------------------+-------------+---------+
| v2.0 | SUPPORTED | 2011-01-21T11:33:21Z | | |
| v2.1 | CURRENT | 2013-07-23T11:33:21Z | 2.1 | 2.25 |
+------+-----------+----------------------+-------------+---------+

///////////////////////////////////////////////

root@node-1:~# nova --os-compute-api-version 2.2 list
ERROR (UnsupportedVersion): Server doesn't support microversions

///////////////////////////////////////////////

root@node-1:~# nova --service-type computev3 --os-compute-api-version 2.1 list
ERROR (KeyError): 'version'

///////////////////////////////////////////////

root@node-1:~# openstack service list
+----------------------------------+----------+---------------------+
| ID | Name | Type |
+----------------------------------+----------+---------------------+
| 038622c29251433b91525dd682b64b22 | murano | application-catalog |
| 06255afd8cfd469aa6d98d1d6b21e870 | heat | orchestration |
| 1e5ce79bba8142e59cb258cc6b6e51ec | glare | artifact |
| 319df230648048d28ec729917102e779 | swift | object-store |
| 5311c3e145254a37b5c9ac8addd70ac3 | keystone | identity |
| 6289e77f1a084749999ec10ad1b98cc5 | nova | compute |
| 79159afedaaa4bf39b43205ae0973c0e | neutron | network |
| 8e09b13334db4029867924a3a459ffc4 | novav3 | computev3 |
| b5e6c0bceee34a978ed89070f8390cd2 | glance | image |
| ba6c3fcd29214f21a14c51f0715e1501 | cinder | volume |
| c1b7f9d5162f43a692c662ad0c1d2aa8 | heat-cfn | cloudformation |
| c2eed8e074184c5fa3ceade336fa57a9 | cinderv2 | volumev2 |
| c8491c150617445eb7fc568b2b9272d1 | cinderv3 | volumev3 |
+----------------------------------+----------+---------------------+

[Timur Nurlygayanov (tnurlygayanov)]

root@node-1:~# nova --debug --os-compute-api-version 2.1 list
DEBUG (extension:157) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:157) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:157) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (session:248) REQ: curl -g -i -X GET http://192.168.0.2:5000/ -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.3.0 python-requests/2.9.1 CPython/2.7.6"
INFO (connectionpool:207) Starting new HTTP connection (1): 192.168.0.2
DEBUG (connectionpool:387) "GET / HTTP/1.1" 300 591
DEBUG (session:277) RESP: [300] Content-Length: 591 Vary: X-Auth-Token Server: Apache Connection: close Date: Fri, 13 May 2016 10:41:25 GMT Content-Type: application/json
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.6", "links": [{"href": "http://192.168.0.2:5000/v3/", "rel": "self"}]}, {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.0.2:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

DEBUG (v2:63) Making authentication request to http://192.168.0.2:5000/v2.0/tokens
INFO (connectionpool:242) Resetting dropped connection: 192.168.0.2
DEBUG (connectionpool:387) "POST /v2.0/tokens HTTP/1.1" 200 4863
DEBUG (session:248) REQ: curl -g -i -X GET http://192.168.0.2:8774/v2/d9e2d67d894b4943aa55f6edc038c5d4 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}55149916cd912dd7d6c02f0407d418bef90a142c"
INFO (connectionpool:207) Starting new HTTP connection (1): 192.168.0.2
DEBUG (connectionpool:387) "GET /v2/d9e2d67d894b4943aa55f6edc038c5d4 HTTP/1.1" 404 52
DEBUG (session:277) RESP: [404] Date: Fri, 13 May 2016 10:41:25 GMT Connection: close Content-Type: text/plain; charset=UTF-8 Content-Length: 52 X-Compute-Request-Id: req-5a2d2d67-7bf2-4f94-b1df-5145e3ec22fc
RESP BODY: 404 Not Found

The resource could not be found.

DEBUG (session:248) REQ: curl -g -i -X GET h...

[Timur Nurlygayanov (tnurlygayanov)]

root@node-1:~# openstack service list
+----------------------------------+----------+---------------------+
| ID | Name | Type |
+----------------------------------+----------+---------------------+
| 038622c29251433b91525dd682b64b22 | murano | application-catalog |
| 06255afd8cfd469aa6d98d1d6b21e870 | heat | orchestration |
| 1e5ce79bba8142e59cb258cc6b6e51ec | glare | artifact |
| 319df230648048d28ec729917102e779 | swift | object-store |
| 5311c3e145254a37b5c9ac8addd70ac3 | keystone | identity |
| 6289e77f1a084749999ec10ad1b98cc5 | nova | compute |
| 79159afedaaa4bf39b43205ae0973c0e | neutron | network |
| 8e09b13334db4029867924a3a459ffc4 | novav3 | computev3 |
| b5e6c0bceee34a978ed89070f8390cd2 | glance | image |
| ba6c3fcd29214f21a14c51f0715e1501 | cinder | volume |
| c1b7f9d5162f43a692c662ad0c1d2aa8 | heat-cfn | cloudformation |
| c2eed8e074184c5fa3ceade336fa57a9 | cinderv2 | volumev2 |
| c8491c150617445eb7fc568b2b9272d1 | cinderv3 | volumev3 |
+----------------------------------+----------+---------------------+

[Andriy Kurilin (andreykurilin)]

Based on error message, I can assume that novaclient knows what is microversions stuff.
It looks like the problem is in endpoint configuration:
First of all, computev3 endpoint should be removed. Nova V3 was not finished and it was fully removed in Liberty. We should not have endpoint for it
The second one, there is no endpoint for nova v2.1 (which supports microversions). In upstream we had two endpoint compute (nova legacy v2.0) and computev21(nova 2.1) in liberty. In mitaka endpoing computev21 replaced compute and nova v2.0 was turned off. Since computev3 was not removed, I think that compute endpoint was not updated too.

[Andriy Kurilin (andreykurilin)]

Small update:

> In mitaka endpoing computev21 replaced compute and nova v2.0 was turned off.

nova v2.0 (legacy api) was not turned off. It was moved to another endpoint - `compute_legacy`. I think we should keep it too, so I suggest to modify nova's endpoints as:

- service-type: compute ; service_name: nova; endpoint: <novas_url>/v2.1
- service-type: compute_legacy ; service_name: nova_legacy; endpoint: <novas_url>/v2

[Ivan Berezovskiy (iberezovskiy)]

Related puppet-nova patch is https://review.openstack.org/#/c/316241/, it probably will break current deployment for 10.0 and noop tests. For 9.0 we should follow another solution

[Ivan Berezovskiy (iberezovskiy)]

Setting priority to hight, because upstream patch won't break Fuel anymore, because related patch https://review.openstack.org/#/c/316999/ was merged

[Timur Nurlygayanov (tnurlygayanov)]

So can we prepare the fix for 9.0 before HCF?

[Ivan Berezovskiy (iberezovskiy)]

Timur, ETA for the fix - May 23.

[Denis Egorenko (degorenko)]

Fix is on review: https://review.openstack.org/#/c/319938/

[Denis Egorenko (degorenko)]

Fix for mitaka branch will be present with cherry-picks from both commits (319938 and 316999)

[Denis Egorenko (degorenko)]

Mitaka fix: https://review.openstack.org/321097

[Anna Babich (ababich)]

Verified on
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 416
cat /etc/fuel_build_number:
 416
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6347.noarch
 fuel-bootstrap-cli-9.0.0-1.mos284.noarch
 fuel-migrate-9.0.0-1.mos8398.noarch
 rubygem-astute-9.0.0-1.mos746.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8709.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos137.noarch
 fuel-openstack-metadata-9.0.0-1.mos8709.noarch
 fuel-notify-9.0.0-1.mos8398.noarch
 nailgun-mcagents-9.0.0-1.mos746.noarch
 python-fuelclient-9.0.0-1.mos316.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6347.noarch
 fuel-utils-9.0.0-1.mos8398.noarch
 fuel-setup-9.0.0-1.mos6347.noarch
 fuel-library9.0-9.0.0-1.mos8398.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos284.noarch
 fuel-ui-9.0.0-1.mos2706.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 fuel-misc-9.0.0-1.mos8398.noarch
 python-packetary-9.0.0-1.mos137.noarch
 fuel-nailgun-9.0.0-1.mos8709.noarch
[root@nailgun ~]#

in accordance with steps to reproduce

[Anna Babich (ababich)]

Verified on devstack with Newton cloned from master at 28.06.2016. The results are:

stack@ubuntu:/nova version-list
Client supported API versions:
Minimum version 2.1
Maximum version 2.32

Server supported API versions:
+------+-----------+----------------------+-------------+---------+
| Id | Status | Updated | Min Version | Version |
+------+-----------+----------------------+-------------+---------+
| v2.0 | SUPPORTED | 2011-01-21T11:33:21Z | | |
| v2.1 | CURRENT | 2013-07-23T11:33:21Z | 2.1 | 2.36 |
+------+-----------+----------------------+-------------+---------+
stack@ubuntu:/opt/devstack$ nova --os-compute-api-version 2.1 list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------+
| a6f28bd8-cddf-42f0-8179-a20718fe70d9 | vm1 | ACTIVE | - | Running | private=10.0.0.2 |
+--------------------------------------+------+--------+------------+-------------+------------------+
stack@ubuntu:/opt/devstack$ nova --os-compute-api-version 2.32 list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------+
| a6f28bd8-cddf-42f0-8179-a20718fe70d9 | vm1 | ACTIVE | - | Running | private=10.0.0.2 |
+--------------------------------------+------+--------+------------+-------------+------------------+
stack@ubuntu:/opt/devstack$ openstack service list
+----------------------------------+-------------+----------------+
| ID | Name | Type |
+----------------------------------+-------------+----------------+
| 0bb61f1d04ed45c798b8342e225d2044 | cinder | volume |
| 6b578e9d07be497cbe8c8ccae6312f7e | nova | compute |
| 8389d1d0974645dab2dba08a07fa85ff | glance | image |
| 8e9b5a8ec7e0426a98fc92ed38fae991 | cinderv2 | volumev2 |
| 9e68d428e06d4e0eb1df6e0c96dbcf68 | heat | orchestration |
| af5d294953dc409e9a5128fd1eb5de3b | nova_legacy | compute_legacy |
| ca039057c2574af7930415926864d6b8 | heat-cfn | cloudformation |
| f214380a73634eee8d851556b5d7f600 | cinderv3 | volumev3 |
| ff7968ee679f4adc92419a135dd0539f | keystone | identity |
+----------------------------------+-------------+----------------+