nova and python-novaclient disagree on volumes API URLs

[Paul Collins]

I noticed the following (Ubuntu 12.04 LTS on the Nova cluster, Ubuntu 12.04 LTS on my machine):

$ nova volume-list
ERROR: n/a (HTTP 404)

Based on the output of "nova --debug volume-list", it looks like python-novaclient is expecting to be able to do "GET /v1.1/pjdc_project/volumes/detail". When I manually constuct a request as follows, using "GET /v1.1/pjdc_project/os-volumes/detail", a sensible-looking chunk of JSON is returned:

$ nc XXX.XXX.XXX.XXX YYYY
GET /v1.1/pjdc_project/os-volumes/detail HTTP/1.1
Host: XXX.XXX.XXX.XXX:YYYY
x-auth-project-id: pjdc_project
x-auth-token: ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
accept-encoding: gzip, deflate
accept: application/json
user-agent: python-novaclient

HTTP/1.1 200 OK
X-Compute-Request-Id: req-WWWWWWWW-WWWW-WWWW-WWWW-WWWWWWWWWWWW
Content-Type: application/json
Content-Length: 345
Date: Mon, 07 May 2012 22:32:58 GMT

{"volumes": [{"status": "in-use", "displayDescription": null, "availabilityZone": "nova", "displayName": null, "attachments": [{"device": "/dev/vdc", "serverId": "VVVVVVVV-VVVV-VVVV-VVVV-VVVVVVVVVVVV", "id": 51, "volumeId": 51}], "volumeType": null, "snapshotId": null, "size": 25, "id": 51, "createdAt": "2012-05-04 03:25:04", "metadata": {}}]}

It would be helpful if this disconnect could be resolved and a fix targeted to Precise.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Vish Ishaya (vishvananda)]

it looks like precise has an older version of novaclient? this was changed around e-3 timeframe when we gave the volume api its own endpoint.

[Chuck Short (zulcss)]

We are shipping the final version of 2012.1 for precise. in the archive.

[William Kelly (the-william-kelly)]

I am not able to reproduce this on current precise 12.04 novaclient packages running against an installation of the 12.04 packaged version of essex.

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=========================-=========================-==================================================================
ii python-novaclient 2012.1-0ubuntu1 client library for OpenStack Compute API

will@swift01:~$ nova --debug volume-list
connect: (172.28.1.148, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 172.28.1.148:5000\r\nContent-Length: 102\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "XXXXXXXXX"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Date: Thu, 31 May 2012 19:10:55 GMT
header: Transfer-Encoding: chunked
connect: (172.28.1.148, 8776)
send: u'GET /v1/5c9e830827e0412b92da25b128f5c63d/volumes/detail HTTP/1.1\r\nHost: 172.28.1.148:8776\r\nx-auth-project-id: admin\r\nx-auth-token: 6a42a4101dd04591979fe5d0df51fc8b\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Compute-Request-Id: req-fd6dd42f-284f-4b34-b69f-28be6e6fe0ae
header: Content-Type: application/json
header: Content-Length: 15
header: Date: Thu, 31 May 2012 19:10:55 GMT
+----+--------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+----+--------+--------------+------+-------------+-------------+
+----+--------+--------------+------+-------------+-------------+

[Paul Collins (pjdc)]

Looking at William's trace, I see some differences with the traces I get. Not posting a full one in the first place was foolish of me. Here it is now.

$ nova --debug volume-list
connect: (XXX.XXX.XXX.XXX, 8774)
send: 'GET /v1.1 HTTP/1.1\r\nHost: XXX.XXX.XXX.XXX:8774\r\nx-auth-project-id: pjdc_project\r\naccept-encoding: gzip, deflate\r\nx-auth-user: pjdc\r\nuser-agent: python-novaclient\r\nx-auth-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\r\naccept: application/json\r\n\r\n'
reply: 'HTTP/1.1 204 No Content\r\n'
header: Content-Length: 0
header: X-Auth-Token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
header: X-Server-Management-Url: http://XXX.XXX.XXX.XXX:8774/v1.1/pjdc_project
header: Content-Type: text/plain; charset=UTF-8
header: Date: Thu, 31 May 2012 21:19:01 GMT
send: 'GET /v1.1/pjdc_project/volumes/detail HTTP/1.1\r\nHost: XXX.XXX.XXX.XXX:8774\r\nx-auth-project-id: pjdc_project\r\nx-auth-token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Content-Length: 52
header: Content-Type: text/plain; charset=UTF-8
header: Date: Thu, 31 May 2012 21:19:02 GMT
DEBUG (shell:416) n/a (HTTP 404)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 413, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 364, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 858, in do_volume_list
    volumes = cs.volumes.list()
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/volumes.py", line 79, in list
    return self._list("/volumes/detail", "volumes")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 71, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 136, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 124, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 107, in request
    raise exceptions.from_response(resp, body)
NotFound: n/a (HTTP 404)
ERROR: n/a (HTTP 404)

Whereas in William's trace the token is obtained with "POST /v2.0/tokens" and the list operation is performed with "GET /v1/5c9e830827e0412b92da25b128f5c63d/volumes/detail".

In the credentials packets we distribute to our Openstack users, we have a file containing environment variables, which includes:

export NOVA_URL="http://XXX.XXX.XXX.XXX:8774/v1.1/"
export NOVA_VERSION="1.1"

However, when I set things up as follows:

export NOVA_URL="http://XXX.XXX.XXX.XXX:8774/v2.0/"
export NOVA_VERSION="2"

I get:

$ nova --debug volume-list
connect: (XXX.XXX.XXX.XXX, 8774)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: XXX.XXX.XXX.XXX:8774\r\nContent-Length: 137\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "pjdc_project", "passwordCredentials": {"username": "pjdc...

[Vish Ishaya (vishvananda)]

it should be getting the url from keystone anyway, so I don't imagine that the nova url is a problem. I think it is far more likely that you have an older version of novaclient that is overiding the one from precise. Perhaps you have pip installed novaclient or done a setup.py install?

[Paul Collins (pjdc)]

This Openstack installation is using deprecated auth, though, not keystone. The following flags are in nova.conf:

--use_deprecated_auth
--auth_strategy=deprecated

I've only used Ubuntu packages on this machine — no devstack, no pip, no setup.py.

[Vish Ishaya (vishvananda)]

Ah, deprecated auth is most likely the problem. I suspect deprecated auth doesn't support multiple endpoints. It really is only still there to allow you to transition to keystone. Is there some reason you haven't switched?

[Thierry Carrez (ttx)]

We cannot solve the issue you reported without more information. Could you please provide the requested information ?

[Paul Collins (pjdc)]

Apologies for the delay in replying. We recently completed our migration to Keystone, and now "nova volume-list" works as expected.