OpenStack Compute (Nova)

nova fails when requesting volumes using '--insecure' flag

Reported by Rafael Durán Castañeda on 2012-03-13
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Unassigned

Bug Description

Hi,

I'm running a testing Openstack where services are under HTTPS load balancer using invalid certs, so when I do any request using the nova client I always use the '--insecure' flag. This works fine except for volume requests, showing a traceback:

nova --debug --insecure volume-list
connect: (192.168.99.101, 443)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 192.168.99.101\r\nContent-Length: 108\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "<email address hidden>", "password": "pass"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Content-Length: 2061
header: Date: Tue, 13 Mar 2012 13:53:14 GMT
REQ: curl -i https://192.168.99.101/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient"

REQ BODY: {"auth": {"tenantName": "admin", "passwordCredentials": {"username": "<email address hidden>", "password": "pass"}}}

RESP:{'date': 'Tue, 13 Mar 2012 13:53:14 GMT', 'content-type': 'application/json', 'content-length': '2061', 'status': '200', 'vary': 'X-Auth-Token'} {"access": {"token": {"expires": "2012-03-14T13:53:14Z", "id": "7fbfc145ba5740179de4b0e04c810369", "tenant": {"description": null, "enabled": true, "id": "6c34665880fb4dab93e20b2b1dab1ce7", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.100.41:8776/v1/6c34665880fb4dab93e20b2b1dab1ce7", "region": "RegionOne", "publicURL": "https://192.168.99.101:446/v1/6c34665880fb4dab93e20b2b1dab1ce7", "internalURL": "http://192.168.100.41:8776/v1/6c34665880fb4dab93e20b2b1dab1ce7"}], "endpoints_links": [], "type": "volume", "name": "'Volume Service'"}, {"endpoints": [{"adminURL": "http://192.168.100.22:9292/v1", "region": "RegionOne", "publicURL": "https://192.168.99.101:445/v1", "internalURL": "http://192.168.100.22:9292/v1"}], "endpoints_links": [], "type": "image", "name": "'Image Service'"}, {"endpoints": [{"adminURL": "http://192.168.100.26:8774/v2/6c34665880fb4dab93e20b2b1dab1ce7", "region": "RegionOne", "publicURL": "https://192.168.99.101:444/v2/6c34665880fb4dab93e20b2b1dab1ce7", "internalURL": "http://192.168.100.26:8774/v2/6c34665880fb4dab93e20b2b1dab1ce7"}], "endpoints_links": [], "type": "compute", "name": "'Compute Service'"}, {"endpoints": [{"adminURL": "http://192.168.100.26:8773/services/Admin", "region": "RegionOne", "publicURL": "http://192.168.99.100:443/services/Cloud", "internalURL": "http://192.168.100.26:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "'EC2 Service'"}, {"endpoints": [{"adminURL": "http://192.168.100.25:35357/v2.0", "region": "RegionOne", "publicURL": "http://192.168.99.101:5000/v2.0", "internalURL": "http://192.168.100.25:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "'Identity Service'"}], "user": {"username": "<email address hidden>", "roles_links": [], "id": "b3d964e627a24c09907df0a27846be2d", "roles": [{"id": "876417f7b12c46ccb98058bdf824a50b", "name": "KeystoneServiceAdmin"}, {"id": "66febe6defe040c18b45fdb3ed9aa253", "name": "KeystoneAdmin"}, {"id": "49858d70ac394647920f472a947afcb0", "name": "Admin"}], "name": "<email address hidden>"}}}

connect: (192.168.99.101, 446)
send: u'GET /v1/6c34665880fb4dab93e20b2b1dab1ce7/volumes/detail HTTP/1.1\r\nHost: 192.168.99.101:446\r\nx-auth-project-id: admin\r\nx-auth-token: 7fbfc145ba5740179de4b0e04c810369\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.0 503 Service Unavailable\r\n'
header: Cache-Control: no-cache
header: Connection: close
header: Content-Type: text/html
REQ: curl -i https://192.168.99.101:446/v1/6c34665880fb4dab93e20b2b1dab1ce7/volumes/detail -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: 7fbfc145ba5740179de4b0e04c810369"

REQ: curl -i https://192.168.99.101:446/v1/6c34665880fb4dab93e20b2b1dab1ce7/volumes/detail -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: 7fbfc145ba5740179de4b0e04c810369"

RESP:{'status': '503', 'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'} <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

RESP:{'status': '503', 'connection': 'close', 'content-type': 'text/html', 'cache-control': 'no-cache'} <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

DEBUG (shell:394) string indices must be integers, not str
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 391, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 342, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 746, in do_volume_list
    volumes = cs.volumes.list()
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/volumes.py", line 76, in list
    return self._list("/volumes/detail", "volumes")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 79, in _list
    data = body[response_key]
TypeError: string indices must be integers, not str
ERROR: string indices must be integers, not str

Using the internal URL as public URL works fine:

 nova --debug --insecure volume-list
connect: (192.168.99.101, 443)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 192.168.99.101\r\nContent-Length: 108\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "<email address hidden>", "password": "pass"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Content-Length: 2061
header: Date: Tue, 13 Mar 2012 14:30:52 GMT
REQ: curl -i https://192.168.99.101/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient"

REQ BODY: {"auth": {"tenantName": "admin", "passwordCredentials": {"username": "<email address hidden>", "password": "pass"}}}

RESP:{'date': 'Tue, 13 Mar 2012 14:30:52 GMT', 'content-type': 'application/json', 'content-length': '2061', 'status': '200', 'vary': 'X-Auth-Token'} {"access": {"token": {"expires": "2012-03-14T14:30:52Z", "id": "bb23165f21034190849b833ce4b74219", "tenant": {"description": null, "enabled": true, "id": "6c34665880fb4dab93e20b2b1dab1ce7", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.100.26:8776/v1/6c34665880fb4dab93e20b2b1dab1ce7", "region": "RegionOne", "publicURL": "http://192.168.100.26:8776/v1/6c34665880fb4dab93e20b2b1dab1ce7", "internalURL": "http://192.168.100.26:8776/v1/6c34665880fb4dab93e20b2b1dab1ce7"}], "endpoints_links": [], "type": "volume", "name": "'Volume Service'"}, {"endpoints": [{"adminURL": "http://192.168.100.22:9292/v1", "region": "RegionOne", "publicURL": "https://192.168.99.101:445/v1", "internalURL": "http://192.168.100.22:9292/v1"}], "endpoints_links": [], "type": "image", "name": "'Image Service'"}, {"endpoints": [{"adminURL": "http://192.168.100.26:8774/v2/6c34665880fb4dab93e20b2b1dab1ce7", "region": "RegionOne", "publicURL": "https://192.168.99.101:444/v2/6c34665880fb4dab93e20b2b1dab1ce7", "internalURL": "http://192.168.100.26:8774/v2/6c34665880fb4dab93e20b2b1dab1ce7"}], "endpoints_links": [], "type": "compute", "name": "'Compute Service'"}, {"endpoints": [{"adminURL": "http://192.168.100.26:8773/services/Admin", "region": "RegionOne", "publicURL": "http://192.168.99.100:443/services/Cloud", "internalURL": "http://192.168.100.26:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "'EC2 Service'"}, {"endpoints": [{"adminURL": "http://192.168.100.25:35357/v2.0", "region": "RegionOne", "publicURL": "http://192.168.99.101:5000/v2.0", "internalURL": "http://192.168.100.25:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "'Identity Service'"}], "user": {"username": "<email address hidden>", "roles_links": [], "id": "b3d964e627a24c09907df0a27846be2d", "roles": [{"id": "876417f7b12c46ccb98058bdf824a50b", "name": "KeystoneServiceAdmin"}, {"id": "66febe6defe040c18b45fdb3ed9aa253", "name": "KeystoneAdmin"}, {"id": "49858d70ac394647920f472a947afcb0", "name": "Admin"}], "name": "<email address hidden>"}}}

connect: (192.168.100.26, 8776)
send: u'GET /v1/6c34665880fb4dab93e20b2b1dab1ce7/volumes/detail HTTP/1.1\r\nHost: 192.168.100.26:8776\r\nx-auth-project-id: admin\r\nx-auth-token: bb23165f21034190849b833ce4b74219\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Compute-Request-Id: req-266072e5-4344-4a46-82b4-87678aec542f
header: Content-Type: application/json
header: Content-Length: 585
header: Date: Tue, 13 Mar 2012 14:30:47 GMT
REQ: curl -i http://192.168.100.26:8776/v1/6c34665880fb4dab93e20b2b1dab1ce7/volumes/detail -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: bb23165f21034190849b833ce4b74219"

REQ: curl -i http://192.168.100.26:8776/v1/6c34665880fb4dab93e20b2b1dab1ce7/volumes/detail -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: bb23165f21034190849b833ce4b74219"

RESP:{'status': '200', 'content-length': '585', 'content-location': u'http://192.168.100.26:8776/v1/6c34665880fb4dab93e20b2b1dab1ce7/volumes/detail', 'x-compute-request-id': 'req-266072e5-4344-4a46-82b4-87678aec542f', 'date': 'Tue, 13 Mar 2012 14:30:47 GMT', 'content-type': 'application/json'} {"volumes": [{"status": "creating", "displayDescription": null, "availabilityZone": "nova", "displayName": null, "attachments": [], "volumeType": "None", "snapshotId": null, "size": 1, "id": "1", "createdAt": "2012-03-09 10:33:05", "metadata": {}}, {"status": "in-use", "displayDescription": null, "availabilityZone": "nova", "displayName": null, "attachments": [{"device": "/dev/vdb", "serverId": "74be8284-f0f1-4463-98ff-ed046a353e6c", "id": "2", "volumeId": "2"}], "volumeType": "None", "snapshotId": null, "size": 1, "id": "2", "createdAt": "2012-03-09 12:09:28", "metadata": {}}]}

RESP:{'status': '200', 'content-length': '585', 'content-location': u'http://192.168.100.26:8776/v1/6c34665880fb4dab93e20b2b1dab1ce7/volumes/detail', 'x-compute-request-id': 'req-266072e5-4344-4a46-82b4-87678aec542f', 'date': 'Tue, 13 Mar 2012 14:30:47 GMT', 'content-type': 'application/json'} {"volumes": [{"status": "creating", "displayDescription": null, "availabilityZone": "nova", "displayName": null, "attachments": [], "volumeType": "None", "snapshotId": null, "size": 1, "id": "1", "createdAt": "2012-03-09 10:33:05", "metadata": {}}, {"status": "in-use", "displayDescription": null, "availabilityZone": "nova", "displayName": null, "attachments": [{"device": "/dev/vdb", "serverId": "74be8284-f0f1-4463-98ff-ed046a353e6c", "id": "2", "volumeId": "2"}], "volumeType": "None", "snapshotId": null, "size": 1, "id": "2", "createdAt": "2012-03-09 12:09:28", "metadata": {}}]}

+----+----------+--------------+------+--------------------------------------+
| ID | Status | Display Name | Size | Attached to |
+----+----------+--------------+------+--------------------------------------+
| 1 | creating | None | 1 | |
| 2 | in-use | None | 1 | 74be8284-f0f1-4463-98ff-ed046a353e6c |
+----+----------+--------------+------+--------------------------------------+

I think this may be related to https://bugs.launchpad.net/glance/+bug/949838

Bye

Changed in nova:
milestone: none → essex-rc1
importance: Undecided → High
status: New → Triaged
Ron Pedde (ron-pedde) wrote :

is it possible this is a load-balancer issue? I've tried to duplicate this, but cannot. I wrapped all my services in ssl using socat, and updated my service catalog to show only ssl endpoints, and wasn't able to replicate it.

stack@devstack:~/devstack$ nova --insecure --debug volume-list
connect: (127.0.0.1, 15000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 127.0.0.1:15000\r\nContent-Length: 100\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"tenantName": "demo", "passwordCredentials": {"username": "admin", "password": "secret"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Content-Length: 2172
header: Date: Thu, 15 Mar 2012 01:40:50 GMT
connect: (192.168.122.118, 18776)
send: u'GET /v1/36d50c9cf2bb4fd2bdf098b683094aa3/volumes/detail HTTP/1.1\r\nHost: 192.168.122.118:18776\r\nx-auth-project-id: demo\r\nx-auth-token: f0f13233e58a4fb1ad4d886e7a46be77\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Compute-Request-Id: req-a936f2ca-4943-4292-b9e6-6c878a70d0ea
header: Content-Type: application/json
header: Content-Length: 486
header: Date: Thu, 15 Mar 2012 01:40:50 GMT
+----+-----------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+----+-----------+--------------+------+-------------+-------------+
| 1 | error | foo | 10 | None | |
| 2 | available | foo | 1 | None | |
+----+-----------+--------------+------+-------------+-------------+

is the 503 from the load balancer, or the volume service? do you have nova-volume logs you can show from the same request?

I'd like to be able to reproduce this.

Vish Ishaya (vishvananda) wrote :

marking incomplete since we have no repro. Please set back to new if anyone can reproduce it.

Changed in nova:
status: Triaged → Incomplete
milestone: essex-rc1 → none

Hi,

After reviewing once more my deployment I've got a configuration issue, so it wasn't bug.

Bye

Tom Fifield (fifieldt) on 2012-05-25
Changed in nova:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers