HTTP POST used instead of GET leads to HTTP error 400

Bug #1685342 reported by Asquirrel
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned

Bug Description

I have cirros 0.3.2 uploaded to my Openstack Newton Installation (installed from the repos; not using MAAS or any other automated installer) under the name 'demo_install'. When I try to create a volume (in any way) that uses this image for the purposes of creating a boot-from-disk instance, I get an error stating that either Block Device Mapping failed and/or HTTP error 400 occured when requesting information about the volumes currently present in Openstack.

When I run:

openstack volume create --type lvm --image demo_install --size 1 test -vv

on the command line, I see the following output just before it dies:

curl -g -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}794435ee23aaa48d98a441969019c0f0d848fbb4' -H 'Content-Type: application/octet-stream' http://licom050.domain:9292/v2/schemas/image
GET call to glance-api for http://licom050.domain:9292/v2/schemas/image used request id req-ffdcea72-bb21-4f8f-bb3a-a28ad955b944

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 4149
X-Openstack-Request-Id: req-ffdcea72-bb21-4f8f-bb3a-a28ad955b944
Date: Fri, 21 Apr 2017 18:38:14 GMT
Connection: keep-alive

{"additionalProperties": {"type": "string"}, "name": "image", "links": [{"href": "{self}", "rel": "self"}, {"href": "{file}", "rel": "enclosure"}, {"href": "{schema}", "rel": "describedby"}], "properties": {"status": {"readOnly": true, "enum": ["queued", "saving", "active", "killed", "deleted", "pending_delete", "deactivated"], "type": "string", "description": "Status of the image"}, "tags": {"items": {"type": "string", "maxLength": 255}, "type": "array", "description": "List of strings related to the image"}, "kernel_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"], "description": "ID of image stored in Glance that should be used as the kernel when booting an AMI-style image.", "is_base": false}, "container_format": {"enum": [null, "ami", "ari", "aki", "bare", "ovf", "ova", "docker"], "type": ["null", "string"], "description": "Format of the container"}, "min_ram": {"type": "integer", "description": "Amount of ram (in MB) required to boot image."}, "ramdisk_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"], "description": "ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.", "is_base": false}, "locations": {"items": {"required": ["url", "metadata"], "type": "object", "properties": {"url": {"type": "string", "maxLength": 255}, "metadata": {"type": "object"}}}, "type": "array", "description": "A set of URLs to access the image file kept in external store"}, "visibility": {"enum": ["public", "private"], "type": "string", "description": "Scope of image accessibility"}, "updated_at": {"readOnly": true, "type": "string", "description": "Date and time of the last image modification"}, "owner": {"type": ["null", "string"], "description": "Owner of the image", "maxLength": 255}, "file": {"readOnly": true, "type": "string", "description": "An image file url"}, "min_disk": {"type": "integer", "description": "Amount of disk space (in GB) required to boot image."}, "virtual_size": {"readOnly": true, "type": ["null", "integer"], "description": "Virtual size of image in bytes"}, "id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "An identifier for the image"}, "size": {"readOnly": true, "type": ["null", "integer"], "description": "Size of image file in bytes"}, "instance_uuid": {"type": "string", "description": "Metadata which can be used to record which instance this image is associated with. (Informational only, does not create an instance snapshot.)", "is_base": false}, "os_distro": {"type": "string", "description": "Common name of operating system distribution as specified in http://docs.openstack.org/trunk/openstack-compute/admin/content/adding-images.html", "is_base": false}, "name": {"type": ["null", "string"], "description": "Descriptive name for the image", "maxLength": 255}, "checksum": {"readOnly": true, "type": ["null", "string"], "description": "md5 hash of image contents.", "maxLength": 32}, "created_at": {"readOnly": true, "type": "string", "description": "Date and time of image registration"}, "disk_format": {"enum": [null, "ami", "ari", "aki", "vhd", "vhdx", "vmdk", "raw", "qcow2", "vdi", "iso", "root-tar"], "type": ["null", "string"], "description": "Format of the disk"}, "os_version": {"type": "string", "description": "Operating system version as specified by the distributor", "is_base": false}, "protected": {"type": "boolean", "description": "If true, image will not be deletable."}, "architecture": {"type": "string", "description": "Operating system architecture as specified in http://docs.openstack.org/trunk/openstack-compute/admin/content/adding-images.html", "is_base": false}, "direct_url": {"readOnly": true, "type": "string", "description": "URL to access the image file kept in external store"}, "self": {"readOnly": true, "type": "string", "description": "An image self url"}, "schema": {"readOnly": true, "type": "string", "description": "An image schema url"}}}

REQ: curl -g -i -X POST http://licom050.domain:8776/v2/7e78c35600a24806afc30fc3bd207a38/volumes -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}794435ee23aaa48d98a441969019c0f0d848fbb4" -d '{"volume": {"status": "creating", "user_id": null, "name": "test", "imageRef": "60d97494-26a1-4835-bea9-4fefe2768bec", "availability_zone": null, "description": null, "multiattach": false, "attach_status": "detached", "volume_type": "lvm", "metadata": {}, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "project_id": null, "source_replica": null, "size": 1}}'
RESP: [400] Content-Length: 105 Content-Type: application/json; charset=UTF-8 X-Compute-Request-Id: req-1dacce16-4140-4fc7-8556-1e614064f76f X-Openstack-Request-Id: req-1dacce16-4140-4fc7-8556-1e614064f76f Date: Fri, 21 Apr 2017 18:38:14 GMT Connection: keep-alive
RESP BODY: {"badRequest": {"message": "Invalid image identifier or unable to access requested image.", "code": 400}}

POST call to volumev2 for http://licom050.domain:8776/v2/7e78c35600a24806afc30fc3bd207a38/volumes used request id req-1dacce16-4140-4fc7-8556-1e614064f76f
Invalid image identifier or unable to access requested image. (HTTP 400) (Request-ID: req-1dacce16-4140-4fc7-8556-1e614064f76f)
clean_up CreateVolume: Invalid image identifier or unable to access requested image. (HTTP 400) (Request-ID: req-1dacce16-4140-4fc7-8556-1e614064f76f)
END return value: 1

However, when I get my own openstack token via 'openstack token issue' and run the exact same CURL command with HTTP GET instead of POST, I get the correct data.

curl -g -i -X GET http://licom050.domain:8776/v2/7e78c35600a24806afc30fc3bd207a38/volumes -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABY-liE-bdhUL129qEXvuUFj_l-fm1hMXqPHv3ehkUizOLnGsLwpftD-XqmN0H6XI33HxZuAZgS8OhDKA9k24Vn-Iqgec7_OnSiT0D4Ig3pvWfVMjglx5hKux9HGadDk-wAXA7ZKB3Zo1VazoyJPA_GETWBtLPV4IcIMY5PAaAE7GR-AJg"
HTTP/1.1 200 OK
X-Compute-Request-Id: req-81739811-8fa2-4fe1-8634-55e986f2bfe4
Content-Type: application/json
Content-Length: 370
X-Openstack-Request-Id: req-81739811-8fa2-4fe1-8634-55e986f2bfe4
Date: Fri, 21 Apr 2017 19:12:12 GMT

{"volumes": [{"id": "edb3e378-4924-4221-b6ed-ae684865c51c", "links": [{"href": "http://licom050.domain:8776/v2/7e78c35600a24806afc30fc3bd207a38/volumes/edb3e378-4924-4221-b6ed-ae684865c51c", "rel": "self"}, {"href": "http://licom050.domain:8776/7e78c35600a24806afc30fc3bd207a38/volumes/edb3e378-4924-4221-b6ed-ae684865c51c", "rel": "bookmark"}], "name": "k"}]}

curl -g -i -X POST http://licom050.domain:8776/v2/7e78c35600a24806afc30fc3bd207a38/volumes -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: gAAAAABY-liE-bdhUL129qEXvuUFj_l-fm1hMXqPHv3ehkUizOLnGsLwpftD-XqmN0H6XI33HxZuAZgS8OhDKA9k24Vn-Iqgec7_OnSiT0D4Ig3pvWfVMjglx5hKux9HGadDk-wAXA7ZKB3Zo1VazoyJPA_GETWBtLPV4IcIMY5PAaAE7GR-AJg"
HTTP/1.1 400 Bad Request
Content-Length: 141
Content-Type: application/json; charset=UTF-8
X-Compute-Request-Id: req-11911d72-d68e-44df-b896-0bbce82e540b
X-Openstack-Request-Id: req-11911d72-d68e-44df-b896-0bbce82e540b
Date: Fri, 21 Apr 2017 19:12:43 GMT

Some part of cinder/glance is using HTTP POST intead of GET and that is preventing the creation of volumes from images. Either that or...some setting in my config is wrong and nobody's documenatation makes mention of it.

Revision history for this message
Steffen (bert2002) wrote :

Hi Asquirrel,
i think I am running in to the same problem. Did you find any solution? I am running glance 13.0.0 and cinder 9.1.2 on Newton.

Thanks.

Revision history for this message
Asquirrel (jemcaemc) wrote :

No. I'd have to dig through the presumably python, modules which drive Cinder to find where this request is getting generated and I'm so bad at Python I would probably find it around the time Openstack Zebra came out.

Revision history for this message
wangxiyuan (wangxiyuan) wrote :

Could you show your cinder-api's log? I guess there is something wrong during Cinder and Glance connection. Perhaps Cinder cant' get the image info from Glance.

Revision history for this message
Steffen (bert2002) wrote :

Hi, this is what I see in the cinder-api.logs:

2017-04-27 14:21:23.856 3771 INFO cinder.api.openstack.wsgi [req-8e4d8c8f-04ed-4ded-8707-d7f0914c82e5 217ce8c543b3495fa23b48a99cf9e4c5 3723cf7355e946efbf40f4f29ff845f6 - default default] POST http://controller:8776/v2/3723cf7355e946efbf40f4f29ff845f6/volumes
2017-04-27 14:21:23.858 3771 INFO cinder.api.v2.volumes [req-8e4d8c8f-04ed-4ded-8707-d7f0914c82e5 217ce8c543b3495fa23b48a99cf9e4c5 3723cf7355e946efbf40f4f29ff845f6 - default default] Create volume of 1 GB
2017-04-27 14:21:23.893 3771 INFO cinder.api.openstack.wsgi [req-8e4d8c8f-04ed-4ded-8707-d7f0914c82e5 217ce8c543b3495fa23b48a99cf9e4c5 3723cf7355e946efbf40f4f29ff845f6 - default default] HTTP exception thrown: Invalid image identifier or unable to access requested image.
2017-04-27 14:21:23.896 3771 INFO cinder.api.openstack.wsgi [req-8e4d8c8f-04ed-4ded-8707-d7f0914c82e5 217ce8c543b3495fa23b48a99cf9e4c5 3723cf7355e946efbf40f4f29ff845f6 - default default] http://controller:8776/v2/3723cf7355e946efbf40f4f29ff845f6/volumes returned with HTTP 400
2017-04-27 14:21:23.898 3771 INFO eventlet.wsgi.server [req-8e4d8c8f-04ed-4ded-8707-d7f0914c82e5 217ce8c543b3495fa23b48a99cf9e4c5 3723cf7355e946efbf40f4f29ff845f6 - default default] 172.16.102.99 "POST /v2/3723cf7355e946efbf40f4f29ff845f6/volumes HTTP/1.1" status: 400 len: 392 time: 0.0484591

On the compute nodes the ceph cinder user can access the images pool (tested with rbd). I can't think of anything why it should not be able to access it, but may be I am missing something.

Revision history for this message
Asquirrel (jemcaemc) wrote :
Download full text (4.8 KiB)

cinder-api.log when pushing 'Launch Instance':

2017-04-27 08:55:50.008 3667 INFO cinder.api.openstack.wsgi [req-367d8ecc-be4f-4869-a480-5866077a1f2a 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] GET http://controller:8776/v2/60ff991686af495fbab74e4c0f73290c/snapshots/detail?status=available
2017-04-27 08:55:50.078 3667 INFO cinder.volume.api [req-367d8ecc-be4f-4869-a480-5866077a1f2a 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] Get all snapshots completed successfully.
2017-04-27 08:55:50.082 3667 INFO cinder.api.openstack.wsgi [req-367d8ecc-be4f-4869-a480-5866077a1f2a 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] http://controller:8776/v2/60ff991686af495fbab74e4c0f73290c/snapshots/detail?status=available returned with HTTP 200
2017-04-27 08:55:50.086 3667 INFO eventlet.wsgi.server [req-367d8ecc-be4f-4869-a480-5866077a1f2a 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] 192.168.3.70 "GET /v2/60ff991686af495fbab74e4c0f73290c/snapshots/detail?status=available HTTP/1.1" status: 200 len: 279 time: 0.4997020
2017-04-27 08:55:50.127 3668 INFO cinder.api.openstack.wsgi [req-0b7e4f37-aff9-4297-919a-e3da1bd0549a 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] GET http://controller:8776/v2/60ff991686af495fbab74e4c0f73290c/os-volume-transfer/detail?bootable=1&status=available
2017-04-27 08:55:50.161 3668 INFO cinder.api.openstack.wsgi [req-0b7e4f37-aff9-4297-919a-e3da1bd0549a 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] http://controller:8776/v2/60ff991686af495fbab74e4c0f73290c/os-volume-transfer/detail?bootable=1&status=available returned with HTTP 200
2017-04-27 08:55:50.163 3668 INFO eventlet.wsgi.server [req-0b7e4f37-aff9-4297-919a-e3da1bd0549a 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] 192.168.3.70 "GET /v2/60ff991686af495fbab74e4c0f73290c/os-volume-transfer/detail?bootable=1&status=available HTTP/1.1" status: 200 len: 279 time: 0.5939879
2017-04-27 08:55:50.171 3667 INFO cinder.api.openstack.wsgi [req-5910b6ad-a36e-4c08-b3a4-737215fa5093 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] GET http://controller:8776/v2/60ff991686af495fbab74e4c0f73290c/volumes/detail?bootable=1&status=available
2017-04-27 08:55:50.222 3667 INFO cinder.volume.api [req-5910b6ad-a36e-4c08-b3a4-737215fa5093 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] Get all volumes completed successfully.
2017-04-27 08:55:50.224 3667 INFO cinder.api.openstack.wsgi [req-5910b6ad-a36e-4c08-b3a4-737215fa5093 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] http://controller:8776/v2/60ff991686af495fbab74e4c0f73290c/volumes/detail?bootable=1&status=available returned with HTTP 200
2017-04-27 08:55:50.225 3667 INFO eventlet.wsgi.server [req-5910b6ad-a36e-4c08-b3a4-737215fa5093 66fd472eb3274dcdbdbc286a8811fbb2 60ff991686af495fbab74e4c0f73290c - default default] 192.168.3.70 "GET /v2/60ff991686af495fbab74e4c0f73290...

Read more...

Revision history for this message
Asquirrel (jemcaemc) wrote :

Er, crap. I can't edit my previous comment. Controller = 192.168.3.70

Revision history for this message
Steffen (bert2002) wrote :

Hi, I got it to start a image. I had to enable v1 for glance again.

glance-api.conf
...
enable_v1_api=True
enable_v2_api=True
...

Can you try this?

Revision history for this message
Asquirrel (jemcaemc) wrote :

No joy, problem persists. That said, api=True is actually the default setting (and I had not set them either way) so...'it was already on'.

Revision history for this message
Alejandro Comisario (alejandro-f) wrote :

Same happening in ubuntu/OCATA packages.

Any news ?

Revision history for this message
Alejandro Comisario (alejandro-f) wrote :

regarding comment #8 it's not absolutely right, it didn't worked in my case until i explicitly activated v1 on glance config

Revision history for this message
Alejandro Comisario (alejandro-f) wrote :

JFTR, found the final solution, and think is a configuration issue.

on glance-api.conf:

[DEFAULT]
enable_v1_api=False
enable_v2_api=True

on cinder.conf

[DEFAULT]
glance_api_servers = http://glance_VIP_or_IP:9292
glance_api_version = 2

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.