502 Bad gateway error on image-create

[Ellen Batbouta]

The glance code that I am using is from the upstream master branch (Pike) and I just pulled down the latest code this morning and still can reproduce this problem.

Up until about 2 weeks ago, I was able to upload my database image into glance using this command:

glance image-create --name 'Db 12.1.0.2' --file Oracle12201DBRAC_x86_64-xvdb.qcow2 --container-format bare --disk-format qcow2

However, now it fails as follows:

 glance --debug image-create --name 'Db 12.1.0.2' --file Oracle12201DBRAC_x86_64-xvdb.qcow2 --container-format bare --disk-format qcow2

DEBUG:keystoneauth.session:REQ: curl -g -i -X GET http://172.16.35.10/identity -H "Accept: application/json" -H "User-Agent: glance keystoneauth1/2.21.0 python-requests/2.18.1 CPython/2.7.12"
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 172.16.35.10
DEBUG:urllib3.connectionpool:http://172.16.35.10:80 "GET /identity HTTP/1.1" 300 606
DEBUG:keystoneauth.session:RESP: [300] Date: Wed, 12 Jul 2017 14:26:39 GMT Server: Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token Content-Type: application/json Content-Length: 606 Connection: close
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://172.16.35.10/identity/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://172.16.35.10/identity/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

DEBUG:keystoneauth.identity.v3.base:Making authentication request to http://172.16.35.10/identity/v3/auth/tokens
DEBUG:urllib3.connectionpool:Resetting dropped connection: 172.16.35.10
DEBUG:urllib3.connectionpool:http://172.16.35.10:80 "POST /identity/v3/auth/tokens HTTP/1.1" 201 4893
DEBUG:keystoneauth.identity.v3.base:{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "325205c52aba4b31801e2d71ec95483b", "name": "admin"}], "expires_at": "2017-07-12T15:26:40.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "4aa1233111e140b2a1e4ba170881f092", "name": "demo"}, "catalog": [{"endpoints": [{"url": "http://172.16.35.10/image", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "0d10d85bc3ae4e13a49ed344fcf6f737"}], "type": "image", "id": "01c2acd1845d4dd28c5b69351fa0dbf3", "name": "glance"}, {"endpoints": [{"url": "http://172.16.35.10:8004/v1/4aa1233111e140b2a1e4ba170881f092", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "0fbba7f276e44921ba112edd1e157561"}, {"url": "http://172.16.35.10:8004/v1/4aa1233111e140b2a1e4ba170881f092", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "72abdff47e2940f09db32720b709d01f"}, {"url": "http://172.16.35.10:8004/v1/4aa1233111e140b2a1e4ba170881f092", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "d2789811c71342d69d69e45c09268ebc"}], "type": "orchestration", "id": "343101b65cba48afafb5b70fcbae5c3d", "name": "heat"}, {"endpoints": [{"url": "http://172.16.35.10/compute/v2/4aa1233111e140b2a1e4ba170881f092", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "d7fe183ce05d46d986c7ec7600b583a5"}], "type": "compute_legacy", "id": "3d75e8b88ed14f95b162b5398acfde82", "name": "nova_legacy"}, {"endpoints": [{"url": "http://172.16.35.10:8082", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "65e5e92c5646468583f033cfb05ae0cb"}, {"url": "http://172.16.35.10:8082", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "8cbae4cbce354314aa5f2b5e5c4e4592"}, {"url": "http://172.16.35.10:8082", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "d761a53278654ac690fb56b42752c1a4"}], "type": "application-catalog", "id": "42038a7b5c744771842615613d21f2ba", "name": "murano"}, {"endpoints": [{"url": "http://172.16.35.10/identity", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "4b5c6e820b9446f586be1f64da5ae2f6"}, {"url": "http://172.16.35.10/identity", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f6c18a74f19a4b728eeb5f3916dde7c1"}], "type": "identity", "id": "518a08b01ddf4c38ba2dfb0481aa196f", "name": "keystone"}, {"endpoints": [{"url": "http://172.16.35.10:8776/v1/4aa1233111e140b2a1e4ba170881f092", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "26a34fe1d409488dbc1d4178642de6f1"}], "type": "volume", "id": "57e3dbcfe62741b1bbc735d137caf3c7", "name": "cinder"}, {"endpoints": [{"url": "http://172.16.35.10:9696/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "1ef656cb68144f10a60b3167fc4b6602"}], "type": "network", "id": "7d8a5129ef114b6baf952bec7b3ac370", "name": "neutron"}, {"endpoints": [{"url": "http://172.16.35.10:8776/v3/4aa1233111e140b2a1e4ba170881f092", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "0dcc6673cd9e43e38d4ff98d5766dd4a"}], "type": "volumev3", "id": "7d8ec460d29b4306b2dba773c4eb2d42", "name": "cinderv3"}, {"endpoints": [{"url": "http://172.16.35.10/placement", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "a8aa98db377e4251b0b0a2860875e65e"}], "type": "placement", "id": "b0069601a94547878fe2247671a67e42", "name": "placement"}, {"endpoints": [{"url": "http://172.16.35.10:8776/v2/4aa1233111e140b2a1e4ba170881f092", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "d4fafa2ac6ef42429280d6b95a5b60e4"}], "type": "volumev2", "id": "bd4751a9312545fa9dddfc9c9ff9443f", "name": "cinderv2"}, {"endpoints": [{"url": "http://172.16.35.10:8000/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "78a41e339c03440890b8317353a5fddf"}, {"url": "http://172.16.35.10:8000/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "c126e21adbe347459c9d90017a5661e3"}, {"url": "http://172.16.35.10:8000/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "cbf70d8f60e94bf8974b289e6e91d655"}], "type": "cloudformation", "id": "bdf32e61289d4fb687d4075d0e9f7fab", "name": "heat-cfn"}, {"endpoints": [{"url": "http://172.16.35.10/compute/v2.1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "59a8b8c8c8234e3a91b9c49c61b91567"}], "type": "compute", "id": "bf544eeb97c54b35819d60fa7b3a99e2", "name": "nova"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "4508a16b6ed34cd989feb695cf0bf1c1"}, "audit_ids": ["Ob5pbubJSuGlK_nkZUlJYQ"], "issued_at": "2017-07-12T14:26:40.000000Z"}}
DEBUG:keystoneauth.session:REQ: curl -g -i -X GET http://172.16.35.10/image/v2/schemas/image -H "User-Agent: python-glanceclient" -H "Content-Type: application/octet-stream" -H "X-Auth-Token: {SHA1}918f5d5463ced3694e5a1d252c562fef879d4578"
DEBUG:urllib3.connectionpool:Resetting dropped connection: 172.16.35.10
DEBUG:urllib3.connectionpool:http://172.16.35.10:80 "GET /image/v2/schemas/image HTTP/1.1" 200 4137
DEBUG:keystoneauth.session:RESP: [200] Date: Wed, 12 Jul 2017 14:26:40 GMT Server: Apache/2.4.18 (Ubuntu) Content-Type: application/json Content-Length: 4137 x-openstack-request-id: req-d8714cbf-77aa-439e-93cc-e9fe351883a5 Connection: close
RESP BODY: {"additionalProperties": {"type": "string"}, "name": "image", "links": [{"href": "{self}", "rel": "self"}, {"href": "{file}", "rel": "enclosure"}, {"href": "{schema}", "rel": "describedby"}], "properties": {"status": {"readOnly": true, "enum": ["queued", "saving", "active", "killed", "deleted", "pending_delete", "deactivated"], "type": "string", "description": "Status of the image"}, "tags": {"items": {"type": "string", "maxLength": 255}, "type": "array", "description": "List of strings related to the image"}, "kernel_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"], "description": "ID of image stored in Glance that should be used as the kernel when booting an AMI-style image.", "is_base": false}, "container_format": {"enum": [null, "ami", "ari", "aki", "bare", "ovf", "ova", "docker"], "type": ["null", "string"], "description": "Format of the container"}, "min_ram": {"type": "integer", "description": "Amount of ram (in MB) required to boot image."}, "ramdisk_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"], "description": "ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.", "is_base": false}, "locations": {"items": {"required": ["url", "metadata"], "type": "object", "properties": {"url": {"type": "string", "maxLength": 255}, "metadata": {"type": "object"}}}, "type": "array", "description": "A set of URLs to access the image file kept in external store"}, "visibility": {"enum": ["community", "public", "private", "shared"], "type": "string", "description": "Scope of image accessibility"}, "updated_at": {"readOnly": true, "type": "string", "description": "Date and time of the last image modification"}, "owner": {"type": ["null", "string"], "description": "Owner of the image", "maxLength": 255}, "file": {"readOnly": true, "type": "string", "description": "An image file url"}, "min_disk": {"type": "integer", "description": "Amount of disk space (in GB) required to boot image."}, "virtual_size": {"readOnly": true, "type": ["null", "integer"], "description": "Virtual size of image in bytes"}, "id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "An identifier for the image"}, "size": {"readOnly": true, "type": ["null", "integer"], "description": "Size of image file in bytes"}, "instance_uuid": {"type": "string", "description": "Metadata which can be used to record which instance this image is associated with. (Informational only, does not create an instance snapshot.)", "is_base": false}, "os_distro": {"type": "string", "description": "Common name of operating system distribution as specified in https://docs.openstack.org/cli-reference/glance-property-keys.html", "is_base": false}, "name": {"type": ["null", "string"], "description": "Descriptive name for the image", "maxLength": 255}, "checksum": {"readOnly": true, "type": ["null", "string"], "description": "md5 hash of image contents.", "maxLength": 32}, "created_at": {"readOnly": true, "type": "string", "description": "Date and time of image registration"}, "disk_format": {"enum": [null, "ami", "ari", "aki", "vhd", "vhdx", "vmdk", "raw", "qcow2", "vdi", "iso", "ploop"], "type": ["null", "string"], "description": "Format of the disk"}, "os_version": {"type": "string", "description": "Operating system version as specified by the distributor", "is_base": false}, "protected": {"type": "boolean", "description": "If true, image will not be deletable."}, "architecture": {"type": "string", "description": "Operating system architecture as specified in https://docs.openstack.org/cli-reference/glance-property-keys.html", "is_base": false}, "direct_url": {"readOnly": true, "type": "string", "description": "URL to access the image file kept in external store"}, "self": {"readOnly": true, "type": "string", "description": "An image self url"}, "schema": {"readOnly": true, "type": "string", "description": "An image schema url"}}}

DEBUG:keystoneauth.session:GET call to image for http://172.16.35.10/image/v2/schemas/image used request id req-d8714cbf-77aa-439e-93cc-e9fe351883a5
DEBUG:keystoneauth.session:REQ: curl -g -i -X GET http://172.16.35.10/image/v2/schemas/image -H "User-Agent: python-glanceclient" -H "Content-Type: application/octet-stream" -H "X-Auth-Token: {SHA1}918f5d5463ced3694e5a1d252c562fef879d4578"
DEBUG:urllib3.connectionpool:Resetting dropped connection: 172.16.35.10
DEBUG:urllib3.connectionpool:http://172.16.35.10:80 "GET /image/v2/schemas/image HTTP/1.1" 200 4137
DEBUG:keystoneauth.session:RESP: [200] Date: Wed, 12 Jul 2017 14:26:40 GMT Server: Apache/2.4.18 (Ubuntu) Content-Type: application/json Content-Length: 4137 x-openstack-request-id: req-df6bec7c-b700-4438-a188-cf7d7afb3f69 Connection: close
RESP BODY: {"additionalProperties": {"type": "string"}, "name": "image", "links": [{"href": "{self}", "rel": "self"}, {"href": "{file}", "rel": "enclosure"}, {"href": "{schema}", "rel": "describedby"}], "properties": {"status": {"readOnly": true, "enum": ["queued", "saving", "active", "killed", "deleted", "pending_delete", "deactivated"], "type": "string", "description": "Status of the image"}, "tags": {"items": {"type": "string", "maxLength": 255}, "type": "array", "description": "List of strings related to the image"}, "kernel_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"], "description": "ID of image stored in Glance that should be used as the kernel when booting an AMI-style image.", "is_base": false}, "container_format": {"enum": [null, "ami", "ari", "aki", "bare", "ovf", "ova", "docker"], "type": ["null", "string"], "description": "Format of the container"}, "min_ram": {"type": "integer", "description": "Amount of ram (in MB) required to boot image."}, "ramdisk_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"], "description": "ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.", "is_base": false}, "locations": {"items": {"required": ["url", "metadata"], "type": "object", "properties": {"url": {"type": "string", "maxLength": 255}, "metadata": {"type": "object"}}}, "type": "array", "description": "A set of URLs to access the image file kept in external store"}, "visibility": {"enum": ["community", "public", "private", "shared"], "type": "string", "description": "Scope of image accessibility"}, "updated_at": {"readOnly": true, "type": "string", "description": "Date and time of the last image modification"}, "owner": {"type": ["null", "string"], "description": "Owner of the image", "maxLength": 255}, "file": {"readOnly": true, "type": "string", "description": "An image file url"}, "min_disk": {"type": "integer", "description": "Amount of disk space (in GB) required to boot image."}, "virtual_size": {"readOnly": true, "type": ["null", "integer"], "description": "Virtual size of image in bytes"}, "id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "An identifier for the image"}, "size": {"readOnly": true, "type": ["null", "integer"], "description": "Size of image file in bytes"}, "instance_uuid": {"type": "string", "description": "Metadata which can be used to record which instance this image is associated with. (Informational only, does not create an instance snapshot.)", "is_base": false}, "os_distro": {"type": "string", "description": "Common name of operating system distribution as specified in https://docs.openstack.org/cli-reference/glance-property-keys.html", "is_base": false}, "name": {"type": ["null", "string"], "description": "Descriptive name for the image", "maxLength": 255}, "checksum": {"readOnly": true, "type": ["null", "string"], "description": "md5 hash of image contents.", "maxLength": 32}, "created_at": {"readOnly": true, "type": "string", "description": "Date and time of image registration"}, "disk_format": {"enum": [null, "ami", "ari", "aki", "vhd", "vhdx", "vmdk", "raw", "qcow2", "vdi", "iso", "ploop"], "type": ["null", "string"], "description": "Format of the disk"}, "os_version": {"type": "string", "description": "Operating system version as specified by the distributor", "is_base": false}, "protected": {"type": "boolean", "description": "If true, image will not be deletable."}, "architecture": {"type": "string", "description": "Operating system architecture as specified in https://docs.openstack.org/cli-reference/glance-property-keys.html", "is_base": false}, "direct_url": {"readOnly": true, "type": "string", "description": "URL to access the image file kept in external store"}, "self": {"readOnly": true, "type": "string", "description": "An image self url"}, "schema": {"readOnly": true, "type": "string", "description": "An image schema url"}}}

DEBUG:keystoneauth.session:GET call to image for http://172.16.35.10/image/v2/schemas/image used request id req-df6bec7c-b700-4438-a188-cf7d7afb3f69
DEBUG:keystoneauth.session:REQ: curl -g -i -X POST http://172.16.35.10/image/v2/images -H "User-Agent: python-glanceclient" -H "Content-Type: application/json" -H "X-Auth-Token: {SHA1}918f5d5463ced3694e5a1d252c562fef879d4578" -d '{"container_format": "bare", "disk_format": "qcow2", "name": "Db 12.1.0.2"}'
DEBUG:urllib3.connectionpool:Resetting dropped connection: 172.16.35.10
DEBUG:urllib3.connectionpool:http://172.16.35.10:80 "POST /image/v2/images HTTP/1.1" 201 555
DEBUG:keystoneauth.session:RESP: [201] Date: Wed, 12 Jul 2017 14:26:40 GMT Server: Apache/2.4.18 (Ubuntu) Content-Length: 555 Content-Type: application/json Location: http://127.0.0.1:60999/v2/images/11f8a7ab-375a-47f9-8b69-10ad48e229b0 x-openstack-request-id: req-b1a50e1b-006e-4322-b0a4-dc91367b4fcd Connection: close
RESP BODY: {"status": "queued", "name": "Db 12.1.0.2", "tags": [], "container_format": "bare", "created_at": "2017-07-12T14:26:40Z", "size": null, "disk_format": "qcow2", "updated_at": "2017-07-12T14:26:40Z", "visibility": "shared", "self": "/v2/images/11f8a7ab-375a-47f9-8b69-10ad48e229b0", "min_disk": 0, "protected": false, "id": "11f8a7ab-375a-47f9-8b69-10ad48e229b0", "file": "/v2/images/11f8a7ab-375a-47f9-8b69-10ad48e229b0/file", "checksum": null, "owner": "4aa1233111e140b2a1e4ba170881f092", "virtual_size": null, "min_ram": 0, "schema": "/v2/schemas/image"}

DEBUG:keystoneauth.session:POST call to image for http://172.16.35.10/image/v2/images used request id req-b1a50e1b-006e-4322-b0a4-dc91367b4fcd
DEBUG:keystoneauth.session:REQ: curl -g -i -X PUT http://172.16.35.10/image/v2/images/11f8a7ab-375a-47f9-8b69-10ad48e229b0/file -H "User-Agent: python-glanceclient" -H "Content-Type: application/octet-stream" -H "X-Auth-Token: {SHA1}918f5d5463ced3694e5a1d252c562fef879d4578" -d '<generator object _chunk_body at 0x7fd0d879cd20>'
DEBUG:urllib3.connectionpool:Resetting dropped connection: 172.16.35.10
DEBUG:keystoneauth.session:RESP: [502] Date: Wed, 12 Jul 2017 14:26:40 GMT Server: Apache/2.4.18 (Ubuntu) Content-Length: 310 Connection: close Content-Type: text/html; charset=iso-8859-1
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.

Request returned failure status 502.
DEBUG:glanceclient.common.http:Request returned failure status 502.
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2017-07-12T14:26:40Z |
| disk_format | qcow2 |
| id | 11f8a7ab-375a-47f9-8b69-10ad48e229b0 |
| min_disk | 0 |
| min_ram | 0 |
| name | Db 12.1.0.2 |
| owner | 4aa1233111e140b2a1e4ba170881f092 |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2017-07-12T14:26:40Z |
| virtual_size | None |
| visibility | shared |
+------------------+--------------------------------------+
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/shell.py", line 706, in main
    OpenStackImagesShell().main(argv)
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/shell.py", line 610, in main
    args.func(client, args)
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 98, in func_wrapper
    return func(gc, args)
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py", line 86, in do_image_create
    do_image_upload(gc, args)
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/v2/shell.py", line 324, in do_image_upload
    gc.images.upload(args.id, image_data, args.size)
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 545, in inner
    return RequestIdProxy(wrapped(*args, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 231, in upload
    resp, body = self.http_client.put(url, headers=hdrs, data=body)
  File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 243, in put
    return self.request(url, 'PUT', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py", line 352, in request
    return self._handle_response(resp)
  File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py", line 98, in _handle_response
    raise exc.from_response(resp, resp.content)
HTTPBadGateway: 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.18 (Ubuntu) Server at 172.16.35.10 Port 80 (HTTP 502)
502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.18 (Ubuntu) Server at 172.16.35.10 Port 80 (HTTP 502)
labuser@eb-stack1:/opt/stack$

Any ideas?

I have been able to work around it (by enabled the v1 api) using the v1 parameter, --location instead of using --file. But this is only a temporary fix.

I can load smaller images into glance just fine with image-create. This database image is about 16Gb in size.

I also posted a comment in ask.openstack.org where the problem was the same as this. See:

https://ask.openstack.org/en/question/108140/openstack-windows-image-stuck-in-queuing/

Thank you.

[Abhishek Kekane (abhishek-kekane)]

Initial finding:
This is something related to uwsgi socket timeout. Default value for socket-timeout is 30 in /etc/glance/glance-uwsgi.ini file.

For testing purpose I have tried to create 10 GB image using create command:
$ time glance --debug image-create --name dsl --file gentoo_root.img --disk-format iso --container-format bare

real 2m48.539s
user 0m4.076s
sys 0m10.012s

It has failed with “502 bad gateway” after 3 minutes. Then I have increased this timeout to 60 and restarted glance-api service and ran above command again.

$ time glance --debug image-create --name dsl --file gentoo_root.img --disk-format iso --container-format bare

Output:
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 2dd26c4d4799ebd29fa31e48d49e8e53 |
| container_format | bare |
| created_at | 2017-07-13T11:38:24Z |
| disk_format | iso |
| id | 52ffe21f-398d-4d90-8ad7-0b2667f131a9 |
| min_disk | 0 |
| min_ram | 0 |
| name | dsl |
| owner | 6d3cd4fd4059449286a881373d27f1b2 |
| protected | False |
| size | 10737418240 |
| status | active |
| tags | [] |
| updated_at | 2017-07-13T11:46:30Z |
| virtual_size | None |
| visibility | shared |
+------------------+--------------------------------------+

real 8m12.245s
user 0m4.416s
sys 0m10.212s

After this I was able to create image without any issue.

So another workaround is to set socket-timeout to higher value in /etc/glance/glance-uwsgi.ini file.

[Matthew Treinish (treinish)]

My concern with just bumping the uwsgi timeout is that is nothing in this bug that points to why that fixes it. When we bumped it from the default of 4 secs to 30 that was supposed to be an overly conservative value because we were seeing socket timeouts while uwsgi was waiting for chunked data on the wire at high load situations.

I'm afraid just doing that without classifying the underlying cause of uwsgi not responding (or just closing the socket, its not clear from anything here) which causes the apache proxy to return 502 will mask a real bug somewhere else. My other concern with just bumping the timeout is you're only testing with a 10GB what if you want to upload a 200GB image file, or a 2TB image file?

[Rabi Mishra (rabi)]

I think we're encountering this issue intermittently in our gate.

http://logs.openstack.org/20/480820/10/gate/gate-heat-dsvm-functional-convg-mysql-lbaasv2-ubuntu-xenial/69db418/console.html#_2017-07-13_08_01_38_111555

27 failures in last 7 days.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22The%20proxy%20server%20received%20an%20invalid%3A%20response%20from%20an%20upstream%20server%5C%22%20AND%20project%3A%20%5C%22openstack%2Fheat%5C%22

[Abhishek Kekane (abhishek-kekane)]

Recent update,

I have ran couple of scenarios by increasing socket-timeout,
Set socket-timeout = 300 in glance-uwsgi.ini

Ran below scenarios:
1. create image of 10 GB - Success
2. Create image of 20 GB - Success
3. Create image of 40 GB - Success
4. Create image of 70 GB - Success
5. Concurrently create 3 images of 15 GB each - Success
6. Concurrently create 3 images of 20 GB each - Success

After reverting socket-timeout to original (30) below were the results for same scenarios:

1. create image of 10 GB - Failed intermittently
2. Create image of 20 GB - Failed
3. Create image of 40 GB - Failed
4. Create image of 70 GB - Failed
5. Concurrently create 3 images of 15 GB each - Failed
6. Concurrently create 3 images of 20 GB each - Failed

As pointed out by Matthew in comment #2 I will try to figure out why uwsgi is not responding or closing the socket.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/484611

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rico Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/484611
Reason: We not required this anymore

[Rico Lin (rico-lin)]

We still require this fix, but we have a patch to work around this issue. Let's try to fix this in Glance

[Dave McCowan (dave-mccowan)]

Any update on this? Barbican is also getting a 502 response in some tests after introducing uwsgi.

[Jianghua Wang (wjh-fresh)]

I'm also wondering if anyone is working on this issue. I met the same error.

[Brian Rosmaita (brian-rosmaita)]

@rico-lin: What workaround did you use that allowed you to abandon your skip-test patch?

[Brian Rosmaita (brian-rosmaita)]

Note to anyone encountering this problem in DevStack:

Somehow apache is losing connectivity with glance, causing the 502 response. It would be helpful if you could paste log extracts from apache and glance at paste.openstack.org and leave a link in this bug.

For a temporary workaround, see Abhishek's comment in #1 above. But before doing the workaround, please paste the logs for us!

[Ellen Batbouta (ellen-batbouta)]

I can easily reproduce this problem. When the problem occurs, there is no info written to the g-api.log and g-reg.log files. I will look for more pertinent log files.

[Ellen Batbouta (ellen-batbouta)]

The log file /var/log/apache2/access.log does not have any info either. Any other ideas of where to look for additional info?

[Matthew Treinish (treinish)]

An interesting thing is we recently discovered devstack was misconfiguring apache and had apache caching all the chunked requests to send the glance api a request with a content-length. (which obviously can take more than 30secs on a large image) I'm wondering if the fix for that bug: https://review.openstack.org/#/c/492651/ will also address this. If people who are effected by this issue could test if they still have it with 492651 applied and let me know that'll be useful in ruling that out.

[Eric Fried (efried)]

@treinish, yes, with your fix I encountered the 502.

And so did jenkins: http://logs.openstack.org/51/492651/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/14746f6/logs/devstack-early.txt.gz :)

[Eric Fried (efried)]

...and increasing the timeout per comment #1 doesn't seem to do anything.

FWIW, I get the error way quicker than whatever the timeout is set to:

neo@p8-100-neo:/opt/stack/devstack$ openstack image create --container-format bare --disk-format raw --file ~/base_os.img --public base_os
502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.18 (Ubuntu) Server at 9.47.79.13 Port 80 (HTTP 502)
neo@p8-100-neo:/opt/stack/devstack$ vi /etc/glance/glance-uwsgi.ini
neo@p8-100-neo:/opt/stack/devstack$ sudo systemctl restart devstack@g-* apache2.service
neo@p8-100-neo:/opt/stack/devstack$ openstack image create --container-format bare --disk-format raw --file ~/base_os.img --public base_os
502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.18 (Ubuntu) Server at 9.47.79.13 Port 80 (HTTP 502)
neo@p8-100-neo:/opt/stack/devstack$ time openstack image create --container-format bare --disk-format raw --file ~/base_os.img --public base_os
502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.18 (Ubuntu) Server at 9.47.79.13 Port 80 (HTTP 502)

real 0m8.911s
user 0m2.440s
sys 0m2.100s

Which logs would you like?

[Eric Fried (efried)]

journalctl had logs for g-api (http://paste.openstack.org/raw/618100/) but NONE for apache2 or g-reg.

[Matthew Treinish (treinish)]

502 isn't a unique error condition for this bug, since apache will return it whenever something goes wrong in it's communication to uwsgi. In case of rev 1 of that patch, both uwsgi and apache were misconfigured for handling chunked data which was triggering uwsgi to reset the connection. (causing the 502) I've uploaded a second rev that should address that issue in rev 1 and, at least for me locally, seems to work.

[Matthew Treinish (treinish)]

So just following up here using https://review.openstack.org/#/c/493654/ and glance and https://review.openstack.org/492651 on devstack should resolve this issue without requiring bumping the socket timeout. Once those 2 patches land I'm planning to close this bug. So if people who were affected by this bug could test running with those 2 patches and let me know if they still have issues.

[Ellen Batbouta (ellen-batbouta)]

Success! I have taken the changes from both reviews, https://review.openstack.org/#/c/493654/ and https://review.openstack.org/492651 and my large database image now works with the glance image-create comnmand. I no longer need to incresae the socket timeout. I hope these changes get merged soon!

[Abhishek Kekane (abhishek-kekane)]

I have taken the changes from both reviews, https://review.openstack.org/#/c/493654/ (Glance) and https://review.openstack.org/492651 (devstack) and,

Ran below scenarios:
1. create image of 10 GB - Success
2. Create image of 20 GB - Success
3. Create image of 40 GB - Success
4. Create image of 70 GB - Success
5. Concurrently create 3 images of 15 GB each - Success
6. Concurrently create 3 images of 20 GB each - Success

Thanks Matthew for working on this.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/493654
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=bf7887a102ca33dd46ef24b67b07eb77154ce133
Submitter: Zuul
Branch: master

commit bf7887a102ca33dd46ef24b67b07eb77154ce133
Author: Matthew Treinish <email address hidden>
Date: Wed Aug 16 10:29:46 2017 -0400

    Replace body_file with class to call uwsgi.chunked_read()

    Since the WebOb 1.7 release webob doesn't natively support receiving
    chunked transfer encoding bodies. [1] When glance is run under the
    eventlet wsgi server this was fine, because eventlet will dechunk the
    input on read() (or readline()) calls, so from the webob perspective
    it's just a file object. However, the effort to remove the dependence
    on using eventlet as the web server and deploy glance as a traditional
    wsgi script we lose this mechanism. The wsgi spec doesn't provide a
    consistent mechanism

    When we run glance under uwsgi the uwsgi server provides an api to read
    chunked data. [2] However, we need to explicitly call that api when to
    dechunk the data and pass it to glance code which expects a file object.
    This commit solves this issue by creating a fake file class that will
    call the chunked_read() api from uwsgi on read() calls. This object is
    then used if we're running the api code under uwsgi and the body has a
    chunked transfer-encoding.

    In conjuction with devstack change
    Iab2e2848877fa1497008d18c05b0154892941589 this closes glance bug 1703856

    [1] https://docs.pylonsproject.org/projects/webob/en/stable/whatsnew-1.7.html#backwards-incompatibility
    [2] http://uwsgi-docs.readthedocs.io/en/latest/Chunked.html

    Partial-bug 1703856

    Co-Authored-By: Chris Dent <email address hidden>

    Change-Id: Idf6b4b891ba31cccbeb53d373b40fce5380cea64

[Matthew Treinish (treinish)]

The devstack change has also merged fixing this issue completely

[Brian Rosmaita (brian-rosmaita)]

This is included in the Glance Queens-1 milestone release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/525994

[OpenStack Infra (hudson-openstack)]

Change abandoned by wangxiyuan (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/525994
Reason: Abandon now. Better to stop supporting uwsgi for Glance in devstack in Q, P