In horizon, uploading a large image to glance failed with gateway timeout.

Bug #1675637 reported by Chao Guo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
High
Chao Guo

Bug Description

I uploaded a large image(>7GB) using horizon , reported 'data' is not defined.

In horizion log:
[Fri Mar 24 13:28:28.338914 2017] [wsgi:error] [pid 27:tid 140129894897408] Unable to retrieve project list.
[Fri Mar 24 13:28:28.339029 2017] [wsgi:error] [pid 27:tid 140129894897408] Traceback (most recent call last):
[Fri Mar 24 13:28:28.339046 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_auth/user.py", line 318, in authorized_tenants
[Fri Mar 24 13:28:28.339061 2017] [wsgi:error] [pid 27:tid 140129894897408] is_federated=self.is_federated)
[Fri Mar 24 13:28:28.339074 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../openstack_auth/utils.py", line 366, in get_project_list
[Fri Mar 24 13:28:28.339088 2017] [wsgi:error] [pid 27:tid 140129894897408] projects = client.projects.list(user=kwargs.get('user_id'))
[Fri Mar 24 13:28:28.339101 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../positional/__init__.py", line 101, in inner
[Fri Mar 24 13:28:28.339149 2017] [wsgi:error] [pid 27:tid 140129894897408] return wrapped(*args, **kwargs)
[Fri Mar 24 13:28:28.339163 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/v3/projects.py", line 119, in list
[Fri Mar 24 13:28:28.339175 2017] [wsgi:error] [pid 27:tid 140129894897408] **kwargs)
[Fri Mar 24 13:28:28.339187 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/base.py", line 74, in func
[Fri Mar 24 13:28:28.339199 2017] [wsgi:error] [pid 27:tid 140129894897408] return f(*args, **new_kwargs)
[Fri Mar 24 13:28:28.339211 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/base.py", line 386, in list
[Fri Mar 24 13:28:28.339223 2017] [wsgi:error] [pid 27:tid 140129894897408] self.collection_key)
[Fri Mar 24 13:28:28.339235 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneclient/base.py", line 124, in _list
[Fri Mar 24 13:28:28.339247 2017] [wsgi:error] [pid 27:tid 140129894897408] resp, body = self.client.get(url, **kwargs)
[Fri Mar 24 13:28:28.339259 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/adapter.py", line 187, in get
[Fri Mar 24 13:28:28.339270 2017] [wsgi:error] [pid 27:tid 140129894897408] return self.request(url, 'GET', **kwargs)
[Fri Mar 24 13:28:28.339282 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/adapter.py", line 344, in request
[Fri Mar 24 13:28:28.339294 2017] [wsgi:error] [pid 27:tid 140129894897408] resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
[Fri Mar 24 13:28:28.339306 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/adapter.py", line 112, in request
[Fri Mar 24 13:28:28.339318 2017] [wsgi:error] [pid 27:tid 140129894897408] return self.session.request(url, method, **kwargs)
[Fri Mar 24 13:28:28.339329 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../positional/__init__.py", line 101, in inner
[Fri Mar 24 13:28:28.339341 2017] [wsgi:error] [pid 27:tid 140129894897408] return wrapped(*args, **kwargs)
[Fri Mar 24 13:28:28.339352 2017] [wsgi:error] [pid 27:tid 140129894897408] File "/var/lib/kolla/venv/lib/python2.7/site-packages/openstack_dashboard/wsgi/../../keystoneauth1/session.py", line 594, in request
[Fri Mar 24 13:28:28.339364 2017] [wsgi:error] [pid 27:tid 140129894897408] raise exceptions.from_response(resp, method, url)
[Fri Mar 24 13:28:28.339375 2017] [wsgi:error] [pid 27:tid 140129894897408] GatewayTimeout: Gateway Timeout (HTTP 504)

In glance-api log:
2017-03-24 12:48:48.310 22 INFO eventlet.wsgi.server [req-af459a85-fa9f-4c71-9b54-0042e0955ee1 ac9f0c728bcf4d9a8ec1c9a6359b37b1 8ebc162ad2ad473fb0f7ac11834b41a6 - default default] Traceback (most recent call last):
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 506, in handle_one_response
    write(b''.join(towrite))
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 453, in write
    wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer

Chao Guo (jimmygc)
Changed in kolla:
assignee: nobody → Chao Guo (jimmygc)
Chao Guo (jimmygc)
affects: kolla → kolla-ansible
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.openstack.org/449440

Changed in kolla-ansible:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.openstack.org/449440
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=538a9ebf2d2cbfea2510e3d80fdbabae0be62b6f
Submitter: Jenkins
Branch: master

commit 538a9ebf2d2cbfea2510e3d80fdbabae0be62b6f
Author: jimmygc <email address hidden>
Date: Fri Mar 24 14:19:16 2017 +0800

    Fix glance Upload large image haproxy timeout

    Change-Id: I1a4d5ae561a944a138512d7573fe16b5197050af
    Closes-Bug: #1675637

Changed in kolla-ansible:
status: In Progress → Fix Released
Changed in kolla-ansible:
importance: Undecided → High
milestone: none → pike-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/464224

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 5.0.0.0b2

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.openstack.org/475988

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/ocata)

Reviewed: https://review.openstack.org/464224
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=ccb7952ddb4bf2e6e008eb15341efb0ed5355bc3
Submitter: Jenkins
Branch: stable/ocata

commit ccb7952ddb4bf2e6e008eb15341efb0ed5355bc3
Author: jimmygc <email address hidden>
Date: Fri Mar 24 14:19:16 2017 +0800

    Fix glance Upload large image haproxy timeout

    Change-Id: I1a4d5ae561a944a138512d7573fe16b5197050af
    Closes-Bug: #1675637
    (cherry picked from commit 538a9ebf2d2cbfea2510e3d80fdbabae0be62b6f)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.openstack.org/475988
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=1daef3512ce1c22acb3665920c22ec8bdac76f4c
Submitter: Jenkins
Branch: master

commit 1daef3512ce1c22acb3665920c22ec8bdac76f4c
Author: liyingjun <email address hidden>
Date: Tue Jun 20 23:25:13 2017 -0400

    Fix glance Upload large image haproxy timeout when using external

    timeout options are added to glance-api but not for glance_api_external,
    it should be added there also.

    Change-Id: Idbd3dabbe24a5c71e70154dfde96c5c70eeefc10
    Partial-bug: 1675637

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 4.0.3

This issue was fixed in the openstack/kolla-ansible 4.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.