Gates failing with 504 gateway timeout

[Hugh Saunders]

Various gates have failed with 504s, latest example:

    17:13:06 ======================================================================
    17:13:06 FAIL: tempest.api.image.v1.test_image_members.ImageMembersTest.test_get_shared_images[gate]
    17:13:06 ----------------------------------------------------------------------
    17:13:06 _StringException: pythonlogging:'': {#{
    17:13:06 2014-12-12 11:09:38,068 2454 INFO [tempest.common.glance_http] Request: POST http://10.127.52.23:9292/v1/images
    17:13:06 2014-12-12 11:09:38,068 2454 INFO [tempest.common.glance_http] Request Headers: {'x-image-meta-container_format': 'bare', 'Transfer-Encoding': 'chunked', 'X-Auth-Token': '74577c21c2d646e3a47d5fd758d26b95', 'x-image-meta-is_public': 'False', 'User-Agent': 'tempest', 'Content-Type': 'application/octet-stream', 'x-image-meta-disk_format': 'raw', 'x-image-meta-name': 'ImageMembersTest-instance-230510866'}
    17:13:06 2014-12-12 11:09:38,069 2454 DEBUG [tempest.common.glance_http] Actual Path: /v1/images
    17:13:06 2014-12-12 11:10:28,073 2454 INFO [tempest.common.glance_http] Response Status: 504
    17:13:06 2014-12-12 11:10:28,074 2454 INFO [tempest.common.glance_http] Response Headers: [('connection', 'close'), ('content-type', 'text/html'), ('cache-control', 'no-cache')]
    17:13:06 #}}
    17:13:06
    17:13:06 Traceback (most recent call last):
    17:13:06 File "/opt/tempest_master/tempest/api/image/v1/test_image_members.py", line 35, in test_get_shared_images
    17:13:06 image = self._create_image()
    17:13:06 File "/opt/tempest_master/tempest/api/image/base.py", line 101, in _create_image
    17:13:06 data=image_file)
    17:13:06 File "/opt/tempest_master/tempest/api/image/base.py", line 71, in create_image
    17:13:06 disk_format, **kwargs)
    17:13:06 File "/opt/tempest_master/tempest/services/image/v1/json/image_client.py", line 155, in create_image
    17:13:06 return self._create_with_data(headers, kwargs.get('data'))
    17:13:06 File "/opt/tempest_master/tempest/services/image/v1/json/image_client.py", line 118, in _create_with_data
    17:13:06 body_iter)
    17:13:06 File "/opt/tempest_master/tempest/common/rest_client.py", line 550, in _error_checker
    17:13:06 raise exceptions.UnexpectedResponseCode(str(resp.status))
    17:13:06 UnexpectedResponseCode: Unexpected response code received
    17:13:06 Details: 504
    17:13:06
    17:13:06 -------------------- >> begin captured logging << --------------------
    17:13:06 tempest.common.glance_http: INFO: Request: POST http://10.127.52.23:9292/v1/images
    17:13:06 tempest.common.glance_http: INFO: Request Headers: {'x-image-meta-container_format': 'bare', 'Transfer-Encoding': 'chunked', 'X-Auth-Token': '74577c21c2d646e3a47d5fd758d26b95', 'x-image-meta-is_public': 'False', 'User-Agent': 'tempest', 'Content-Type': 'application/octet-stream', 'x-image-meta-disk_format': 'raw', 'x-image-meta-name': 'ImageMembersTest-instance-230510866'}
    17:13:06 tempest.common.glance_http: DEBUG: Actual Path: /v1/images
    17:13:06 tempest.common.glance_http: INFO: Response Status: 504
    17:13:06 tempest.common.glance_http: INFO: Response Headers: [('connection', 'close'), ('content-type', 'text/html'), ('cache-control', 'no-cache')]
    17:13:06 --------------------- >> end captured logging << ---------------------
    17:13:06
    17:13:06 ----------------------------------------------------------------------
    17:13:06 Ran 59 tests in 214.628s
    17:13:06
    17:13:06 FAILED (errors=1, failures=1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140605
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=eaf2e2a72b0bcfd85b9d2eb83d9084b8f91bc022
Submitter: Jenkins
Branch: master

commit eaf2e2a72b0bcfd85b9d2eb83d9084b8f91bc022
Author: Hugh Saunders <email address hidden>
Date: Wed Dec 10 08:09:44 2014 +0000

    Increase HAP server timeout

    Various gate jobs have failed with 504 timeouts. This patch makes the
    hap client and server timeouts configurable in the user config so that
    they can be raised in the gate.

    closes-bug: #1402580

    Change-Id: Ia9663eafd7934a76a1839a40a5e250eefdc2085e

[Hugh Saunders (hughsaunders)]

Previous update to 'increase' the timeout omitted the 's' suffix and actually decreased the timeout from 50 seconds to 90 milliseconds #fail

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/142153

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/142153
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=8c2f84bd8a9ea049cf05016790d96e4f8e4a8dbd
Submitter: Jenkins
Branch: master

commit 8c2f84bd8a9ea049cf05016790d96e4f8e4a8dbd
Author: Hugh Saunders <email address hidden>
Date: Tue Dec 16 16:42:18 2014 +0000

    Specify units in haproxy configuration

    Previous update omitted the 's' suffix so the value was interpreted as
    miliseconds. This patch corrects that.

    Change-Id: Ie7fa32a472105b2e7ba1074122b3abd8b4467f4c
    Closes-Bug: #1402580

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: juno
Review: https://review.openstack.org/151016

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: juno
Review: https://review.openstack.org/151202

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/151016
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=31dfe40020d18224709f72444cfaf4d2bbeab2fa
Submitter: Jenkins
Branch: juno

commit 31dfe40020d18224709f72444cfaf4d2bbeab2fa
Author: Hugh Saunders <email address hidden>
Date: Wed Dec 10 08:09:44 2014 +0000

    Increase HAP server timeout

    Various gate jobs have failed with 504 timeouts. This patch makes the
    hap client and server timeouts configurable in the user config so that
    they can be raised in the gate.

    closes-bug: #1402580

    Change-Id: Ia9663eafd7934a76a1839a40a5e250eefdc2085e
    (cherry picked from commit eaf2e2a72b0bcfd85b9d2eb83d9084b8f91bc022)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/151202
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=5c57886297d1596566d172621ad7e8e69e361d6a
Submitter: Jenkins
Branch: juno

commit 5c57886297d1596566d172621ad7e8e69e361d6a
Author: Hugh Saunders <email address hidden>
Date: Tue Dec 16 16:42:18 2014 +0000

    Specify units in haproxy configuration

    Previous update omitted the 's' suffix so the value was interpreted as
    miliseconds. This patch corrects that.

    Change-Id: Ie7fa32a472105b2e7ba1074122b3abd8b4467f4c
    Closes-Bug: #1402580
    (cherry picked from commit 8c2f84bd8a9ea049cf05016790d96e4f8e4a8dbd)