Too small timeout parameter in haproxy.cfg

Bug #1326082 reported by Evgeny Kozhemyakin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Dmitry Borodaenko
4.1.x
Fix Released
Medium
Dmitry Borodaenko
5.0.x
Fix Released
Medium
Dmitry Borodaenko

Bug Description

HA deploiment.
Some time haproxy may give "504 Gateway Timeout" error because of
too small "timeout server" parameter in /etc/haproxy/haproxy.cfg.
(After uploading a big image file into dashboard e.g.)
Now "timeout server" is 1 minute.

Revision history for this message
Mike Scherbakov (mihgen) wrote :

Relevant discussion in openstack-dev:
http://<email address hidden>/msg25859.html

Changed in fuel:
milestone: none → 5.1
Revision history for this message
Mike Scherbakov (mihgen) wrote :

According to what was said, image is still uploaded fine to Glance, and there is only reporting from Horizon that it timed out. I want this to be confirmed though.
If it is not the case, and upload actually fails, then it is rather High priority issue as it prevents users from being able to upload large images in Fuel HA mode.

Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Setting to Incomplete until we have a confirmation that only Horizon is affected, and Glance itself isn't.

Changed in fuel:
importance: Undecided → Medium
status: New → Incomplete
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/97645

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dmitry Borodaenko (dborodaenko)
status: Incomplete → In Progress
Changed in fuel:
status: In Progress → Incomplete
assignee: Dmitry Borodaenko (dborodaenko) → Evgeny Kozhemyakin (ekozhemyakin)
Revision history for this message
Evgeny Kozhemyakin (ekozhemyakin) wrote :

I've reproduced the issue a number of times and confirm Glance is not affected, images are still uploading with valid md5sums.

Changing "timeout server" to 20min allows uploading images up to 1.3G without the 504 error
(in my slow & overloaded virtual environment).

Changed in fuel:
status: Incomplete → Confirmed
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

There is a patch in review, can you please clarify does the patch fix the issue as well?

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Evgeny Kozhemyakin (ekozhemyakin) wrote :

Yes, the patch fixes the issue.
I've successfully uploaded 4G image.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/98877

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/4.1)

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/98878

Changed in fuel:
status: In Progress → Fix Committed
assignee: Evgeny Kozhemyakin (ekozhemyakin) → Dmitry Borodaenko (dborodaenko)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/97645
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=e794a1b92c8a5d814436b7dc730b0fb2c6d752b2
Submitter: Jenkins
Branch: master

commit e794a1b92c8a5d814436b7dc730b0fb2c6d752b2
Author: Dmitry Borodaenko <email address hidden>
Date: Tue Jun 3 16:11:30 2014 -0700

    Set HAProxy timeout to 3h for Horizon HTTP

    puppet-haproxy default timeout value of 1m is causing 504 Gateway Timeout
    errors when uploading large images via Horizon Dashboard.

    Change-Id: I07a666b17c20d39d4adcbcd1be69df42c2a1ddc2
    Closes-Bug: #1326082

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/4.1)

Reviewed: https://review.openstack.org/98878
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5642f9555985cc14e8600bd7983cd72343d90175
Submitter: Jenkins
Branch: stable/4.1

commit 5642f9555985cc14e8600bd7983cd72343d90175
Author: Dmitry Borodaenko <email address hidden>
Date: Tue Jun 3 16:11:30 2014 -0700

    Set HAProxy timeout to 3h for Horizon HTTP

    puppet-haproxy default timeout value of 1m is causing 504 Gateway Timeout
    errors when uploading large images via Horizon Dashboard.

    Change-Id: I07a666b17c20d39d4adcbcd1be69df42c2a1ddc2
    Closes-Bug: #1326082

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/98877
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3451a19a8fa8d7ae536ce8b27b32f8944ac87445
Submitter: Jenkins
Branch: stable/5.0

commit 3451a19a8fa8d7ae536ce8b27b32f8944ac87445
Author: Dmitry Borodaenko <email address hidden>
Date: Tue Jun 3 16:11:30 2014 -0700

    Set HAProxy timeout to 3h for Horizon HTTP

    puppet-haproxy default timeout value of 1m is causing 504 Gateway Timeout
    errors when uploading large images via Horizon Dashboard.

    Change-Id: I07a666b17c20d39d4adcbcd1be69df42c2a1ddc2
    Closes-Bug: #1326082

Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

{"build_id": "2014-07-03_00-41-14", "mirantis": "yes", "build_number": "382", "nailgun_sha": "bdc747cd70721874d1e8af36181fd4179cce4fce", "ostf_sha": "f4f15b4d98459650c1945b0efc30290a619be824", "fuelmain_sha": "ae77ae1c21a8715a63c7b1ca3f7a7ecadaff68aa", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1.1", "fuellib_sha": "2a5f7cb14825d619a632213c040477b77f0ad123"}

Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

Fix released:
{"build_id": "2014-07-21_00-31-14", "mirantis": "yes", "build_number": "142", "ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f", "nailgun_sha": "17444180b7e8c0c454488e63a05693881168f76a", "production": "docker", "api": "1.0", "fuelmain_sha": "312121d05c780e44dbf6db138847547b45bc3848", "astute_sha": "9a74b788be9a7c5682f1c52a892df36e4766ce3f", "release": "5.0.1", "fuellib_sha": "1a39cf33b217535cdbb6374da855085a8b420605"}

Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

Fix released:
{"build_id": "2014-07-21_10-32-30", "ostf_sha": "9863db951a6e159f4fa6e6861c8331e1af069cf8", "build_number": "340", "auth_required": false, "api": "1.0", "nailgun_sha": "bdd0bdec2b45eea843d559b7648bd5dca4873c66", "production": "docker", "fuelmain_sha": "539a5bf7493a5d14690a34bb18c3ad1c75b4f37f", "astute_sha": "fd9b8e3b6f59b2727b1b037054f10e0dd7bd37f1", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "1ec799bc6c8b08b8c9c6243c426507cb7a46459b"}

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.