tripleclient fails to retry when it gets a 504 from heat/haproxy

Bug #1833452 reported by Alex Schultz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Alex Schultz

Bug Description

While we wait for a stack to become ready, we may get a 504 from haproxy/heat indicating that the service is busy and the request timed out. We should retry rather than straight out failing because the action may eventually complete.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (master)

Fix proposed to branch: master
Review: https://review.opendev.org/666429

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (master)

Reviewed: https://review.opendev.org/666429
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=cb42cfe30fb0ca8f95ae3e7d93e616c106c6c191
Submitter: Zuul
Branch: master

commit cb42cfe30fb0ca8f95ae3e7d93e616c106c6c191
Author: Alex Schultz <email address hidden>
Date: Wed Jun 19 13:45:23 2019 -0600

    Retry heat stack polling on gateway problems

    If the heat api is overloaded or temporarily unavailable, we might get a
    503 or 504 from haproxy during the deployment. We should retry polling
    for events in this case as to not prematurely exit the deployment.

    Change-Id: I947cd0f9bf4a97e46c3d2bf3e9b986f7d38e9357
    Closes-Bug: #1833452

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/666753

Changed in tripleo:
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/stein)

Reviewed: https://review.opendev.org/666753
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=34bde896acd88f6a7daf5c90a128a231fef275b7
Submitter: Zuul
Branch: stable/stein

commit 34bde896acd88f6a7daf5c90a128a231fef275b7
Author: Alex Schultz <email address hidden>
Date: Wed Jun 19 13:45:23 2019 -0600

    Retry heat stack polling on gateway problems

    If the heat api is overloaded or temporarily unavailable, we might get a
    503 or 504 from haproxy during the deployment. We should retry polling
    for events in this case as to not prematurely exit the deployment.

    Change-Id: I947cd0f9bf4a97e46c3d2bf3e9b986f7d38e9357
    Closes-Bug: #1833452
    (cherry picked from commit cb42cfe30fb0ca8f95ae3e7d93e616c106c6c191)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/666916

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/666917

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/rocky)

Reviewed: https://review.opendev.org/666916
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=2e83e38be5daee54d3ccd206a3befe95581235f7
Submitter: Zuul
Branch: stable/rocky

commit 2e83e38be5daee54d3ccd206a3befe95581235f7
Author: Alex Schultz <email address hidden>
Date: Wed Jun 19 13:45:23 2019 -0600

    Retry heat stack polling on gateway problems

    If the heat api is overloaded or temporarily unavailable, we might get a
    503 or 504 from haproxy during the deployment. We should retry polling
    for events in this case as to not prematurely exit the deployment.

    Change-Id: I947cd0f9bf4a97e46c3d2bf3e9b986f7d38e9357
    Closes-Bug: #1833452
    (cherry picked from commit cb42cfe30fb0ca8f95ae3e7d93e616c106c6c191)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/queens)

Reviewed: https://review.opendev.org/666917
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=e717083e2af9d73badef959977ea7608467458eb
Submitter: Zuul
Branch: stable/queens

commit e717083e2af9d73badef959977ea7608467458eb
Author: Alex Schultz <email address hidden>
Date: Wed Jun 19 13:45:23 2019 -0600

    Retry heat stack polling on gateway problems

    If the heat api is overloaded or temporarily unavailable, we might get a
    503 or 504 from haproxy during the deployment. We should retry polling
    for events in this case as to not prematurely exit the deployment.

    Change-Id: I947cd0f9bf4a97e46c3d2bf3e9b986f7d38e9357
    Closes-Bug: #1833452
    (cherry picked from commit cb42cfe30fb0ca8f95ae3e7d93e616c106c6c191)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 11.5.0

This issue was fixed in the openstack/python-tripleoclient 11.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 12.1.0

This issue was fixed in the openstack/python-tripleoclient 12.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 10.7.1

This issue was fixed in the openstack/python-tripleoclient 10.7.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 9.3.1

This issue was fixed in the openstack/python-tripleoclient 9.3.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.