tripleoclient doesn't retry when it gets 500 error from heat-api

Bug #1855633 reported by Keigo Noha
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Undecided
Alex Schultz

Bug Description

Description
===========
tripleoclient doesn't retry when it gets 500 error from heat-api.
In very rare situation, like restarting httpd service by logrotate in undercloud causes that heat-api returns 500 code then tripleclient can't get the progress of the deployment.

Steps to reproduce
==================
1. Install undercloud.
2. Deploy overcloud and wait for the start of workflow.
3. Run 'systemctl reload httpd.service' in several times.
4. You will get 500 error from heat-api.

Expected result
===============
tripleclient retries the connection to heat-api and the client shows the progress of the deployment.

Actual result
=============
tripleoclient didn't retry the connection to heat-api then the command can't get the progress of the deployment.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (master)

Fix proposed to branch: master
Review: https://review.opendev.org/697939

Changed in tripleo:
assignee: nobody → Keigo Noha (knoha)
status: New → In Progress
Changed in tripleo:
assignee: Keigo Noha (knoha) → Alex Schultz (alex-schultz)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (master)

Reviewed: https://review.opendev.org/697939
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=932fc6755a8315ccde2e90848ad3224f1fe16735
Submitter: Zuul
Branch: master

commit 932fc6755a8315ccde2e90848ad3224f1fe16735
Author: Keigo Noha <email address hidden>
Date: Mon Dec 9 11:09:25 2019 +0900

    Retry the connection to heat-api when 500 code

    In a rare case like httpd reload by logrotate, heat-api returns
    500 code. If this happens, trpleoclient can't get the status of
    the stack even though the process is still on-going.

    To handle this situation, tipleoclient should retry when it can't
    get the stack information by 500 code.

    Change-Id: I97a6825f4ff9f125eb597e5b7bd0c553c37e49e7
    Closes-Bug: #1855633

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/698900

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/698901

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/698902

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/698903

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/stein)

Reviewed: https://review.opendev.org/698901
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=06e31347cf4f87adb985e79bbe8a8227ba4cbf3e
Submitter: Zuul
Branch: stable/stein

commit 06e31347cf4f87adb985e79bbe8a8227ba4cbf3e
Author: Keigo Noha <email address hidden>
Date: Mon Dec 9 11:09:25 2019 +0900

    Retry the connection to heat-api when 500 code

    In a rare case like httpd reload by logrotate, heat-api returns
    500 code. If this happens, trpleoclient can't get the status of
    the stack even though the process is still on-going.

    To handle this situation, tipleoclient should retry when it can't
    get the stack information by 500 code.

    Change-Id: I97a6825f4ff9f125eb597e5b7bd0c553c37e49e7
    Closes-Bug: #1855633
    (cherry picked from commit 932fc6755a8315ccde2e90848ad3224f1fe16735)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/train)

Reviewed: https://review.opendev.org/698900
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=40bccd51d87eddad89ab59b39e4dec0710b71608
Submitter: Zuul
Branch: stable/train

commit 40bccd51d87eddad89ab59b39e4dec0710b71608
Author: Keigo Noha <email address hidden>
Date: Mon Dec 9 11:09:25 2019 +0900

    Retry the connection to heat-api when 500 code

    In a rare case like httpd reload by logrotate, heat-api returns
    500 code. If this happens, trpleoclient can't get the status of
    the stack even though the process is still on-going.

    To handle this situation, tipleoclient should retry when it can't
    get the stack information by 500 code.

    Change-Id: I97a6825f4ff9f125eb597e5b7bd0c553c37e49e7
    Closes-Bug: #1855633
    (cherry picked from commit 932fc6755a8315ccde2e90848ad3224f1fe16735)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/queens)

Reviewed: https://review.opendev.org/698903
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=70b28932fda9135cb340d500d1dcf3ed2bdfc0b5
Submitter: Zuul
Branch: stable/queens

commit 70b28932fda9135cb340d500d1dcf3ed2bdfc0b5
Author: Keigo Noha <email address hidden>
Date: Mon Dec 9 11:09:25 2019 +0900

    Retry the connection to heat-api when 500 code

    In a rare case like httpd reload by logrotate, heat-api returns
    500 code. If this happens, trpleoclient can't get the status of
    the stack even though the process is still on-going.

    To handle this situation, tipleoclient should retry when it can't
    get the stack information by 500 code.

    Change-Id: I97a6825f4ff9f125eb597e5b7bd0c553c37e49e7
    Closes-Bug: #1855633
    (cherry picked from commit 932fc6755a8315ccde2e90848ad3224f1fe16735)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/rocky)

Reviewed: https://review.opendev.org/698902
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=bcb511376a6f5208d66bc5e92e7df3f3319c4ca3
Submitter: Zuul
Branch: stable/rocky

commit bcb511376a6f5208d66bc5e92e7df3f3319c4ca3
Author: Keigo Noha <email address hidden>
Date: Mon Dec 9 11:09:25 2019 +0900

    Retry the connection to heat-api when 500 code

    In a rare case like httpd reload by logrotate, heat-api returns
    500 code. If this happens, trpleoclient can't get the status of
    the stack even though the process is still on-going.

    To handle this situation, tipleoclient should retry when it can't
    get the stack information by 500 code.

    Change-Id: I97a6825f4ff9f125eb597e5b7bd0c553c37e49e7
    Closes-Bug: #1855633
    (cherry picked from commit 932fc6755a8315ccde2e90848ad3224f1fe16735)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 12.3.1

This issue was fixed in the openstack/python-tripleoclient 12.3.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 13.1.0

This issue was fixed in the openstack/python-tripleoclient 13.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient rocky-eol

This issue was fixed in the openstack/python-tripleoclient rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient queens-eol

This issue was fixed in the openstack/python-tripleoclient queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient stein-eol

This issue was fixed in the openstack/python-tripleoclient stein-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.