periodic fs020 rocky failing at containerized gnocchi

[Quique Llorente]

Periodic job for featureset020 is failing at overcloud deploy,
but stack is succesfully installed
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/5e5abd2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-19_04_12_44
2018-09-19 04:12:44 | Connection is already closed.
2018-09-19 04:12:44 | =============================================================
2018-09-19 04:12:44 |
2018-09-19 04:12:44 | Ansible passed.
2018-09-19 04:12:44 | Overcloud configuration completed.

Possible issue at python-tripleoclient

Possible fix https://review.openstack.org/#/c/603645/

[chandan kumar (chkumar246)]

If we go to the bottom of the above log 2018-09-19 03:12:38 | PLAY [External deployment Post Deploy tasks] ***********************************
2018-09-19 03:12:38 |
2018-09-19 03:12:38 | PLAY RECAP *********************************************************************
2018-09-19 03:12:38 | overcloud-controller-0 : ok=188 changed=91 unreachable=0 failed=0
2018-09-19 03:12:38 | overcloud-novacompute-0 : ok=138 changed=66 unreachable=0 failed=0
2018-09-19 03:12:38 | undercloud : ok=10 changed=7 unreachable=0 failed=0
2018-09-19 03:12:38 |
2018-09-19 03:12:38 | Wednesday 19 September 2018 03:12:37 +0000 (0:00:00.171) 0:18:57.310 ***
2018-09-19 03:12:44 | ==================Waiting for messages on queue 'tripleo' with no timeout.
2018-09-19 04:12:44 | Connection is already closed.
2018-09-19 04:12:44 | =============================================================

It appears the overcloud deployment success. but the output printed is false negative.

2018-09-19 04:12:44 | Overcloud configuration completed.
2018-09-19 04:12:45 | + status_code=1
2018-09-19 04:12:45 | + openstack stack list
2018-09-19 04:12:45 | + grep -q overcloud
2018-09-19 04:12:48 | + openstack stack list
2018-09-19 04:12:48 | + grep -Eq '(CREATE|UPDATE)_COMPLETE'
2018-09-19 04:12:50 | + openstack overcloud status
2018-09-19 04:12:50 | + grep -Eq DEPLOY_SUCCESS
2018-09-19 04:12:55 | + exit 1

The pieces that determines

[OpenStack Infra (hudson-openstack)]

Change abandoned by Quique Llorente (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/603645
Reason: Let's do this with git review -x

[OpenStack Infra (hudson-openstack)]

Change abandoned by Quique Llorente (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/603650
Reason: Let's do this with git review -x

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/603802

[OpenStack Infra (hudson-openstack)]

Change abandoned by Quique Llorente (<email address hidden>) on branch: master
Review: https://review.openstack.org/603802

[OpenStack Infra (hudson-openstack)]

Change abandoned by Quique Llorente (<email address hidden>) on branch: master
Review: https://review.openstack.org/603641

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/603804

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/603806

[Quique Llorente (quiquell)]

Another one
http://logs.openstack.org/45/560445/144/check/tripleo-ci-centos-7-containers-multinode-queens/dcb313c/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-09-19_23_11_16
2018-09-20 01:54:26 |
2018-09-20 01:54:26 | Introspection completed.
2018-09-20 01:54:26 | + openstack overcloud node provide --all-manageable
2018-09-20 01:54:29 | Waiting for messages on queue 'tripleo' with no timeout.
2018-09-20 03:11:51 | Connection is already closed.
2018-09-20 03:11:51 | [{}, {u'result': u'Node b8e50c7d-bd67-40c6-8e3a-69cadc16460e did not reach state "available", the state is "clean wait", error: None'}]

[OpenStack Infra (hudson-openstack)]

Change abandoned by Juan Antonio Osorio Robles (<email address hidden>) on branch: master
Review: https://review.openstack.org/603802
Reason: abandoning temporarily to free up resources

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603802
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=99b7a3d1d15813fd8d07aa36636feb44516dc523
Submitter: Zuul
Branch: master

commit 99b7a3d1d15813fd8d07aa36636feb44516dc523
Author: Quique Llorente <email address hidden>
Date: Wed Sep 19 13:00:54 2018 +0200

    Add a guard to break if no connection

    If the WebSocket connection is already disconnected we don't have to
    wait for messages

    Change-Id: I7543c8e61711f036140db20416a57447da6c4c18
    Closes-Bug: #1793264

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603804
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=d476a6ca714fca39b75a187500ab5eb393f17083
Submitter: Zuul
Branch: stable/rocky

commit d476a6ca714fca39b75a187500ab5eb393f17083
Author: Quique Llorente <email address hidden>
Date: Wed Sep 19 13:00:54 2018 +0200

    Add a guard to break if no connection

    If the WebSocket connection is already disconnected we don't have to
    wait for messages

    Change-Id: I7543c8e61711f036140db20416a57447da6c4c18
    Closes-Bug: #1793264
    (cherry picked from commit 99b7a3d1d15813fd8d07aa36636feb44516dc523)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 11.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603806
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=9290924a343028287d04caa222f1e1fe4e826b62
Submitter: Zuul
Branch: stable/queens

commit 9290924a343028287d04caa222f1e1fe4e826b62
Author: Quique Llorente <email address hidden>
Date: Wed Sep 19 13:00:54 2018 +0200

    Add a guard to break if no connection

    If the WebSocket connection is already disconnected we don't have to
    wait for messages

    Change-Id: I7543c8e61711f036140db20416a57447da6c4c18
    Closes-Bug: #1793264
    (cherry picked from commit 99b7a3d1d15813fd8d07aa36636feb44516dc523)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 10.6.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 9.2.7 release.