Undercloud deployment reports false successfully deployment but deployment was already failed with RuntimeError: wait_api_port_ready: Max retries 30 reached

Bug #1889394 reported by Sandeep Yadav
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Rabi Mishra

Bug Description

In below update job, Undercloud deploy was already failed with already failed with "RuntimeError: wait_api_port_ready: Max retries 30 reached" but still undercloud reports successfull deployment.

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d7b/743529/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/d7b78ea/job-output.txt

~~~
2020-07-29 07:10:21.281908 | primary | TASK [undercloud-deploy : Copy stackrc created at the UC user $HOME to the working dir] ***
2020-07-29 07:10:21.282822 | primary | Wednesday 29 July 2020 07:10:21 +0000 (0:00:00.393) 0:43:17.054 ********
2020-07-29 07:10:24.978150 | primary | fatal: [undercloud]: FAILED! => {
2020-07-29 07:10:24.978334 | primary | "changed": false
2020-07-29 07:10:24.978441 | primary | }
2020-07-29 07:10:24.978618 | primary |
~~~

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d7b/743529/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/d7b78ea/logs/undercloud/home/zuul/undercloud_install.log

~~~
Exception: wait_api_port_ready: Max retries 30 reached
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1240, in _standalone_deploy
    utils.wait_api_port_ready(parsed_args.heat_api_port)
  File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1743, in wait_api_port_ready
    "wait_api_port_ready: Max retries {} reached".format(max_retries))
RuntimeError: wait_api_port_ready: Max retries 30 reached
None
Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
Not cleaning ansible directory /home/zuul/undercloud-ansible-2jni0ee_
Install artifact is located at /home/zuul/undercloud-install-20200729070959.tar.bzip2

########################################################

Deployment successful!

########################################################

Writing the stack virtual update mark file /var/lib/tripleo-heat-installer/update_mark_undercloud

##########################################################
~~~

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (master)

Fix proposed to branch: master
Review: https://review.opendev.org/743744

Changed in tripleo:
assignee: nobody → Emilien Macchi (emilienm)
status: Triaged → In Progress
tags: added: queens-backport-potential train-backport-potential ussuri-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/743866

Changed in tripleo:
assignee: Emilien Macchi (emilienm) → Rabi Mishra (rabi)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on python-tripleoclient (master)

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.opendev.org/743744

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (master)

Reviewed: https://review.opendev.org/743866
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=a29dd8ef79dcf9fbde5dae1b98f0e88694b14b7a
Submitter: Zuul
Branch: master

commit a29dd8ef79dcf9fbde5dae1b98f0e88694b14b7a
Author: Rabi Mishra <email address hidden>
Date: Wed Jul 29 18:11:57 2020 +0530

    Simplify error handling in tripleo_deploy

    Don't use return code from ansible_runner and manage with a flag.
    This also removes the fail_on_rc flag from run_ansible_playbook()
    and makes it consistent to raise RuntimeError if rc !=0
    everywhere including _standalone_deploy().

    Change-Id: Ia5971af601d1d9500f33045768b38ac7937117f5
    Closes-Bug: #1889394

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/744422

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/ussuri)

Reviewed: https://review.opendev.org/744422
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=e352506699f8f52f1ce8cf05862f9ba9c2520a0c
Submitter: Zuul
Branch: stable/ussuri

commit e352506699f8f52f1ce8cf05862f9ba9c2520a0c
Author: Rabi Mishra <email address hidden>
Date: Wed Jul 29 18:11:57 2020 +0530

    Simplify error handling in tripleo_deploy

    Don't use return code from ansible_runner and manage with a flag.
    This also removes the fail_on_rc flag from run_ansible_playbook()
    and makes it consistent to raise RuntimeError if rc !=0
    everywhere including _standalone_deploy().

    Conflicts:
            tripleoclient/tests/v1/tripleo/test_tripleo_deploy.py
            tripleoclient/utils.py
            tripleoclient/v1/tripleo_deploy.py

    Change-Id: Ia5971af601d1d9500f33045768b38ac7937117f5
    Closes-Bug: #1889394
    (cherry picked from commit a29dd8ef79dcf9fbde5dae1b98f0e88694b14b7a)

tags: added: in-stable-ussuri
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.