tripleoclient report successful upgrade when ansible failed

[Emilien Macchi]

2020-01-10 13:51:46.062 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Generating default ansible config file /home/vagrant/ansible.cfg
2020-01-10 14:14:16.483 961513 ERROR tripleoclient.utils.utils [ ] Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2
2020-01-10 14:14:16.483 961513 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] Exception: Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2
2020-01-10 14:14:16.485 961513 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] None
2020-01-10 14:14:17.132 961513 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] ** Found ansible errors for undercloud deployment! **
2020-01-10 14:14:17.133 961513 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] [
 [
  "Enable and start systemd service for neutron_dhcp",
  {
   "invocation": {
    "module_args": {
     "no_block": false,
     "force": null,
     "name": "tripleo_neutron_dhcp.service",
     "daemon_reexec": false,
     "enabled": true,
     "daemon_reload": true,
     "state": "restarted",
     "user": null,
     "scope": null,
     "masked": null
    }
   },
   "msg": "Unable to start service tripleo_neutron_dhcp.service: Job for tripleo_neutron_dhcp.service failed because a configured resource limit was exceeded. See \"systemctl status tripleo_neutron_dhcp.service\" and \"journalctl -xe\" for details.\n",
   "changed": false,
   "_ansible_no_log": false
  }
 ]
]
2020-01-10 14:14:17.133 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning working directory /home/vagrant/tripleo-heat-installer-templates
2020-01-10 14:14:17.155 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning ansible directory /home/vagrant/undercloud-ansible-LHiho4
2020-01-10 14:14:17.175 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Install artifact is located at /home/vagrant/undercloud-install-20200110141416.tar.bzip2
2020-01-10 14:14:17.175 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ]
########################################################

Deployment successful!

########################################################

2020-01-10 14:14:17.175 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Writing the stack virtual update mark file /var/lib/tripleo-heat-installer/update_mark_undercloud
2020-01-10 14:14:17.242 961513 ERROR tripleoclient.utils.utils [ ] reset failed: reset: standard error: Inappropriate ioctl for device

2020-01-10 14:14:17.242 961513 WARNING tripleoclient.utils.utils [ ] Unable to reset command line. Try manually running "reset" if the command line is broken.
2020-01-10 14:14:17.324 961266 WARNING tripleoclient.v1.undercloud.UpgradeUndercloud [ ]
##########################################################

The Undercloud has been successfully upgraded.

Useful files:

Password file is at ~/undercloud-passwords.conf
The stackrc file is at ~/stackrc

Use these files to interact with OpenStack services, and
ensure they are secured.

##########################################################

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/701980

[Emilien Macchi (emilienm)]

Note that I've tested a regular install (with ansible failure) and tripleoclient returns a proper error. So this bug is only about upgrades.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Luke Short (<email address hidden>) on branch: master
Review: https://review.opendev.org/701980
Reason: The code never lies.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/702161

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702161
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=3ccd4f023f30f781d6539b134d3869010c48cfba
Submitter: Zuul
Branch: master

commit 3ccd4f023f30f781d6539b134d3869010c48cfba
Author: Rabi Mishra <email address hidden>
Date: Mon Jan 13 08:03:48 2020 +0530

    Handle ansible_runner return code properly

    returning and raising exceptions are mutually exclusive. we need to
    return rc properly from run_ansible_playbook() for it to be used
    later in tripleo deploy.

    Change-Id: Ia07433fb6886931530afebad49c8b6bf1f062af5
    Closes-Bug: #1859182

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 13.1.0 release.

[Emilien Macchi (emilienm)]

I still see that bug today:

None: AnsibleRunnerException: Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2

2020-02-20 01:38:44 | TASK [tripleo_container_manage : Enable and start systemd service for memcached] ***
2020-02-20 01:38:44 | Thursday 20 February 2020 01:38:44 +0000 (0:00:01.242) 0:13:40.804 *****
2020-02-20 01:38:45 | fatal: [standalone]: FAILED! => changed=false
2020-02-20 01:38:45 | msg: |-
2020-02-20 01:38:45 | Unable to start service tripleo_memcached.service: Failed to start tripleo_memcached.service: Unit not found.
2020-02-20 01:38:45 |
2020-02-20 01:38:45 | NO MORE HOSTS LEFT *************************************************************
2020-02-20 01:38:45 |
2020-02-20 01:38:45 | PLAY RECAP *********************************************************************
2020-02-20 01:38:45 | standalone : ok=376 changed=87 unreachable=0 failed=1 skipped=165 rescued=0 ignored=1
2020-02-20 01:38:45 | undercloud : ok=14 changed=8 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0
2020-02-20 01:38:45 | Thursday 20 February 2020 01:38:45 +0000 (0:00:01.035) 0:13:41.839 *****
2020-02-20 01:38:45 | ===============================================================================
2020-02-20 01:38:45 | tripleo_container_manage : Check podman create status ----------------- 107.04s
2020-02-20 01:38:45 | tripleo_container_manage : Check podman create status ------------------ 70.20s
2020-02-20 01:38:45 | tripleo_container_manage : Wait for containers to be exited ------------ 56.48s
2020-02-20 01:38:45 | tripleo_container_manage : Wait for containers to be exited ------------ 54.52s
2020-02-20 01:38:45 | tripleo_container_manage : Wait for containers to be exited ------------ 45.58s
2020-02-20 01:38:45 | Write kolla config json files ------------------------------------------ 21.80s
2020-02-20 01:38:45 | tripleo_firewall : Manage firewall rules ------------------------------- 20.25s
2020-02-20 01:38:45 | tripleo_container_manage : Check podman create status ------------------ 17.53s
2020-02-20 01:38:45 | Creating container startup configs for step_4 -------------------------- 17.40s
2020-02-20 01:38:45 | Wait for puppet host configuration to finish --------------------------- 16.77s
2020-02-20 01:38:45 | Creating container startup configs for step_3 -------------------------- 10.54s
2020-02-20 01:38:45 | Run NetworkConfig script ----------------------------------------------- 10.53s
2020-02-20 01:38:45 | tripleo_container_manage : Gather podman infos -------------------------- 6.79s
2020-02-20 01:38:45 | tripleo_container_manage : Gather podman infos -------------------------- 6.35s
2020-02-20 01:38:45 | Creating container startup configs for step_2 --------------------------- 6.34s
2020-02-20 01:38:45 | tripleo_container_manage : Check podman create status ------------------- 6.27s
2020-02-20 01:38:45 | tripleo_container_manage : Gather podman infos -------------------------- 5.90s
2020-02-20 01:38:45 | Write container config scripts ------------------------------------------ 5.41s
2020-02-20 01:38:45 ...

[Emilien Macchi (emilienm)]

Link: https://0793b39e56ef753a822c-5061d1dd61c75bd0e8246a27130264c7.ssl.cf1.rackcdn.com/700738/37/check/tripleo-ci-centos-7-standalone-upgrade/a8e5cd8/logs/undercloud/home/zuul/standalone_upgrade.log

[Rabi Mishra (rabi)]

> I still see that bug today:

After the fix, it was again broken by https://review.opendev.org/#/c/704390/3/tripleoclient/utils.py

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/708942

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708942
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=9902d7469ad53bef4fdcb91793fb3b9d1d443a3d
Submitter: Zuul
Branch: master

commit 9902d7469ad53bef4fdcb91793fb3b9d1d443a3d
Author: Kevin Carter <email address hidden>
Date: Thu Feb 20 12:13:34 2020 -0600

    Add handler for exception raising

    This change adds a switch that will enable or disable raising an
    exception when a playbook executes. This will allow some methods to
    return the RC and status information when a playbook is run, even when
    there's a failure. The default behaviour is to raise an exception on
    failure, but when fail_on_rc is set to False the run_ansible_playbook
    method will return the status and rc information regardless of any
    failures.

    To ensure we're not raising an exception from the ansible runner library
    its been changed to RuntimeError.

    Change-Id: I3af652615b5227144256074c05170d148f19bc1d
    Closes-Bug: #1859182
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 13.2.0 release.