tripleo

upgrade of controller fails w/ ansible syntax error, gather_facts_undercloud

Bug #1861335 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
wes hayutin
tripleo ussuri-2 "tripleo ussuri-2"

Bug Description

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1f4/704683/2/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/1f4f704/logs/undercloud/home/zuul/overcloud_update_run_Controller.log

2020-01-29 19:40:05 | 2020-01-29 19:40:05.317 170091 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:201
2020-01-29 19:40:09 | 2020-01-29 19:40:09.837 170091 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.
2020-01-29 19:40:15 | ansible-playbook-2 2.8.6
2020-01-29 19:40:15 | config file = /var/lib/mistral/f121e350-72df-427b-8628-7a1ba1824508/ansible.cfg
2020-01-29 19:40:15 | configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/tripleo-plugins/modules', u'/usr/share/ansible/plugins/modules', u'/usr/share/openstack-tripleo-validations/library']
2020-01-29 19:40:15 | ansible python module location = /usr/lib/python2.7/site-packages/ansible
2020-01-29 19:40:15 | executable location = /bin/ansible-playbook-2
2020-01-29 19:40:15 | python version = 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
2020-01-29 19:40:15 | Using /var/lib/mistral/f121e350-72df-427b-8628-7a1ba1824508/ansible.cfg as config file
2020-01-29 19:40:15 | ERROR! Syntax Error while loading YAML.
2020-01-29 19:40:15 | found undefined alias
2020-01-29 19:40:15 |
2020-01-29 19:40:15 | The error appears to be in '/var/lib/mistral/f121e350-72df-427b-8628-7a1ba1824508/update_steps_playbook.yaml': line 1, column 3, but may
2020-01-29 19:40:15 | be elsewhere in the file depending on the exact syntax problem.
2020-01-29 19:40:15 |
2020-01-29 19:40:15 | The offending line appears to be:
2020-01-29 19:40:15 |
2020-01-29 19:40:15 |
2020-01-29 19:40:15 | - *gather_facts_undercloud
2020-01-29 19:40:15 | ^ here
2020-01-29 19:40:15 |
2020-01-29 19:40:15 | Ansible failed, check log at /var/lib/mistral/f121e350-72df-427b-8628-7a1ba1824508/ansible.log.
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun [-] Exception occured while running the command: RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/f121e350-72df-427b-8628-7a1ba1824508/ansible.log.
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun Traceback (most recent call last):
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 32, in run
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun super(Command, self).run(parsed_args)
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun return super(Command, self).run(parsed_args)
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python2.7/site-packages/cliff/command.py", line 187, in run
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun return_code = self.take_action(parsed_args) or 0
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_update.py", line 171, in take_action
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun priv_key=key)
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 1444, in run_update_ansible_action
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun verbosity=verbosity, extra_vars=extra_vars)
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 127, in update_ansible
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun raise RuntimeError('Update failed with: {}'.format(payload['message']))
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/f121e350-72df-427b-8628-7a1ba1824508/ansible.log.
2020-01-29 19:40:15 | 2020-01-29 19:40:15.116 170091 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun 
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack [-] Update failed with: Ansible failed, check log at /var/lib/mistral/f121e350-72df-427b-8628-7a1ba1824508/ansible.log.: RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/f121e350-72df-427b-8628-7a1ba1824508/ansible.log.
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack Traceback (most recent call last):
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack result = cmd.run(parsed_args)
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 32, in run
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack super(Command, self).run(parsed_args)
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack return super(Command, self).run(parsed_args)
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack File "/usr/lib/python2.7/site-packages/cliff/command.py", line 187, in run
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack return_code = self.take_action(parsed_args) or 0
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_update.py", line 171, in take_action
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack priv_key=key)
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 1444, in run_update_ansible_action
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack verbosity=verbosity, extra_vars=extra_vars)
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/package_update.py", line 127, in update_ansible
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack raise RuntimeError('Update failed with: {}'.format(payload['message']))
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/f121e350-72df-427b-8628-7a1ba1824508/ansible.log.
2020-01-29 19:40:15 | 2020-01-29 19:40:15.121 170091 ERROR openstack 

Tags: alert upgrade
Revision history for this message
wes hayutin (weshayutin) wrote :

failed in the gate.. please have a look
https://review.opendev.org/#/c/704683/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/704885

Changed in tripleo:
assignee: nobody → Alex Schultz (alex-schultz)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/704885
Reason: clearing currently running jobs.

Changed in tripleo:
assignee: Alex Schultz (alex-schultz) → wes hayutin (weshayutin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Chandan Kumar (raukadah) (<email address hidden>) on branch: master
Review: https://review.opendev.org/704885
Reason: Since scenaro10 failed in gate looks tempest random failure will look seperatly

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/704885
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1f98e96d3533121db7e57e43b8e69db73337e074
Submitter: Zuul
Branch: master

commit 1f98e96d3533121db7e57e43b8e69db73337e074
Author: Alex Schultz <email address hidden>
Date: Wed Jan 29 21:04:51 2020 +0000

    Revert "Use YAML anchors/aliases to reduce playbook task repetition"

    This reverts commit bc4df9c5a994183c71347691e9eb3633e1c00951.

    These don't always exist in the the same file because of
    the usage of template. Some of these work but the updates
    is broken now.

    Change-Id: Id45eee817aff0a780f0fc2da2453bb8fec66d380
    Closes-Bug: #1861335

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 12.1.0

This issue was fixed in the openstack/tripleo-heat-templates 12.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.