OpenStack-Ansible

When run-upgrade.sh fails, the list of restart commands are wrong

  1. Series liberty
  2. Bug #1582272
Bug #1582272 reported by Bjoern
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
High
Jesse Pretorius
Liberty
Fix Released
Critical
git-harry
Mitaka
Fix Released
Critical
Jean-Philippe Evrard

Bug Description

During a kilo to Liberty upgrade I noticed inconsistent output of the playbooks to run.
My upgrade got stuck at repo_build (unrelated issue here) and the remaining task list contains lines with no playbooks (note lines 5-8):

ran setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true'
******************** failure ********************
The upgrade script has encountered a failure.
Failed on task setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true'
Re-run the run-upgrade.sh script, or
execute the remaining tasks manually:
openstack-ansible -e 'pip_install_options=--force-reinstall'
openstack-ansible -e 'pip_install_options=--force-reinstall'
openstack-ansible -e 'pip_install_options=--force-reinstall'
openstack-ansible -e 'pip_install_options=--force-reinstall'
openstack-ansible setup-hosts.yml --limit '!galera_all[0]' -e 'pip_install_options=--force-reinstall'
openstack-ansible lxc-containers-create.yml --limit galera_all[0] -e 'pip_install_options=--force-reinstall'
openstack-ansible -e 'pip_install_options=--force-reinstall'
openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e 'pip_install_options=--force-reinstall'
openstack-ansible /opt/rpc-openstack/openstack-ansible/scripts/upgrade-utilities/playbooks/disable-neutron-port-security.yml -e 'pip_install_options=--force-reinstall'
openstack-ansible /opt/rpc-openstack/openstack-ansible/scripts/upgrade-utilities/playbooks/memcached-flush.yml -e 'pip_install_options=--force-reinstall'
openstack-ansible setup-openstack.yml -e 'pip_install_options=--force-reinstall'
openstack-ansible /opt/rpc-openstack/openstack-ansible/scripts/upgrade-utilities/playbooks/cleanup-rabbitmq-vhost.yml -e 'pip_install_options=--force-reinstall'
******************** failure ********************

Also I saw in the logs that the previous, successful ran plays, got marked as OK, but still have been included in the output as plays to run:

setup-hosts.yml --limit '!galera_all[0]' has been marked as success
lxc-containers-create.yml --limit galera_all[0] has been marked as success

Perhaps the -e 'pip_install_options=--force-reinstall' threw off the logic in the run-upgrade script

See original description
Jean-Philippe Evrard (jean-philippe-evrard)
Changed in openstack-ansible:
status: New → Incomplete
importance: Undecided → Low
Revision history for this message
Bjoern (bjoern-t) wrote :

There has been no question been provided so I do not know what information is considered missing.

Changed in openstack-ansible:
status: Incomplete → New
Ian Cordasco (icordasc)
Changed in openstack-ansible:
assignee: nobody → Xin (zxuuzx)
Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

I've edited the description to make the bug clearer.

description: updated
summary: - Restart plays not correctly defined when run-upgrade.sh fails
+ When run-upgrade.sh fails, the list of restart commands are wrong
Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

This needs an independent verification that this is an issue, and under what conditions it can be repeated with the same result every time.

Revision history for this message
Ian Cordasco (icordasc) wrote :

I have verified this personally.

Changed in openstack-ansible:
status: New → Triaged
importance: Low → High
Revision history for this message
Darren Birkett (darren-birkett) wrote :

Here is an example, after a failure in the upgrade script at the point of cleaning mariadb repos from containers:

12:27:27 ran /jenkins/workspace/RPC-AIO/openstack-ansible/scripts/upgrade-utilities/playbooks/mariadb-apt-cleanup.yml
12:27:27 ******************** failure ********************
12:27:27 The upgrade script has encountered a failure.
12:27:27 Failed on task /jenkins/workspace/RPC-AIO/openstack-ansible/scripts/upgrade-utilities/playbooks/mariadb-apt-cleanup.yml
12:27:27 Re-run the run-upgrade.sh script, or
12:27:27 execute the remaining tasks manually:
12:27:27 openstack-ansible -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible /jenkins/workspace/RPC-AIO/openstack-ansible/scripts/upgrade-utilities/playbooks/mariadb-apt-cleanup.yml -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible setup-hosts.yml --limit '!galera_all[0]' -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible lxc-containers-create.yml --limit galera_all[0]:repo_all -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible /jenkins/workspace/RPC-AIO/openstack-ansible/scripts/upgrade-utilities/playbooks/repo-server-pip-conf-removal.yml -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible /jenkins/workspace/RPC-AIO/openstack-ansible/scripts/upgrade-utilities/playbooks/disable-neutron-port-security.yml -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible /jenkins/workspace/RPC-AIO/openstack-ansible/scripts/upgrade-utilities/playbooks/memcached-flush.yml -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible setup-openstack.yml -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible /jenkins/workspace/RPC-AIO/openstack-ansible/scripts/upgrade-utilities/playbooks/cleanup-rabbitmq-vhost.yml -e 'pip_install_options=--force-reinstall'
12:27:27 openstack-ansible /jenkins/workspace/RPC-AIO/openstack-ansible/scripts/upgrade-utilities/playbooks/glance-db-storage-url-fix.yml -e 'pip_install_options=--force-reinstall'
12:27:27 ******************** failure ********************

Ian Cordasco (icordasc)
Changed in openstack-ansible:
status: Triaged → New
importance: High → Undecided
status: New → Triaged
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (master)

Fix proposed to branch: master
Review: https://review.openstack.org/352488

Changed in openstack-ansible:
status: Triaged → In Progress
OpenStack Infra (hudson-openstack)
Changed in openstack-ansible:
assignee: Xin (zxuuzx) → Miguel Alejandro Cantu (miguel-cantu)
OpenStack Infra (hudson-openstack)
Changed in openstack-ansible:
assignee: Miguel Alejandro Cantu (miguel-cantu) → Jesse Pretorius (jesse-pretorius)
OpenStack Infra (hudson-openstack)
Changed in openstack-ansible:
assignee: Jesse Pretorius (jesse-pretorius) → Kevin Carter (kevin-carter)
OpenStack Infra (hudson-openstack)
Changed in openstack-ansible:
assignee: Kevin Carter (kevin-carter) → Jesse Pretorius (jesse-pretorius)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (master)

Reviewed: https://review.openstack.org/352488
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=619b40c35102964e1a861dd473e7635d2eb7b642
Submitter: Jenkins
Branch: master

commit 619b40c35102964e1a861dd473e7635d2eb7b642
Author: Frank Zhang <email address hidden>
Date: Wed Aug 3 21:44:18 2016 +0000

    Print remaining tasks on failed upgrade correctly

    run-upgrade.sh is designated to re-run from the place
    where it failed. Previous behavior is to print entire tasks list
    when failures happen. Printing tasks that successfully
    completed is not desired. The new update will only show leftover
    tasks for the users debugging.

    Change-Id: I2e33c43d55696bb91cb40ff590e167546f5f6d18
    Closes-Bug:1582272

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/358641

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (stable/mitaka)

Reviewed: https://review.openstack.org/358641
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=00901b047190f92035db8000a226852a7614080e
Submitter: Jenkins
Branch: stable/mitaka

commit 00901b047190f92035db8000a226852a7614080e
Author: Frank Zhang <email address hidden>
Date: Wed Aug 3 21:44:18 2016 +0000

    Print remaining tasks on failed upgrade correctly

    run-upgrade.sh is designated to re-run from the place
    where it failed. Previous behavior is to print entire tasks list
    when failures happen. Printing tasks that successfully
    completed is not desired. The new update will only show leftover
    tasks for the users debugging.

    Change-Id: I2e33c43d55696bb91cb40ff590e167546f5f6d18
    Closes-Bug:1582272
    (cherry picked from commit 619b40c35102964e1a861dd473e7635d2eb7b642)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (liberty)

Fix proposed to branch: liberty
Review: https://review.openstack.org/360385

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (liberty)

Reviewed: https://review.openstack.org/360385
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=43c3baa1098a356179a944814b0cde6055b0a555
Submitter: Jenkins
Branch: liberty

commit 43c3baa1098a356179a944814b0cde6055b0a555
Author: Frank Zhang <email address hidden>
Date: Wed Aug 3 21:44:18 2016 +0000

    Print remaining tasks on failed upgrade correctly

    run-upgrade.sh is designated to re-run from the place
    where it failed. Previous behavior is to print entire tasks list
    when failures happen. Printing tasks that successfully
    completed is not desired. The new update will only show leftover
    tasks for the users debugging.

    Change-Id: I2e33c43d55696bb91cb40ff590e167546f5f6d18
    Closes-Bug:1582272
    (cherry picked from commit 619b40c35102964e1a861dd473e7635d2eb7b642)

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/openstack-ansible 13.3.2

This issue was fixed in the openstack/openstack-ansible 13.3.2 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/openstack-ansible 12.2.2

This issue was fixed in the openstack/openstack-ansible 12.2.2 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/openstack-ansible 14.0.0.0b3

This issue was fixed in the openstack/openstack-ansible 14.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.