When doing upgrade recoverable checks are leaving the cluster in a unknown state

[Sofer Athlan-Guyot]

Hi,

when doing the upgrade, numerous static checks are done during the major pacemaker upgrade step. They arrive in the script at various moments, like the check on rpm-python package, the disk size left of the bootstrap node and so on.

All those checks, if they fail, leave the cluster in more or less an unknown state. One has to go to the controller check what happen and put the cluster back into shape, fix the detected error and then maybe be able to upgrade again.

This is less than optimal situation.

A better way would be for all those tests to happen at the beginning of the upgrade. Then the operator would only have to fix the detected issue and re-run the upgrade again.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

After re-reading the code again, they all happen before any serious change. But it would be nice to refactor them to make it obvious.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/357750

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/357750
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=575e42b0287e37d3ef261c040fb3d331d3419801
Submitter: Jenkins
Branch: master

commit 575e42b0287e37d3ef261c040fb3d331d3419801
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Aug 25 11:58:56 2016 +0200

    Refactor upgrade checks.

    We make it clear that recoverable checks happen before starting the
    upgrade to be able to run the upgrade after the offending error has been
    manually corrected.

    Add new check for the pcsd cluster status.

    Add new check for galera password file: BZ 1357112

    Closes-Bug: 1614907
    Change-Id: If736c79121e1ffe0eaeb814bdb73ccbc0b64edcd