Multiple AIO SX upgrade_data files cause upgrade failures

[Bart Wensley]

Brief Description
-----------------
When an AIO simplex subcloud upgrade is done as part of a Distributed Cloud upgrade, multiple upgrade_data files found on the /opt/platform-backup filesystem cause the upgrade to fail. Two fixes are required:
- When an AIO simplex upgrade is aborted or completed, the upgrade_data files need to be deleted from the /opt/platform-backup filesystem.
- If an AIO simplex upgrade is started and there are multiple upgrade_data files in the /opt/platform-backup partition (e.g. because of a previous failure that didn't result in file deletion), the most recent file should be used.

Severity
--------
Major: Upgrade fails if the extra upgrade_data files are not deleted.

Steps to Reproduce
------------------
Perform multiple AIO simplex upgrades on the same host.

Expected Behavior
------------------
The upgrade should pass.

Actual Behavior
----------------
The upgrade fails.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Distributed Cloud with AIO simplex subcloud

Branch/Pull Time/Commit
-----------------------
starlingx master af of June 27, 2020

Last Pass
---------
Never

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Feature Testing

Workaround
----------
The extra upgrade_data files under /opt/platform-backup must be manually deleted before the upgrade is done.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/739125

[Ghada Khalil (gkhalil)]

stx.4.0 / high priority - multiple upgrades on the same simplex host will fail

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738297
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=766016fdf7667f14192de11322682b063fc9394d
Submitter: Zuul
Branch: master

commit 766016fdf7667f14192de11322682b063fc9394d
Author: David Sullivan <email address hidden>
Date: Fri Jun 26 22:06:22 2020 -0400

    Select the latest upgrade_data file during upgrade

    When the upgrade_data_file isn't specified we search for files starting
    with upgrade_data under /opt/platform-backup. With this change if
    multiple files are found we select the most recent file.

    Change-Id: I5e63cb1e76985522f764ea6e906f660c89838eb9
    Partial-Bug: 1886041
    Story: 2007403
    Task: 40236
    Signed-off-by: David Sullivan <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739125
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=a552b048cb1c64dd562c433cdbc43b0da79db504
Submitter: Zuul
Branch: master

commit a552b048cb1c64dd562c433cdbc43b0da79db504
Author: David Sullivan <email address hidden>
Date: Thu Jul 2 13:22:00 2020 -0400

    Multiple AIO SX upgrade_data files cause upgrade failures

    When an AIO simplex subcloud upgrade is done as part of a Distributed
    Cloud upgrade, multiple upgrade_data files found on the
    /opt/platform-backup filesystem cause the upgrade to fail.

    This change will remove the upgrade_data file on upgrade_complete (after
    the upgrade is successful). The upgrade_data is already removed when the
    upgrade is aborted.

    Change-Id: Ic9ba16e542045b0ce7b53b14ed7d2deb69cb58b9
    Closes-Bug: 1886041
    Signed-off-by: David Sullivan <email address hidden>

[Ghada Khalil (gkhalil)]

@David, since this LP is tagged for stx.4.0, please cherrypick the fix to the r/stx.4.0 branch asap. thx.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.4.0
Review: https://review.opendev.org/741456

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.4.0
Review: https://review.opendev.org/741458

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741456
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=4caad45d618a49d8b6444cc104b5637ce5eb9aae
Submitter: Zuul
Branch: r/stx.4.0

commit 4caad45d618a49d8b6444cc104b5637ce5eb9aae
Author: David Sullivan <email address hidden>
Date: Thu Jul 2 13:22:00 2020 -0400

    Multiple AIO SX upgrade_data files cause upgrade failures

    When an AIO simplex subcloud upgrade is done as part of a Distributed
    Cloud upgrade, multiple upgrade_data files found on the
    /opt/platform-backup filesystem cause the upgrade to fail.

    This change will remove the upgrade_data file on upgrade_complete (after
    the upgrade is successful). The upgrade_data is already removed when the
    upgrade is aborted.

    Change-Id: Ic9ba16e542045b0ce7b53b14ed7d2deb69cb58b9
    Closes-Bug: 1886041
    Signed-off-by: David Sullivan <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741458
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=341c1a2a07f8e7f3448212917e6a3f93d491d3e2
Submitter: Zuul
Branch: r/stx.4.0

commit 341c1a2a07f8e7f3448212917e6a3f93d491d3e2
Author: David Sullivan <email address hidden>
Date: Fri Jun 26 22:06:22 2020 -0400

    Select the latest upgrade_data file during upgrade

    When the upgrade_data_file isn't specified we search for files starting
    with upgrade_data under /opt/platform-backup. With this change if
    multiple files are found we select the most recent file.

    Change-Id: I5e63cb1e76985522f764ea6e906f660c89838eb9
    Partial-Bug: 1886041
    Story: 2007403
    Task: 40236
    Signed-off-by: David Sullivan <email address hidden>