Backup & Restore fails to restore on Standard Storage lab

[Kristine Bujold]

Brief Description
-----------------
Perform a backup:
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/backup.yml -e "ansible_become_pass=<password> admin_password=<password>*"

Re-install the lab and perform the restore:
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_platform.yml -e "backup_filename=pv1_localhost_platform_backup_2019_11_11_18_27_43.tgz admin_password=<password> ansible_become_pass=<password> initial_backup_dir=/home/sysadmin"

The restore fails with error

TASK [restore-platform/restore-more-data : Set ceph crushmap directory to /etc/sysinv if it is AIO system] *******************************************************************************

TASK [restore-platform/restore-more-data : Set ceph crushmap directory to /opt/platform/sysinv if it is non-AIO system] ******************************************************************
ok: [localhost]

TASK [restore-platform/restore-more-data : Restore ceph crush map] ***********************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["tar", "-C", "/opt/platform/sysinv/19.09", "-xpf", "/home/sysadmin/localhost_platform_backup_2019_11_08_20_40_27.tgz", "--transform=s,.*/,,", "*/crushmap.bin.backup"], "delta": "0:00:09.890870", "end": "2019-11-11 15:18:39.219260", "msg": "non-zero return code", "rc": 2, "start": "2019-11-11 15:18:29.328390", "stderr": "tar: /opt/platform/sysinv/19.09: Cannot open: No such file or directory\ntar: Error is not recoverable: exiting now", "stderr_lines": ["tar: /opt/platform/sysinv/19.09: Cannot open: No such file or directory", "tar: Error is not recoverable: exiting now"], "stdout": "", "stdout_lines": []}

PLAY RECAP *******************************************************************************************************************************************************************************
localhost : ok=333 changed=192 unreachable=0 failed=1

Severity
--------
Major

Steps to Reproduce
------------------
See Brief Description

Expected Behavior
------------------
Restore should complete successfully

Actual Behavior
----------------
Restore fails

Reproducibility
---------------
Always

System Configuration
--------------------
Standard Storage lab

Last Pass
---------
Unknown

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20191111T000000Z"
JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="313"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-11-11 00:00:00 +0000"

Timestamp/Logs
--------------
NA

Test Activity
-------------
Developer Testing

[Ovidiu Poncea (ovidiuponcea)]

Issue most likely happens because sysinv subfolder of /opt/platform doesn't exists. Before https://review.opendev.org/691713 it was created by sysinv, but no longer happens.

Therefore, this seems to be a duplicate of: https://bugs.launchpad.net/starlingx/+bug/1852065 . I'm not going to mark it as duplicate as it will hide it from search.

[Ghada Khalil (gkhalil)]

stx.3.0 / medium - issue introduced by recent code changes and affects stx.3.0 B&R feature functionality

[Ovidiu Poncea (ovidiuponcea)]

This is also breaking SX, as found by Kristine, platform-integ-apps does not start.

Error in sysinv log is:
sysinv 2019-11-13 15:14:30.118 105737 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Prerequisites not met

One of the prerequisites is /opt/platform/sysinv/19.09/.crushmap_applied to exists, which doesn't exist because of:

sysinv 2019-11-13 15:12:22.355 105737 WARNING sysinv.common.ceph [-] Failed to create flag file: /opt/platform/sysinv/19.09/.crushmap_applied. Reason: [Errno 2] No such file or directory: '/opt/platform/sysinv/19.09/.crushmap_applied': IOError: [Errno 2] No such file or directory: '/opt/platform/sysinv/19.09/.crushmap_applied'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/694774

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694774
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=dcdeea0254149233f1c0e5a6536a561cf7453cec
Submitter: Zuul
Branch: master

commit dcdeea0254149233f1c0e5a6536a561cf7453cec
Author: Ovidiu Poncea <email address hidden>
Date: Mon Nov 18 15:48:23 2019 +0200

    Fix missing content in /opt/platform/sysinv/19.09

    Due to changes in https://review.opendev.org/#/c/692439 and
    https://review.opendev.org/#/c/691714, sysinv/19.09 subfolder
    of /opt/platform is no longer created, nor its content.
    This breaks the initial assumptions that:
    1. This folder exists => is ok to just create files there
    2. The content of this folder is recreated on each unlock

    To return #1 assumption we now create the folder in advance
    and for #2 we back-up and restore its content.

    Change-Id: I8dd686a66fcc62bbb05b72fda56e86c353d25fee
    Closes-Bug: 1851424
    Closes-Bug: 1852065
    Closes-Bug: 1852127
    Signed-off-by: Ovidiu Poncea <email address hidden>