Backup & Restore: Ansible restore times out while waiting for kubernetes essential services

[Ghada Khalil]

Brief Description
-----------------
The execution of the ansible restore playbook in a Storage system with the option wipe_ceph_osds=true fails, and it seems that only occurs in large Storage systems such wcp_15_22.

Severity
--------
Major: System/Feature is usable but degraded

Steps to Reproduce
------------------
Perform a backup for the system with the ansible backup playbook
Power off all hosts but controller-0
Fresh install controller-0
Run the ansible restore playbook with the option wipe_ceph_osds=true and the backup tgz generated before

Expected Behavior
------------------
The ansible execution finishes without errors.

Actual Behavior
----------------
The ansible execution finishes with errors.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Ipv6 Storage system: 4 computes - 2 storages.

Branch/Pull Time/Commit
-----------------------
2020-11-18_20-00-07

Last Pass
---------
Unknown

Timestamp/Logs
--------------
[2020-11-20 13:01:33,414] 154 INFO MainThread restore_helper.restore_system:: Run ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_platform.yml -e "initial_backup_dir=/home/sysadmin/ ansible_become_pass=Li69nux* admin_password=Li69nux* wipe_ceph_osds=true backup_filename=localhost_platform_backup_2020_11_20_03_33_52.tgz"

E TASK [bootstrap/bringup-essential-services : Fail if any of the Kubernetes component, Networking or Tiller pods are not ready by this time] **************************************************************************************************************
 E fatal: [localhost]: FAILED! => {"msg": "The conditional check 'item.stdout is not search(\" condition met\")' failed. The error was: Unexpected templating type error occurred on (
{% if item.stdout is not search(\" condition met\") %}
True {% else %} False {% endif %}): expected string or buffer\n\nThe error appears to have been in '/usr/share/ansible/stx-ansible/playbooks/roles/bootstrap/bringup-essential-services/tasks/main.yml': line 214, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Fail if any of the Kubernetes component, Networking or Tiller pods are not ready by this time\n ^ here\n"}
 E
 E PLAY RECAP ***********************************************************************************************************************************************************************************************************************************************

Test Activity
-------------
Feature testing

Workaround
----------
None

[Cole Walker (cwalops)]

Fixed by https://review.opendev.org/c/starlingx/ansible-playbooks/+/764857

Merged on 2020-12-11

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794280

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794280

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/163ec9989cc7360dba4c572b2c43effd10306048
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 4e96b762f549aadb0291cc9bcf3352ae923e94eb
Author: Mihnea Saracin <email address hidden>
Date: Sat May 22 15:48:19 2021 +0000

    Revert "Restore host filesystems with collected sizes"

    This reverts commit 255488739efa4ac072424b19f2dbb7a3adb0254e.

    Reason for revert: Did a rework to fix https://bugs.launchpad.net/starlingx/+bug/1926591. The original problem was in puppet, and this fix in ansible was not good enough, it generated some other problems.

    Change-Id: Iea79701a874effecb7fe995ac468d50081d1a84f
    Depends-On: I55ae6954d24ba32e40c2e5e276ec17015d9bba44

commit c064aacc377c8bd5336ceab825d4bcbf5af0b5e8
Author: Angie Wang <email address hidden>
Date: Fri May 21 21:28:02 2021 -0400

    Ensure apiserver keys are present before extract from tarball

    This is to fix the upgrade playbook issue that happens during
    AIO-SX upgrade from stx4.0 to stx5.0 which introduced by
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/792093.
    The apiserver keys are not available in stx4.0 side so we need
    to ensure the keys under /etc/kubernetes/pki are present in the
    backed-up tarball before extracting, otherwise playbook fails
    because the keys are not found in the archive.

    Change-Id: I8602f07d1b1041a7fd3fff21e6f9a422b9784ab5
    Closes-Bug: 928925
    Signed-off-by: Angie Wang <email address hidden>

commit 0261f22ff7c23d2a8608fe3b51725c9f29931281
Author: Don Penney <email address hidden>
Date: Thu May 20 23:09:07 2021 -0400

    Update SX to DX migration to wait for coredns config

    This commit updates the SX to DX migration playbook to wait after
    modifying the system mode to duplex until the runtime manifest that
    updates coredns config has completed. The playbook will wait for up to
    20 minutes to allow for the possibilty that sysinv has multiple
    runtime manifests queued up, each of which could take several minutes.

    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/792494
    Depends-On: https://review.opendev.org/c/starlingx/config/+/792496
    Change-Id: I3bf94d3493ae20eeb16b3fdcb27576ee18c0dc4d
    Closes-Bug: 1929148
    Signed-off-by: Don Penney <email address hidden>

commit 7c4f17bd0d92fc1122823211e1c9787829d206a9
Author: Daniel Safta <email address hidden>
Date: Wed May 19 09:08:16 2021 +0000

    Fixed missing apiserver-etcd-client certs

    When controller-1 is the active controller
    the backup archive does not contain
    /etc/etcd/apiserver-etcd-client.{crt, key}

    This change adds a new task which brings
    the certs from /etc/kubernetes/pki

    Closes-bug: 1928925
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: I3c68377603e1af9a71d104e5b1108e9582497a09

commit e221ef8fbe51aa6ca229b584fb5632fe512ad5cb
Author: David Sullivan <email address hidden>
Date: Wed May 19 16:01:27 2021 -0500

    Support boo...