Backup & Restore: Backup fails due to undefined variable

[Senthil Mukundakumar]

Brief Description
-----------------
Backup from active controller fails due to following error:

TASK [backup/backup-system : Get the openstack Helm overrides from the from the database] **************
changed: [localhost]

TASK [backup/backup-system : Generate postgres update commands for Helm overrides] *********************

TASK [backup/backup-system : Backup Helm overrides] ****************************************************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'updates_list' is undefined\n\nThe error appears to have been in '/usr/share/ansible/stx-ansible/playbooks/roles/backup/backup-system/tasks/main.yml': line 180, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Backup Helm overrides\n ^ here\n"}

TASK [backup/backup-system : Remove the temp dir] ******************************************************
changed: [localhost]

TASK [backup/backup-system : Remove the backup in progress flag file] **********************************
changed: [localhost]

TASK [backup/backup-system : Clear backup_in_progress alarm] *******************************************
changed: [localhost]

TASK [backup/backup-system : Fail if alarm script throws an exception] *********************************

PLAY RECAP *********************************************************************************************
localhost : ok=30 changed=23 unreachable=0 failed=1

Severity
--------
Provide the severity of the defect.
Critical: Controller-0 failed to backup

Steps to Reproduce
------------------
1. Bring up the IPV6 regular system
2. Backup the system locally
3. Re-install the controller with the same load
4. Restore the system locally from controller

Expected Behavior
------------------
The active controller must be restored from backup

Actual Behavior
----------------
Failed to backup active controller

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Any system (WCP_78_79, WCP_92_98)

Branch/Pull Time/Commit
-----------------------
master BUILD_ID="2020-01-19_00-10-00"

Test Activity
-------------
Feature Testing

[Ghada Khalil (gkhalil)]

Marking as stx.4.0 / high priority -- current assumption is that B&R was working in stx.3.0. If this is an issue in that release as well, we'll need to update the release tag and cherrypick to r/stx.3.0.

[Angie Wang (angiewang)]

I hit the same issue. It's introduced in commit https://opendev.org/starlingx/ansible-playbooks/commit/f8dc7a8faf225609ebbd7f0e72e4c412898aeaaa.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/703653
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=e03b1efbaf581ab49b88397a0c9f3802d64381a6
Submitter: Zuul
Branch: master

commit e03b1efbaf581ab49b88397a0c9f3802d64381a6
Author: Mihnea Saracin <email address hidden>
Date: Tue Jan 21 14:59:05 2020 +0200

    B&R: Fix backup when openstack is not installed

    When the stx-openstack is not installed, a variable
    is left undefined in the ansible playbook, so we fix
    this by adding a default value for that variable.

    Change-Id: I3a2480456074022c0a1227bc19cea92f318afb1f
    Closes-Bug: #1860357
    Signed-off-by: Mihnea Saracin <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/705831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705831
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=6670caf7ceda5fe0dc46f2f82033b68abf00ed5e
Submitter: Zuul
Branch: f/centos8

commit bf8d081a95a9b1776964960a6d9089b1449f2c58
Author: Angie Wang <email address hidden>
Date: Thu Jan 30 17:57:05 2020 -0500

    Support k8s networking upgrade based on k8s version

    Update to support a set of k8s networking templates
    based on kubernetes release. The kubernetes version
    needs to be passed to the ansible playbook
    k8s-networking-upgrade.yml to determine which set
    of networking manifests should be applied for the
    current kubernetes.

    Story: 2006781
    Task: 37584
    Change-Id: I3a0b9f56608ddb1323b36f9ecedb8a5488c222c9
    Signed-off-by: Angie Wang <email address hidden>

commit 2b0cd43e5fa75628d8eff78be7045ba4fc82d980
Author: Jerry Sun <email address hidden>
Date: Thu Dec 19 13:22:50 2019 -0500

    Add Dex parameters to ansible bootstrap

    Add oidc_groups_claim as a new parameters for ansible
    config. We now have 2 valid configs: the previous 3 parameters
    for a microsoft azure authentication deployment, and the previous
    3 in addition to oidc_groups_claim for a dex authentication
    deployment.

    Story: 2006711
    Task: 37850
    Change-Id: I265d2f7872eb31e2b295eeff6a3165543673497c
    Depends-On: https://review.opendev.org/702798
    Signed-off-by: Jerry Sun <email address hidden>

commit 92ca122652733805b62fc16940861ca4e83e2bb1
Author: David Sullivan <email address hidden>
Date: Wed Jan 22 21:33:19 2020 -0500

    Install secondary controller nodes with kubeadm join

    Kubeadm init is no longer supported for installing secondary nodes in an
    HA kubernetes cluster. kubeadm join with the --controller-plane option
    should be used.

    Change-Id: I64aaf02b09053608c884149d73bc1a3f2b62d98a
    Partial-Bug: 1846829
    Depends-On: https://review.opendev.org/702797
    Signed-off-by: David Sullivan <email address hidden>

commit 393379bd7671aeec5e9852679a69bdc29577426a
Author: Angie Wang <email address hidden>
Date: Tue Jan 28 14:01:10 2020 -0500

    Fix the image download failure on IPv6 system

    "crictl pull" failed to pull images on IPv6 system with
    proxy setting since Containerd doesn't work with the
    NO_PROXY environment variable that has IPv6 addresses
    with square brackets. This commit updates to strip out
    the square brackets from NO_PROXY environment variable.

    Verified on both IPv4 and IPv6 labs.

    Change-Id: I70bd00439b2cc39d2b25dd62746994a524be4998
    Partial-Bug: 1859835
    Signed-off-by: Angie Wang <email address hidden>

commit 792ea357e2b6d2bd23b441aa1657e0dc46f7ef5d
Author: Jim Somerville <email address hidden>
Date: Mon Jan 27 16:08:48 2020 -0500

    Security: Add nospectre_v1 to the default setting

    Most of the v1 mitigation is baked into the kernel and not
    optional. The swapgs barriers are, however, optional.
    They have a negative performance impact so we disable them
    by using the nospectre_v1 kernel bootarg.

    C...