B&R: backup failed with netapp storage backend

[Angie Wang]

Brief Description
-----------------
With netapp storage backend configured, backup user local registry images failed due to authentication issue when pulling netapp images from local registry.

Severity
--------
Medium

Steps to Reproduce
------------------
1. Install a lab with netapp stroage backend configured
2. Run backup playbook with "backup_user_local_registry"
   ansible-playbook /usr/share/ansible/stx-ansible/playbooks/backup.yml -e "ansible_become_pass=Li69nux* admin_password=Li69nux*" -e "backup_user_local_registry=true"

Expected Behavior
------------------
System backup succeed

Actual Behavior
----------------
System backup failed

Reproducibility
---------------
100%

System Configuration
--------------------
All system types

Timestamp/Logs
--------------
TASK [backup/backup-system : Pull images from local registry to docker filesystem]
fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-sysadmin/tmp/ansible-tmp-1600394729.9-67720743565920/push_pull_local_registry.py\", line 142, in <module>\n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['registry.local:9001/quay.io/k8scsi/csi-node-driver-registrar:v1.3.0', 'registry.local:9001/quay.io/k8scsi/csi-snapshotter:v2.1.0', 'registry.local:9001/gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0', 'registry.local:9001/gcr.io/kubernetes-helm/tiller:v2.13.1', 'registry.local:9001/tis-lab-registry.cumulus.wrs.com/wind-river/cloud-platform-deployment-manager:WRCP_20.06', 'registry.local:9001/quay.io/k8scsi/csi-provisioner:v1.6.0', 'registry.local:9001/docker.io/starlingx/rvmc:stx.4.0-v1.0.0', 'registry.local:9001/docker.io/netapp/trident:20.04.0', 'registry.local:9001/quay.io/k8scsi/csi-attacher:v2.2.0', 'registry.local:9001/quay.io/k8scsi/csi-resizer:v0.5.0']\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/tmp/.ansible-sysadmin/tmp/ansible-tmp-1600394729.9-67720743565920/push_pull_local_registry.py\", line 142, in <module>", " raise Exception(\"Failed to download images %s\" % failed_downloads)", "Exception: Failed to download images ['registry.local:9001/quay.io/k8scsi/csi-node-driver-registrar:v1.3.0', 'registry.local:9001/quay.io/k8scsi/csi-snapshotter:v2.1.0', 'registry.local:9001/gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0', 'registry.local:9001/gcr.io/kubernetes-helm/tiller:v2.13.1', 'registry.local:9001/tis-lab-registry.cumulus.wrs.com/wind-river/cloud-platform-deployment-manager:WRCP_20.06', 'registry.local:9001/quay.io/k8scsi/csi-provisioner:v1.6.0', 'registry.local:9001/docker.io/starlingx/rvmc:stx.4.0-v1.0.0', 'registry.local:9001/docker.io/netapp/trident:20.04.0', 'registry.local:9001/quay.io/k8scsi/csi-attacher:v2.2.0', 'registry.local:9001/quay.io/k8scsi/csi-resizer:v0.5.0']"], "stdout": " Image download failed: registry.local:9001/quay.io/k8scsi/csi-node-driver-registrar:v1.3.0 500 Server Error: Internal Server Error (\"Get https://registry.local:9001/v2/quay.io/k8scsi/csi-node-driver-registrar/manifests/v1.3.0: unauthorized: authentication required\")\nSleep 10s before retry downloading image registry.local:9001/quay.io/k8scsi/csi-node-driver-registrar:v1.3.0 ...\n Image download failed: registry.local:9001/quay.io/k8scsi/csi-node-driver-registrar:v1.3.0 500 Server Error: Internal Server Error (\"Get https://registry.local:9001/v2/quay.io/k8scsi/csi-node-driver-registrar/manifests/v1.3.0: unauthorized: authentication required\")\nSleep 10s before retry downloading image registry.local:9001/quay.io/k8scsi/csi-node-driver-registrar:v1.3.0 ...\n Image download failed: registry.local:9001/quay.io/k8scsi/csi-node-driver-registrar:v1.3.0 500 Server Error: Internal Server Error (\"Get https://registry.local:9001/v2/quay.io/k8scsi/csi-node-driver-registrar/manifests/v1.3.0: unauthorized: authentication required\")\nSleep 10s before retry
TASK [backup/backup-system : Clear backup_in_progress alarm]
changed: [localhost]

TASK [backup/backup-system : Fail if alarm script throws an exception]

PLAY RECAP
localhost : ok=99 changed=45 unreachable=0 failed=1

Test Activity
-------------
Testing

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/752730

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/752730
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=fd60f4b2714ff952c17564dea88fa6d439836283
Submitter: Zuul
Branch: master

commit fd60f4b2714ff952c17564dea88fa6d439836283
Author: Angie Wang <email address hidden>
Date: Fri Sep 18 12:47:24 2020 -0400

    B&R: Pass the credential of local registry when pulling images

    The commit fixes the netapp images backup failure due to docker
    pull authentication issue.

    Change-Id: I6aea692c6c7d507d58ac564808eb91b6b0fcebaf
    Closes-Bug: 1896253
    Signed-off-by: Angie Wang <email address hidden>