Update resgistry credentials failed when openstack secret doesn't exist

[Yuxing]

Brief Description

After a platform upgrade from stx.5.0 to stx.6.0, the subcloud rehome failed due to the update_docker_registry.sh failed

Severity

Major

Steps to Reproduce

1. Upgrade a subcloud from 21.05 to 21.12
2. Rehome this subcloud to a 21.12 DC system

Expected Behavior

Rehome successful

Actual Behavior

Rehome failed in the step: Update docker registry credentials

Reproducibility

Reproducible

System Configuration

DC

Branch/Pull Time/Commit
SW_VERSION="21.12"

Last Pass
NA

Timestamp/Logs

TASK [rehome-subcloud/update-keystone-data : Restart keystone service] *********
Saturday 04 December 2021 14:12:12 +0000 (0:00:00.406) 0:00:31.663 *****
changed: [subcloud11]

TASK [rehome-subcloud/update-keystone-data : Wait until keystone is restarted] ***
Saturday 04 December 2021 14:12:13 +0000 (0:00:01.433) 0:00:33.097 *****
FAILED - RETRYING: Wait until keystone is restarted (10 retries left).
changed: [subcloud11]

TASK [rehome-subcloud/update-keystone-data : Update docker registry credentials] ***
Saturday 04 December 2021 14:12:24 +0000 (0:00:10.879) 0:00:43.976 *****
fatal: [subcloud11]: FAILED! => changed=true
cmd:

update_docker_registry_auth.sh
sysinv
O4LUr=X7LvF*f=k4
delta: '0:01:08.868264'
end: '2021-12-04 14:13:33.679879'
msg: non-zero return code
rc: 2
start: '2021-12-04 14:12:24.811615'
stderr: |-
usage: openstack secret get [-h] [-f {json,shell,table,value,yaml}]
[-c COLUMN] [--max-width <integer>] [--fit-width]
[--print-empty] [--noindent] [--prefix PREFIX]
[--decrypt | --payload | --file <filename>]
[--payload_content_type PAYLOAD_CONTENT_TYPE]
URI
openstack secret get: error: too few arguments
stderr_lines:
- 'usage: openstack secret get [-h] [-f {json,shell,table,value,yaml}
]'

' [-c COLUMN] [--max-width <integer>] [--fit-width]'
' [--print-empty] [--noindent] [--prefix PREFIX]'
' [--decrypt | --payload | --file <filename>]'
' [--payload_content_type PAYLOAD_CONTENT_TYPE]'
' URI'
'openstack secret get: error: too few arguments'
stdout: |2-
Updating docker-registry credentials ...... done.
Validating docker-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4

Updating quay-registry credentials ...... done.
Validating quay-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4

Updating elastic-registry credentials ...... done.
Validating elastic-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4

Updating gcr-registry credentials ...... done.
Validating gcr-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4

Updating k8s-registry credentials ...... done.
Validating k8s-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4

Updating ghcr-registry credentials ..
stdout_lines: <omitted>

PLAY RECAP *********************************************************************
subcloud11 : ok=36 changed=20 unreachable=0 failed=1

Saturday 04 December 2021 14:13:33 +0000 (0:01:09.094) 0:01:53.071 *****

controller-0:~$ source /etc/platform/openrc
[sysadmin@controller-0 ~(keystone_admin)]$ openstack secret list
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Secret href Name Created Status Content types Algorithm Bit length Secret type Mode Expiration
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

http://controller:9311/v1/secrets/de3fe51c-a0bd-4fe9-9e66-5ba7d5c49edc k8s-registry-secret 2021-12-04T14:13:23.899474+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/e3b40483-925f-404d-bcb8-db3d1975f3d1 gcr-registry-secret 2021-12-04T14:13:11.197411+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/194687fe-ad84-432f-a0c5-469e8540abf3 elastic-registry-secret 2021-12-04T14:12:58.203490+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/800b70dd-f7d5-4980-8446-e8ecfb0385e2 quay-registry-secret 2021-12-04T14:12:45.409269+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/dc04e11a-6004-432f-b638-d056cbf771bc docker-registry-secret 2021-12-04T14:12:31.652723+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system service-parameter-list
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

uuid service section name value personality resource
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

dd489d53-bc4b-4528-9dfd-8c7e76319244 radosgw config fs_size_mb 25 None None
c1481786-2068-4ccb-937f-ec10a04e4c76 http config http_port 8080 None None
18accb80-e186-441e-a814-108050e89714 http config https_port 8443 None None
ee615917-ecde-42b5-b74a-2127128c693e kubernetes config pod_max_pids 10000 None None
a4878eec-c4c1-474f-a247-837d727fbdc7 radosgw config service_enabled false None None
f39ee3c7-de9a-4020-ae75-a1c058a7b7e5 identity config token_expiration 3600 None None
b5a36f18-4fd5-4ea0-81f5-d32fbb01adfa docker docker-registry auth-secret dc04e11a-6004-432f-b638-d056cbf771bc None None
bd1b7c7a-1e9b-4361-a859-09115f9cbd8a docker docker-registry type docker None None
02d2fc97-5caf-45d1-9365-7980cb16e303 docker docker-registry url registry.central:9001/docker.io None None
a941da2a-986c-4c01-a503-a709ce51cd70 docker elastic-registry auth-secret 194687fe-ad84-432f-a0c5-469e8540abf3 None None
ac35122f-d84a-4e77-9645-b9b7b64e2180 docker elastic-registry type docker None None
4e2b246b-5771-4b6f-9720-9e45544c86a5 docker elastic-registry url registry.central:9001/docker.elastic.co None None
6913eafe-a441-4ab9-9aaf-2dd700d82288 docker gcr-registry auth-secret e3b40483-925f-404d-bcb8-db3d1975f3d1 None None
5105c878-9c10-4aee-a593-f7ae5770688c docker gcr-registry type docker None None
6354fb56-ba40-47fe-b061-228a0236a8cf docker gcr-registry url registry.central:9001/gcr.io None None
95a809cc-ecf9-4671-96cc-c0169b1b3d6f docker ghcr-registry auth-secret 68bd9137-6103-4b3c-8b2a-8acdd74e6ac9 None None
23ce0761-0480-4a27-a190-7c1d06924cf8 docker ghcr-registry type docker None None
6169287f-a278-468a-90f0-0cb6162c9269 docker ghcr-registry url registry.central:9001/ghcr.io None None
96ac8bf3-3e5d-4362-9508-44e6cfa93ba3 docker k8s-registry auth-secret de3fe51c-a0bd-4fe9-9e66-5ba7d5c49edc None None
ce7ef65e-f88a-4141-96dd-617800e5d0e1 docker k8s-registry type docker None None
118a1a08-3d22-4458-b486-adec01ce4400 docker k8s-registry url registry.central:9001/k8s.gcr.io None None
9b5b5037-7671-4e3d-b1cc-bb7ed33e1fbc platform kernel audit 0 None None
f6f1f57f-441e-47ca-9be6-bb43b422146f platform maintenance controller_boot_timeout 1200 None None
c2f0ce80-b25b-4f8e-93ff-ca7c39ba1268 platform maintenance heartbeat_degrade_threshold 6 None None
93ec6f8c-a289-472c-a940-5ed044b9e1e8 platform maintenance heartbeat_failure_action fail None None
a93eb86f-6e2b-4b1b-b23e-0ab48fc1c4b0 platform maintenance heartbeat_failure_threshold 10 None None
c3e91588-5895-4985-b549-56c5df9affd0 platform maintenance heartbeat_period 100 None None
98569e52-c482-47bf-b3a9-b7a2d5f75c55 platform maintenance mnfa_threshold 2 None None
3e6b93bb-903d-42ca-892d-97befa5d4db3 platform maintenance mnfa_timeout 0 None None
47339286-fec3-4e7c-a04b-c3201ff18c43 platform maintenance worker_boot_timeout 720 None None
24947921-ec5f-4a7c-854f-4366352963c7 docker proxy http_proxy http://yow-proxomatic.wrs.com:3128 None None
720ded1b-fb1d-4e87-83a0-a9854d8ff946 docker proxy no_proxy localhost,127.0.0.1,registry.local,[fd04::1],[fd01:8c::2],[fd01:8c::3],[2620:10a:a001:a103::1135], None None
        registry.central,[2620:10a:a001:a103::167],tis-lab-registry.cumulus.wrs.com

4f88e981-d345-4fd7-b2e6-27b0b4131d8f docker quay-registry auth-secret 800b70dd-f7d5-4980-8446-e8ecfb0385e2 None None
d91b5b80-092f-4ab5-86cf-a3d8782e203d docker quay-registry type docker None None
cc800ccb-140a-4c94-9311-84d6c2dbedbd docker quay-registry url registry.central:9001/quay.io None None
5beae5bf-8a74-4285-a28b-63c786b9a4f0 identity security_compliance lockout_retries 5 None None
b14aa612-70b6-4a68-9b7e-795c7958433b identity security_compliance lockout_seconds 1800 None None

Alarms

NA

Test Activity

Regression Testing

Workaround

NA

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/utilities/+/820575

[Ghada Khalil (gkhalil)]

screening: marking as low / not gating. There is no official upgrades support in stx. It's best effort. Issue can be fixed, but will not hold up any upcoming release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/utilities/+/820575
Committed: https://opendev.org/starlingx/utilities/commit/f0addaecbd23ef2e52b584be8a4db3073b26d2b3
Submitter: "Zuul (22348)"
Branch: master

commit f0addaecbd23ef2e52b584be8a4db3073b26d2b3
Author: Yuxing Jiang <email address hidden>
Date: Mon Dec 6 09:58:00 2021 -0600

    Skip the secret deletion if it doesn't exist

    If the registry secret exists in the service parameters but doesn't
    exist in the OpenStack secret list, the current update script will
    throw out an error. This commit skips the secret deletion operation if
    the secret doesn't exists, allows creating a new secret and updating
    it to the service parameters.

    Test:
    1. Bring up a DC(DX central cloud + SX subcloud) with this change.
    2. Delete the ghcr registry secret manually, update the secrets again
    with sysinv credentials in the subcloud.
    3. Delete the ghcr registry secret manually, migrate the subcloud to
    another central cloud.

    Closes-bug: 1953365
    Signed-off-by: Yuxing Jiang <email address hidden>
    Change-Id: Id6afea988fa8559d8a80eaa5ddeec0a35a386014