| 2021-12-07 23:26:29 |
Ghada Khalil |
description |
Brief Description
After a platform upgrade from 21.05 to 21.12, the subcloud rehome failed due to the update_docker_registry.sh failed
Severity
Major
Steps to Reproduce
1. Upgrade a subcloud from 21.05 to 21.12
2. Rehome this subcloud to a 21.12 DC system
Expected Behavior
Rehome successful
Actual Behavior
Rehome failed in the step: Update docker registry credentials
Reproducibility
Reproducible
System Configuration
DC
Branch/Pull Time/Commit
SW_VERSION="21.12"
Last Pass
NA
Timestamp/Logs
TASK [rehome-subcloud/update-keystone-data : Restart keystone service] *********
Saturday 04 December 2021 14:12:12 +0000 (0:00:00.406) 0:00:31.663 *****
changed: [subcloud11]
TASK [rehome-subcloud/update-keystone-data : Wait until keystone is restarted] ***
Saturday 04 December 2021 14:12:13 +0000 (0:00:01.433) 0:00:33.097 *****
FAILED - RETRYING: Wait until keystone is restarted (10 retries left).
changed: [subcloud11]
TASK [rehome-subcloud/update-keystone-data : Update docker registry credentials] ***
Saturday 04 December 2021 14:12:24 +0000 (0:00:10.879) 0:00:43.976 *****
fatal: [subcloud11]: FAILED! => changed=true
cmd:
update_docker_registry_auth.sh
sysinv
O4LUr=X7LvF*f=k4
delta: '0:01:08.868264'
end: '2021-12-04 14:13:33.679879'
msg: non-zero return code
rc: 2
start: '2021-12-04 14:12:24.811615'
stderr: |-
usage: openstack secret get [-h] [-f {json,shell,table,value,yaml}]
[-c COLUMN] [--max-width <integer>] [--fit-width]
[--print-empty] [--noindent] [--prefix PREFIX]
[--decrypt | --payload | --file <filename>]
[--payload_content_type PAYLOAD_CONTENT_TYPE]
URI
openstack secret get: error: too few arguments
stderr_lines:
- 'usage: openstack secret get [-h] [-f {json,shell,table,value,yaml}
]'
' [-c COLUMN] [--max-width <integer>] [--fit-width]'
' [--print-empty] [--noindent] [--prefix PREFIX]'
' [--decrypt | --payload | --file <filename>]'
' [--payload_content_type PAYLOAD_CONTENT_TYPE]'
' URI'
'openstack secret get: error: too few arguments'
stdout: |2-
Updating docker-registry credentials ...... done.
Validating docker-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating quay-registry credentials ...... done.
Validating quay-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating elastic-registry credentials ...... done.
Validating elastic-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating gcr-registry credentials ...... done.
Validating gcr-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating k8s-registry credentials ...... done.
Validating k8s-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating ghcr-registry credentials ..
stdout_lines: <omitted>
PLAY RECAP *********************************************************************
subcloud11 : ok=36 changed=20 unreachable=0 failed=1
Saturday 04 December 2021 14:13:33 +0000 (0:01:09.094) 0:01:53.071 *****
controller-0:~$ source /etc/platform/openrc
[sysadmin@controller-0 ~(keystone_admin)]$ openstack secret list
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Secret href Name Created Status Content types Algorithm Bit length Secret type Mode Expiration
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
http://controller:9311/v1/secrets/de3fe51c-a0bd-4fe9-9e66-5ba7d5c49edc k8s-registry-secret 2021-12-04T14:13:23.899474+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/e3b40483-925f-404d-bcb8-db3d1975f3d1 gcr-registry-secret 2021-12-04T14:13:11.197411+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/194687fe-ad84-432f-a0c5-469e8540abf3 elastic-registry-secret 2021-12-04T14:12:58.203490+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/800b70dd-f7d5-4980-8446-e8ecfb0385e2 quay-registry-secret 2021-12-04T14:12:45.409269+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/dc04e11a-6004-432f-b638-d056cbf771bc docker-registry-secret 2021-12-04T14:12:31.652723+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system service-parameter-list
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid service section name value personality resource
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dd489d53-bc4b-4528-9dfd-8c7e76319244 radosgw config fs_size_mb 25 None None
c1481786-2068-4ccb-937f-ec10a04e4c76 http config http_port 8080 None None
18accb80-e186-441e-a814-108050e89714 http config https_port 8443 None None
ee615917-ecde-42b5-b74a-2127128c693e kubernetes config pod_max_pids 10000 None None
a4878eec-c4c1-474f-a247-837d727fbdc7 radosgw config service_enabled false None None
f39ee3c7-de9a-4020-ae75-a1c058a7b7e5 identity config token_expiration 3600 None None
b5a36f18-4fd5-4ea0-81f5-d32fbb01adfa docker docker-registry auth-secret dc04e11a-6004-432f-b638-d056cbf771bc None None
bd1b7c7a-1e9b-4361-a859-09115f9cbd8a docker docker-registry type docker None None
02d2fc97-5caf-45d1-9365-7980cb16e303 docker docker-registry url registry.central:9001/docker.io None None
a941da2a-986c-4c01-a503-a709ce51cd70 docker elastic-registry auth-secret 194687fe-ad84-432f-a0c5-469e8540abf3 None None
ac35122f-d84a-4e77-9645-b9b7b64e2180 docker elastic-registry type docker None None
4e2b246b-5771-4b6f-9720-9e45544c86a5 docker elastic-registry url registry.central:9001/docker.elastic.co None None
6913eafe-a441-4ab9-9aaf-2dd700d82288 docker gcr-registry auth-secret e3b40483-925f-404d-bcb8-db3d1975f3d1 None None
5105c878-9c10-4aee-a593-f7ae5770688c docker gcr-registry type docker None None
6354fb56-ba40-47fe-b061-228a0236a8cf docker gcr-registry url registry.central:9001/gcr.io None None
95a809cc-ecf9-4671-96cc-c0169b1b3d6f docker ghcr-registry auth-secret 68bd9137-6103-4b3c-8b2a-8acdd74e6ac9 None None
23ce0761-0480-4a27-a190-7c1d06924cf8 docker ghcr-registry type docker None None
6169287f-a278-468a-90f0-0cb6162c9269 docker ghcr-registry url registry.central:9001/ghcr.io None None
96ac8bf3-3e5d-4362-9508-44e6cfa93ba3 docker k8s-registry auth-secret de3fe51c-a0bd-4fe9-9e66-5ba7d5c49edc None None
ce7ef65e-f88a-4141-96dd-617800e5d0e1 docker k8s-registry type docker None None
118a1a08-3d22-4458-b486-adec01ce4400 docker k8s-registry url registry.central:9001/k8s.gcr.io None None
9b5b5037-7671-4e3d-b1cc-bb7ed33e1fbc platform kernel audit 0 None None
f6f1f57f-441e-47ca-9be6-bb43b422146f platform maintenance controller_boot_timeout 1200 None None
c2f0ce80-b25b-4f8e-93ff-ca7c39ba1268 platform maintenance heartbeat_degrade_threshold 6 None None
93ec6f8c-a289-472c-a940-5ed044b9e1e8 platform maintenance heartbeat_failure_action fail None None
a93eb86f-6e2b-4b1b-b23e-0ab48fc1c4b0 platform maintenance heartbeat_failure_threshold 10 None None
c3e91588-5895-4985-b549-56c5df9affd0 platform maintenance heartbeat_period 100 None None
98569e52-c482-47bf-b3a9-b7a2d5f75c55 platform maintenance mnfa_threshold 2 None None
3e6b93bb-903d-42ca-892d-97befa5d4db3 platform maintenance mnfa_timeout 0 None None
47339286-fec3-4e7c-a04b-c3201ff18c43 platform maintenance worker_boot_timeout 720 None None
24947921-ec5f-4a7c-854f-4366352963c7 docker proxy http_proxy http://yow-proxomatic.wrs.com:3128 None None
720ded1b-fb1d-4e87-83a0-a9854d8ff946 docker proxy no_proxy localhost,127.0.0.1,registry.local,[fd04::1],[fd01:8c::2],[fd01:8c::3],[2620:10a:a001:a103::1135], None None
registry.central,[2620:10a:a001:a103::167],tis-lab-registry.cumulus.wrs.com
4f88e981-d345-4fd7-b2e6-27b0b4131d8f docker quay-registry auth-secret 800b70dd-f7d5-4980-8446-e8ecfb0385e2 None None
d91b5b80-092f-4ab5-86cf-a3d8782e203d docker quay-registry type docker None None
cc800ccb-140a-4c94-9311-84d6c2dbedbd docker quay-registry url registry.central:9001/quay.io None None
5beae5bf-8a74-4285-a28b-63c786b9a4f0 identity security_compliance lockout_retries 5 None None
b14aa612-70b6-4a68-9b7e-795c7958433b identity security_compliance lockout_seconds 1800 None None
Alarms
NA
Test Activity
Regression Testing
Workaround
NA |
Brief Description
After a platform upgrade from stx.5.0 to stx.6.0, the subcloud rehome failed due to the update_docker_registry.sh failed
Severity
Major
Steps to Reproduce
1. Upgrade a subcloud from 21.05 to 21.12
2. Rehome this subcloud to a 21.12 DC system
Expected Behavior
Rehome successful
Actual Behavior
Rehome failed in the step: Update docker registry credentials
Reproducibility
Reproducible
System Configuration
DC
Branch/Pull Time/Commit
SW_VERSION="21.12"
Last Pass
NA
Timestamp/Logs
TASK [rehome-subcloud/update-keystone-data : Restart keystone service] *********
Saturday 04 December 2021 14:12:12 +0000 (0:00:00.406) 0:00:31.663 *****
changed: [subcloud11]
TASK [rehome-subcloud/update-keystone-data : Wait until keystone is restarted] ***
Saturday 04 December 2021 14:12:13 +0000 (0:00:01.433) 0:00:33.097 *****
FAILED - RETRYING: Wait until keystone is restarted (10 retries left).
changed: [subcloud11]
TASK [rehome-subcloud/update-keystone-data : Update docker registry credentials] ***
Saturday 04 December 2021 14:12:24 +0000 (0:00:10.879) 0:00:43.976 *****
fatal: [subcloud11]: FAILED! => changed=true
cmd:
update_docker_registry_auth.sh
sysinv
O4LUr=X7LvF*f=k4
delta: '0:01:08.868264'
end: '2021-12-04 14:13:33.679879'
msg: non-zero return code
rc: 2
start: '2021-12-04 14:12:24.811615'
stderr: |-
usage: openstack secret get [-h] [-f {json,shell,table,value,yaml}]
[-c COLUMN] [--max-width <integer>] [--fit-width]
[--print-empty] [--noindent] [--prefix PREFIX]
[--decrypt | --payload | --file <filename>]
[--payload_content_type PAYLOAD_CONTENT_TYPE]
URI
openstack secret get: error: too few arguments
stderr_lines:
- 'usage: openstack secret get [-h] [-f {json,shell,table,value,yaml}
]'
' [-c COLUMN] [--max-width <integer>] [--fit-width]'
' [--print-empty] [--noindent] [--prefix PREFIX]'
' [--decrypt | --payload | --file <filename>]'
' [--payload_content_type PAYLOAD_CONTENT_TYPE]'
' URI'
'openstack secret get: error: too few arguments'
stdout: |2-
Updating docker-registry credentials ...... done.
Validating docker-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating quay-registry credentials ...... done.
Validating quay-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating elastic-registry credentials ...... done.
Validating elastic-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating gcr-registry credentials ...... done.
Validating gcr-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating k8s-registry credentials ...... done.
Validating k8s-registry credentials updated to: username:sysinv password:O4LUr=X7LvF*f=k4
Updating ghcr-registry credentials ..
stdout_lines: <omitted>
PLAY RECAP *********************************************************************
subcloud11 : ok=36 changed=20 unreachable=0 failed=1
Saturday 04 December 2021 14:13:33 +0000 (0:01:09.094) 0:01:53.071 *****
controller-0:~$ source /etc/platform/openrc
[sysadmin@controller-0 ~(keystone_admin)]$ openstack secret list
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Secret href Name Created Status Content types Algorithm Bit length Secret type Mode Expiration
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
http://controller:9311/v1/secrets/de3fe51c-a0bd-4fe9-9e66-5ba7d5c49edc k8s-registry-secret 2021-12-04T14:13:23.899474+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/e3b40483-925f-404d-bcb8-db3d1975f3d1 gcr-registry-secret 2021-12-04T14:13:11.197411+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/194687fe-ad84-432f-a0c5-469e8540abf3 elastic-registry-secret 2021-12-04T14:12:58.203490+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/800b70dd-f7d5-4980-8446-e8ecfb0385e2 quay-registry-secret 2021-12-04T14:12:45.409269+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
http://controller:9311/v1/secrets/dc04e11a-6004-432f-b638-d056cbf771bc docker-registry-secret 2021-12-04T14:12:31.652723+00:00 ACTIVE {u'default': u'text/plain'} aes 256 opaque cbc None
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system service-parameter-list
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uuid service section name value personality resource
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dd489d53-bc4b-4528-9dfd-8c7e76319244 radosgw config fs_size_mb 25 None None
c1481786-2068-4ccb-937f-ec10a04e4c76 http config http_port 8080 None None
18accb80-e186-441e-a814-108050e89714 http config https_port 8443 None None
ee615917-ecde-42b5-b74a-2127128c693e kubernetes config pod_max_pids 10000 None None
a4878eec-c4c1-474f-a247-837d727fbdc7 radosgw config service_enabled false None None
f39ee3c7-de9a-4020-ae75-a1c058a7b7e5 identity config token_expiration 3600 None None
b5a36f18-4fd5-4ea0-81f5-d32fbb01adfa docker docker-registry auth-secret dc04e11a-6004-432f-b638-d056cbf771bc None None
bd1b7c7a-1e9b-4361-a859-09115f9cbd8a docker docker-registry type docker None None
02d2fc97-5caf-45d1-9365-7980cb16e303 docker docker-registry url registry.central:9001/docker.io None None
a941da2a-986c-4c01-a503-a709ce51cd70 docker elastic-registry auth-secret 194687fe-ad84-432f-a0c5-469e8540abf3 None None
ac35122f-d84a-4e77-9645-b9b7b64e2180 docker elastic-registry type docker None None
4e2b246b-5771-4b6f-9720-9e45544c86a5 docker elastic-registry url registry.central:9001/docker.elastic.co None None
6913eafe-a441-4ab9-9aaf-2dd700d82288 docker gcr-registry auth-secret e3b40483-925f-404d-bcb8-db3d1975f3d1 None None
5105c878-9c10-4aee-a593-f7ae5770688c docker gcr-registry type docker None None
6354fb56-ba40-47fe-b061-228a0236a8cf docker gcr-registry url registry.central:9001/gcr.io None None
95a809cc-ecf9-4671-96cc-c0169b1b3d6f docker ghcr-registry auth-secret 68bd9137-6103-4b3c-8b2a-8acdd74e6ac9 None None
23ce0761-0480-4a27-a190-7c1d06924cf8 docker ghcr-registry type docker None None
6169287f-a278-468a-90f0-0cb6162c9269 docker ghcr-registry url registry.central:9001/ghcr.io None None
96ac8bf3-3e5d-4362-9508-44e6cfa93ba3 docker k8s-registry auth-secret de3fe51c-a0bd-4fe9-9e66-5ba7d5c49edc None None
ce7ef65e-f88a-4141-96dd-617800e5d0e1 docker k8s-registry type docker None None
118a1a08-3d22-4458-b486-adec01ce4400 docker k8s-registry url registry.central:9001/k8s.gcr.io None None
9b5b5037-7671-4e3d-b1cc-bb7ed33e1fbc platform kernel audit 0 None None
f6f1f57f-441e-47ca-9be6-bb43b422146f platform maintenance controller_boot_timeout 1200 None None
c2f0ce80-b25b-4f8e-93ff-ca7c39ba1268 platform maintenance heartbeat_degrade_threshold 6 None None
93ec6f8c-a289-472c-a940-5ed044b9e1e8 platform maintenance heartbeat_failure_action fail None None
a93eb86f-6e2b-4b1b-b23e-0ab48fc1c4b0 platform maintenance heartbeat_failure_threshold 10 None None
c3e91588-5895-4985-b549-56c5df9affd0 platform maintenance heartbeat_period 100 None None
98569e52-c482-47bf-b3a9-b7a2d5f75c55 platform maintenance mnfa_threshold 2 None None
3e6b93bb-903d-42ca-892d-97befa5d4db3 platform maintenance mnfa_timeout 0 None None
47339286-fec3-4e7c-a04b-c3201ff18c43 platform maintenance worker_boot_timeout 720 None None
24947921-ec5f-4a7c-854f-4366352963c7 docker proxy http_proxy http://yow-proxomatic.wrs.com:3128 None None
720ded1b-fb1d-4e87-83a0-a9854d8ff946 docker proxy no_proxy localhost,127.0.0.1,registry.local,[fd04::1],[fd01:8c::2],[fd01:8c::3],[2620:10a:a001:a103::1135], None None
registry.central,[2620:10a:a001:a103::167],tis-lab-registry.cumulus.wrs.com
4f88e981-d345-4fd7-b2e6-27b0b4131d8f docker quay-registry auth-secret 800b70dd-f7d5-4980-8446-e8ecfb0385e2 None None
d91b5b80-092f-4ab5-86cf-a3d8782e203d docker quay-registry type docker None None
cc800ccb-140a-4c94-9311-84d6c2dbedbd docker quay-registry url registry.central:9001/quay.io None None
5beae5bf-8a74-4285-a28b-63c786b9a4f0 identity security_compliance lockout_retries 5 None None
b14aa612-70b6-4a68-9b7e-795c7958433b identity security_compliance lockout_seconds 1800 None None
Alarms
NA
Test Activity
Regression Testing
Workaround
NA |
|
