Comment 0 for bug 1992384

Revision history for this message
Li Zhu (lzhu1) wrote :

Brief Description
-----------------
During the operation of DC rehoming, in the phase of migrate_keystone_password, sometimes
the "Update docker registry credentials" task would fail due to not getting openstack secret list
or service parameter list.

Severity
--------
Major: Rehoming would fail due to this issue.

Steps to Reproduce
------------------
1. Comment out the following four tasks in the migrate_keystone_passwords.yml ansible playbook.
- name: Verify barbican services
  ...

- block:
  - name: Restart barbican services
    ...

  - name: "Check barbican-keystone-listener service status"
    ...

  when: barbican_check_result.rc != 0

- name: Wait system service-parameter-list available
  ...
2. Run DC rehoming.

Expected Behavior
------------------
Rehoming succeeds

Actual Behavior
----------------
Sometimes failed in the "Update docker registry credentials" task.

Reproducibility
---------------
60% reproducible

System Configuration
--------------------
Two DCs with at least one subcloud that can be used for rehoming.

Branch/Pull Time/Commit
-----------------------
22.12

Last Pass
---------
This is a new test scenario.

Timestamp/Logs
--------------
1.TASK [rehome-subcloud/update-keystone-data : Update docker registry credentials] ***
Wednesday 28 September 2022 00:20:55 +0000 (0:00:00.390) 0:08:22.566 ***
fatal: [subcloud12]: FAILED! => changed=true
  cmd:
  - update_docker_registry_auth.sh
  - sysinv
  - LYq_h0=H22jrsuDj
  delta: '0:00:28.110285'
  end: '2022-09-28 00:21:23.525941'
  msg: non-zero return code
  rc: 1
  start: '2022-09-28 00:20:55.415656'
  stderr: |-
    'errors'
    'errors'
    'errors'
    'errors'
    'errors'
    'errors'
    'errors'
  stderr_lines:
  - '''errors'''
  - '''errors'''
  - '''errors'''
  - '''errors'''
  - '''errors'''
  - '''errors'''
  - '''errors'''
  stdout: |2-

    Updating docker-registry credentials .No docker-registry entry in service-parameters

    Updating quay-registry credentials .No quay-registry entry in service-parameters

    Updating elastic-registry credentials .No elastic-registry entry in service-parameters

    Updating gcr-registry credentials .No gcr-registry entry in service-parameters

    Updating k8s-registry credentials .No k8s-registry entry in service-parameters

    Updating ghcr-registry credentials .No ghcr-registry entry in service-parameters
  stdout_lines: <omitted>

2.Failed in running “openstack secret list” command on the subcloud.

Test Activity
-------------
Developer Testing

Workaround
----------
Added four tasks in the migrate_keystone_passwords.yml ansible playbook before
the "Update docker registry credentials" task to restart barbican services and
wait until "system service-parameter-list" is available.