Cert migration playbook fails when target_list=all_online_subclouds, someother

[Reinildes Oliveira]

Brief Description
-------------------------------
platform cert migration playbook fails when run the playbook with parameter "target_list" with two values where one of them is "all_online_subclouds". ie: "target_list=localhost,all_online_subclouds".

When target_list param is used only with "all_online_subclouds" value the playbook change the certificates for localhost and all the subclouds.

Generally it is possible to run target_list with two or more parameters i.e "target_list=localhost,subcloud1,subcloud2..subcloud1000" then use "target_list=localhost,all_online_subclouds" could be used by the customer, in that case the playbook shouldn't broke instead it could apply the playbook for localhost and all the subclouds.

Severity
-----------------------------------

<Minor: System/Feature is usable with minor issue>

Stes to reproduce
--------------------------------

create an inventory file as follows:

all:
  vars:
    ica_cert: base64
    ica_key: base64
  children:
    target_group:
      vars:
        dns_domain: xyz.com
        duration: 3650h # 90d
        renewBefore: 460h # 15d
        subject_C: Canada
        subject_ST: Ontario
        subject_L: Ottawa
        subject_O: pvtest
        subject_OU: engineering
        subject_CN: pvtest.wrs.com
        subject_prefix: pvtest
        # SSH password to connect to all subclouds
        ansible_ssh_user: sysadmin
        ansible_ssh_pass: pwd
        # Sudo password
        ansible_become_pass: pwd

Now run the playbook:

ansible-playbook /usr/share/ansible/stx-ansible/playbooks/migrate-platform-certificates-to-certmanager.yml -i migration-inventory.yml --extra-vars "target_list={*}localhost{*}, *all_online_subclouds* mode=update" --ask-vault-pass

Output:

PLAY [localhost] **************************************************************************************************************************************************************************************************
TASK [Fail if target_list is not defined] *************************************************************************************************************************************************************************
Thursday 27 January 2022 19:19:33 +0000 (0:00:00.072) 0:00:00.072 ******
skipping: [localhost]
TASK [Get online subclouds from dcmanager] ************************************************************************************************************************************************************************
Thursday 27 January 2022 19:19:33 +0000 (0:00:00.018) 0:00:00.090 ******
skipping: [localhost]
TASK [Add host to target_group] ***********************************************************************************************************************************************************************************
Thursday 27 January 2022 19:19:33 +0000 (0:00:00.015) 0:00:00.106 ******
skipping: [localhost]
TASK [Get subcloud from extra-vars] *******************************************************************************************************************************************************************************
Thursday 27 January 2022 19:19:33 +0000 (0:00:00.015) 0:00:00.121 ******
changed: [localhost] => (item=localhost)
changed: [localhost] => (item=all_online_subclouds)
 [WARNING]: A duplicate localhost-like entry was found (localhost). First found localhost was localhost

TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Save the specified ICA to a file] *********************************************************************************************************
Thursday 27 January 2022 19:19:33 +0000 (0:00:00.035) 0:00:00.157 ******
changed: [localhost]
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Get CA information from certificate] ******************************************************************************************************
Thursday 27 January 2022 19:19:34 +0000 (0:00:00.405) 0:00:00.562 ******
changed: [localhost]
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Fail when ICA certificate is not an actual CA certificate] ********************************************************************************
Thursday 27 January 2022 19:19:34 +0000 (0:00:00.227) 0:00:00.790 ******
skipping: [localhost]
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Get years for ICA duration validation] ****************************************************************************************************
Thursday 27 January 2022 19:19:34 +0000 (0:00:00.019) 0:00:00.809 ******
ok: [localhost]
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Check that ICA certificate remaining duration is longer than 3 years] *********************************************************************
Thursday 27 January 2022 19:19:34 +0000 (0:00:00.059) 0:00:00.869 ******
changed: [localhost]
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Fail when ICA certificate remaining duration is shorter than 3 years] *********************************************************************
Thursday 27 January 2022 19:19:34 +0000 (0:00:00.225) 0:00:01.094 ******
skipping: [localhost]
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Install ICA] ******************************************************************************************************************************
Thursday 27 January 2022 19:19:34 +0000 (0:00:00.018) 0:00:01.112 ******
changed: [localhost]
PLAY [target_group] ***********************************************************************************************************************************************************************************************
Thursday 27 January 2022 19:19:51 +0000 (0:00:17.330) 0:00:18.443 ******
Thursday 27 January 2022 19:19:51 +0000 (0:00:00.007) 0:00:18.451 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Check for management affecting alarms] **************************************************************************************************
skipping: [localhost]
Thursday 27 January 2022 19:19:51 +0000 (0:00:00.012) 0:00:18.463 ******
skipping: [all_online_subclouds]
Thursday 27 January 2022 19:19:52 +0000 (0:00:00.008) 0:00:18.472 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Get address pool information for system] ************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:19:53 +0000 (0:00:01.937) 0:00:20.409 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Get floating management ip] *************************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:19:54 +0000 (0:00:00.192) 0:00:20.602 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Get floating oam ip] ********************************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:19:54 +0000 (0:00:00.188) 0:00:20.790 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Get region name] ************************************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:19:56 +0000 (0:00:01.886) 0:00:22.677 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Get distributed_cloud role] *************************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:19:58 +0000 (0:00:01.908) 0:00:24.586 ******
included: /usr/share/ansible/stx-ansible/playbooks/roles/migrate-platform-certificates-to-certmanager/migrate-certificates/tasks/check-certificates-to-be-installed.yml for localhost
Thursday 27 January 2022 19:19:58 +0000 (0:00:00.028) 0:00:24.614 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Check if system is https_enabled] *******************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:00 +0000 (0:00:02.207) 0:00:26.822 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Check if oidc-auth-apps is applied] *****************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:02 +0000 (0:00:01.876) 0:00:28.698 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Make sure /home/sysadmin/certificates_backup/ directory exists] *************************************************************************
ok: [localhost]
Thursday 27 January 2022 19:20:02 +0000 (0:00:00.323) 0:00:29.022 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Backup current registry.local and rest-api-https certificates] **************************************************************************
changed: [localhost] => (item=registry-cert.crt)
changed: [localhost] => (item=server-cert.pem)
Thursday 27 January 2022 19:20:03 +0000 (0:00:00.557) 0:00:29.580 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Backup current local-dex.tls secret] ****************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:03 +0000 (0:00:00.283) 0:00:29.864 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Generate kubernetes yaml for cert-manager resources] ************************************************************************************
ok: [localhost]
Thursday 27 January 2022 19:20:03 +0000 (0:00:00.440) 0:00:30.305 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Apply kubernetes yaml to create cert-manager clusterissuer and certificates] ************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:04 +0000 (0:00:00.542) 0:00:30.847 ******
included: /usr/share/ansible/stx-ansible/playbooks/roles/migrate-platform-certificates-to-certmanager/migrate-certificates/tasks/delete-kubernetes-objects.yml for localhost => (item={u'type': u'secret', u'secret': u'system-registry-local-certificate', u'namespace': u'deployment'})
included: /usr/share/ansible/stx-ansible/playbooks/roles/migrate-platform-certificates-to-certmanager/migrate-certificates/tasks/delete-kubernetes-objects.yml for localhost => (item={u'type': u'secret', u'secret': u'system-restapi-gui-certificate', u'namespace': u'deployment'})
included: /usr/share/ansible/stx-ansible/playbooks/roles/migrate-platform-certificates-to-certmanager/migrate-certificates/tasks/delete-kubernetes-objects.yml for localhost => (item={u'type': u'secret', u'secret': u'oidc-auth-apps-certificate', u'namespace': u'kube-system'})
Thursday 27 January 2022 19:20:04 +0000 (0:00:00.051) 0:00:30.899 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Delete kubernetes objects] **************************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:04 +0000 (0:00:00.262) 0:00:31.161 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Delete kubernetes objects] **************************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:04 +0000 (0:00:00.267) 0:00:31.428 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Delete kubernetes objects] **************************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:05 +0000 (0:00:00.264) 0:00:31.692 ******
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Save the specified ICA to a file] *********************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:05 +0000 (0:00:00.194) 0:00:31.887 ******
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Get CA information from certificate] ******************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:05 +0000 (0:00:00.196) 0:00:32.083 ******
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Fail when ICA certificate is not an actual CA certificate] ********************************************************************************
skipping: [localhost]
Thursday 27 January 2022 19:20:05 +0000 (0:00:00.019) 0:00:32.103 ******
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Get years for ICA duration validation] ****************************************************************************************************
ok: [localhost]
Thursday 27 January 2022 19:20:05 +0000 (0:00:00.038) 0:00:32.141 ******
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Check that ICA certificate remaining duration is longer than 3 years] *********************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:05 +0000 (0:00:00.202) 0:00:32.343 ******
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Fail when ICA certificate remaining duration is shorter than 3 years] *********************************************************************
skipping: [localhost]
Thursday 27 January 2022 19:20:05 +0000 (0:00:00.019) 0:00:32.363 ******
TASK [migrate-platform-certificates-to-certmanager/install-trusted-ca : Install ICA] ******************************************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:56 +0000 (0:00:50.701) 0:01:23.064 ******
included: /usr/share/ansible/stx-ansible/playbooks/roles/migrate-platform-certificates-to-certmanager/migrate-certificates/tasks/reapply-oidc-auth-app.yml for localhost
Thursday 27 January 2022 19:20:56 +0000 (0:00:00.041) 0:01:23.105 ******
included: /usr/share/ansible/stx-ansible/playbooks/roles/migrate-platform-certificates-to-certmanager/migrate-certificates/tasks/delete-kubernetes-objects.yml for localhost => (item={u'type': u'secret', u'secret': u'system-local-ca-oidc-secret', u'namespace': u'kube-system'})
Thursday 27 January 2022 19:20:56 +0000 (0:00:00.029) 0:01:23.134 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Delete kubernetes objects] **************************************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:56 +0000 (0:00:00.263) 0:01:23.398 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Create new dex-client secret based of system-local-ca] **********************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:20:57 +0000 (0:00:00.315) 0:01:23.713 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Create override file for dex-client-secret] *********************************************************************************************
ok: [localhost]
Thursday 27 January 2022 19:20:57 +0000 (0:00:00.315) 0:01:24.029 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Create override file for oidc-auth-apps-certificate] ************************************************************************************
ok: [localhost]
Thursday 27 January 2022 19:20:57 +0000 (0:00:00.317) 0:01:24.346 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Create override file for secret-observer helm chart] ************************************************************************************
ok: [localhost]
Thursday 27 January 2022 19:20:58 +0000 (0:00:00.319) 0:01:24.666 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Perform helm-override-updates to use new certificates] **********************************************************************************
changed: [localhost] => (item={u'overrides_file': u'/tmp/dex-client-secret.yaml', u'chart': u'oidc-client'})
changed: [localhost] => (item={u'overrides_file': u'/tmp/oidc-auth-apps-certificate-override.yaml', u'chart': u'dex'})
changed: [localhost] => (item={u'overrides_file': u'/tmp/secret-observer.yaml', u'chart': u'secret-observer'})
Thursday 27 January 2022 19:21:04 +0000 (0:00:06.495) 0:01:31.162 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Apply oidc-auth-apps so that it picks up new certificates] ******************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:21:06 +0000 (0:00:01.935) 0:01:33.098 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Rollout oidc-auth-apps pods so that they start using new certificates] ******************************************************************
changed: [localhost]
Thursday 27 January 2022 19:21:06 +0000 (0:00:00.279) 0:01:33.377 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Wait for oidc-auth-apps pods to become active] ******************************************************************************************
changed: [localhost]
Thursday 27 January 2022 19:21:18 +0000 (0:00:11.989) 0:01:45.367 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Delete clusterissuer] *******************************************************************************************************************
skipping: [localhost]
Thursday 27 January 2022 19:21:18 +0000 (0:00:00.018) 0:01:45.385 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Delete certificates] ********************************************************************************************************************
skipping: [localhost] => (item={u'namespace': u'deployment', u'secret': u'system-registry-local-certificate', u'type': u'certificate'})
skipping: [localhost] => (item={u'namespace': u'deployment', u'secret': u'system-restapi-gui-certificate', u'type': u'certificate'})
Thursday 27 January 2022 19:21:18 +0000 (0:00:00.027) 0:01:45.413 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Delete secrets] *************************************************************************************************************************
skipping: [localhost] => (item={u'namespace': u'deployment', u'secret': u'system-registry-local-certificate', u'type': u'secret'})
skipping: [localhost] => (item={u'namespace': u'deployment', u'secret': u'system-restapi-gui-certificate', u'type': u'secret'})
Thursday 27 January 2022 19:21:18 +0000 (0:00:00.026) 0:01:45.439 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : debug] **********************************************************************************************************************************
skipping: [localhost]
Thursday 27 January 2022 19:21:18 +0000 (0:00:00.016) 0:01:45.456 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Check certificates to be installed] *****************************************************************************************************
skipping: [localhost]
Thursday 27 January 2022 19:21:19 +0000 (0:00:00.016) 0:01:45.473 ******
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Get summary of certificates installed] **************************************************************************************************
skipping: [localhost] => (item={u'should_run': u'true', u'secret': u'system-registry-local-certificate', u'namespace': u'deployment'})
skipping: [localhost] => (item={u'should_run': True, u'secret': u'system-restapi-gui-certificate', u'namespace': u'deployment'})
skipping: [localhost] => (item={u'should_run': u'true', u'secret': u'oidc-auth-apps-certificate', u'namespace': u'kube-system'})
TASK [migrate-platform-certificates-to-certmanager/migrate-certificates : Get address pool information for system] ************************************************************************************************
fatal: [all_online_subclouds]: UNREACHABLE! => changed=false
  msg: |-
    Failed to connect to the host via ssh: ssh: Could not resolve hostname all_online_subclouds: Name or service not known
  unreachable: true
PLAY RECAP ********************************************************************************************************************************************************************************************************
all_online_subclouds : ok=0 changed=0 unreachable=1 failed=0
localhost : ok=41 changed=28 unreachable=0 failed=0
{code}

Expected Behavior
-------------------------------------

playbook should work fine without any errors

Actual Behavior
-------------------------------------

playbook fails

Reproducibility
-------------------------------------

 100% reproducible

System Configuration
-------------------------------------

IPv6

*+Branch/Pull Time/Commit+*

BUILD_ID="2022-01-20_20-25-15"

Last Pass
-------------------------------------

new feature testing

Test Activity
-------------------------------------

Feature testing

Workaround
-------------------------------------

Run the playbook two times: the first one assing to "target_list" parameter the value "localhost" and after it run the playbook with the second value: "all_online_subclouds mode".

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/827187
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/106a8c7f27b6eae03503790e25626fca3855c687
Submitter: "Zuul (22348)"
Branch: master

commit 106a8c7f27b6eae03503790e25626fca3855c687
Author: Rei Oliveira <email address hidden>
Date: Mon Jan 31 18:22:26 2022 -0300

    Error with target_list=all_online_subclouds,xyz

    Playbook fails when parameter "target_list" has two values where one
    of them is "all_online_subclouds". ie: target_list=localhost,
    all_online_subclouds

    Test Plan:
    PASS: Test playbook with 'target_list=localhost,all_online_subclouds'
          and verify it targets all online subclouds in that distributed
          system and the system controller
    PASS: Verify that it's possible to target multiple subclouds with
          'target_list=subcloud1,subcloud2'

    Closes-Bug: 1959846
    Change-Id: I55fdfc65a066bc7ec740c1f88b74e8606fc1cef3
    Signed-off-by: Rei Oliveira <email address hidden>