Regression - Bootstrap replay following a cluster_host_subnet reconfiguration fails

[Tee Ngo]

Brief Description
-----------------
The replay of bootstrap playbook following a config change to cluster_host_subnet no longer works.

Severity
--------
Major

Steps to Reproduce
------------------
Step 1: Run bootstrap playbook following a fresh install:

ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml

Step 2: Update the cluster_host_subnet (set it to 192.168.207.0/24 if test is done in vbox) in localhost.yml and rerun the above command to replay bootstrap to apply the change.

Expected Behavior
------------------
Bootstrap replay completes successfully and the config change is applied

Actual Behavior
----------------
Bootstrap replay fails at the following task

TASK [bootstrap/bringup-essential-services : Create etcd client account for root, apiserver and enable etcd auth]***

“failed: [localhost] (item=auth enable) => {"changed": true, "cmd": ["etcdctl", "--cert-file=$ETCD_CERT", "--key-file=$ETCD_KEY", "--ca-file=$ETCD_CA", "--endpoint=$ETCD_ENDPOINT", "auth", "enable"], "delta": "0:00:00.015078", "end": "2021-03-11 21:26:43.833719", "item": "auth enable", "msg": "non-zero return code", "rc": 2, "start": "2021-03-11 21:26:43.818641", "stderr": "Error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.168.207.1:2379: connect: connection refused\n\nerror #0: dial tcp 192.168.207.1:2379: connect: connection refused", "stderr_lines": "Error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.168.207.1:2379: connect: connection refused", "", "error #0: dial tcp 192.168.207.1:2379: connect: connection refused", "stdout": "", "stdout_lines": []}”

This regression is caused by the upstream commit https://opendev.org/starlingx/ansible-playbooks/commit/820f347324d4fb7d17396d5d21f98eb2b674d23a to enable etcd security setting.

The following task is not executed upon replay hence the new cluster_host ip is not updated in etcd config file.

# Add etcd security info to static hieradata so that etcd is configured with security
  # when etc puppet manifest is applied before Kubernetes master is initialized in the later role.

name: Write security settings to static hieradata
  lineinfile:
    path: "{{ hieradata_workdir }}/static.yaml"
    line: "{{ item }}"
  with_items:
    - "platform::etcd::params::security_enabled: true"
    - "platform::etcd::params::bind_address: {{ cluster_floating_address }}"
    - "platform::etcd::params::bind_address_version: {{ etcd_listen_address_version }}"

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Any system type

Branch/Pull Time/Commit
-----------------------
March 10th, 2021 StarlingX load

Last Pass
---------
Before the etcd security feature was introduced

Timestamp/Logs
--------------
See above

Test Activity
-------------
Developer Testing

Workaround
----------

[Bart Wensley (bartwensley)]

Zhipeng - can you take a look at this? We will want a fix for stx.5.0. Please contact Tee Ngo if you need guidance on how to fix this.

[zhipeng liu (zhipengs)]

Hi Tee,

Yes, I saw below.
- { role: bootstrap/apply-bootstrap-manifest, when: not replayed, become: yes }

After run bootstrap first time, etcd puppet is applied with security and default cluster ip, and etcd config file created.
After we rerun bootstrap, we still need to apply etcd puppet again and let etcd config file changed to new specified cluster ip.
Some solutions below.
1) Is it possible to remove this “not replayed” above? Not sure which task in apply-bootstrap-manifest/tasks/main.yml could not rerun during rebootstrap
2) Or we move “not replayed” to apply-bootstrap-manifest/tasks/main.yml
Just let it write static hieradata and apply all puppet like below

- name: Write security settings to static hieradata
  lineinfile:
    path: "{{ hieradata_workdir }}/static.yaml"
    line: "{{ item }}"
  with_items:
    - "platform::etcd::params::security_enabled: true"
    - "platform::etcd::params::bind_address: {{ default_cluster_host_start_address }}"
    - "platform::etcd::params::bind_address_version: {{ etcd_listen_address_version }}"
- name: Applying puppet bootstrap manifest

Any further idea? Thanks!

Thanks!
Zhipeng

[zhipeng liu (zhipengs)]

https://review.opendev.org/c/starlingx/ansible-playbooks/+/781220
Patch submitted and verified by me.

Zhipeng

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/163ec9989cc7360dba4c572b2c43effd10306048
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 4e96b762f549aadb0291cc9bcf3352ae923e94eb
Author: Mihnea Saracin <email address hidden>
Date: Sat May 22 15:48:19 2021 +0000

    Revert "Restore host filesystems with collected sizes"

    This reverts commit 255488739efa4ac072424b19f2dbb7a3adb0254e.

    Reason for revert: Did a rework to fix https://bugs.launchpad.net/starlingx/+bug/1926591. The original problem was in puppet, and this fix in ansible was not good enough, it generated some other problems.

    Change-Id: Iea79701a874effecb7fe995ac468d50081d1a84f
    Depends-On: I55ae6954d24ba32e40c2e5e276ec17015d9bba44

commit c064aacc377c8bd5336ceab825d4bcbf5af0b5e8
Author: Angie Wang <email address hidden>
Date: Fri May 21 21:28:02 2021 -0400

    Ensure apiserver keys are present before extract from tarball

    This is to fix the upgrade playbook issue that happens during
    AIO-SX upgrade from stx4.0 to stx5.0 which introduced by
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/792093.
    The apiserver keys are not available in stx4.0 side so we need
    to ensure the keys under /etc/kubernetes/pki are present in the
    backed-up tarball before extracting, otherwise playbook fails
    because the keys are not found in the archive.

    Change-Id: I8602f07d1b1041a7fd3fff21e6f9a422b9784ab5
    Closes-Bug: 928925
    Signed-off-by: Angie Wang <email address hidden>

commit 0261f22ff7c23d2a8608fe3b51725c9f29931281
Author: Don Penney <email address hidden>
Date: Thu May 20 23:09:07 2021 -0400

    Update SX to DX migration to wait for coredns config

    This commit updates the SX to DX migration playbook to wait after
    modifying the system mode to duplex until the runtime manifest that
    updates coredns config has completed. The playbook will wait for up to
    20 minutes to allow for the possibilty that sysinv has multiple
    runtime manifests queued up, each of which could take several minutes.

    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/792494
    Depends-On: https://review.opendev.org/c/starlingx/config/+/792496
    Change-Id: I3bf94d3493ae20eeb16b3fdcb27576ee18c0dc4d
    Closes-Bug: 1929148
    Signed-off-by: Don Penney <email address hidden>

commit 7c4f17bd0d92fc1122823211e1c9787829d206a9
Author: Daniel Safta <email address hidden>
Date: Wed May 19 09:08:16 2021 +0000

    Fixed missing apiserver-etcd-client certs

    When controller-1 is the active controller
    the backup archive does not contain
    /etc/etcd/apiserver-etcd-client.{crt, key}

    This change adds a new task which brings
    the certs from /etc/kubernetes/pki

    Closes-bug: 1928925
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: I3c68377603e1af9a71d104e5b1108e9582497a09

commit e221ef8fbe51aa6ca229b584fb5632fe512ad5cb
Author: David Sullivan <email address hidden>
Date: Wed May 19 16:01:27 2021 -0500

    Support boo...