BnR AIO-SX omits etcd users and auth enabled configuration

[Michel Thebeau [WIND]]

Brief Description
-----------------
Security enabled etcd was added in R5. After restoring the AIO-SX platform from backup, the etcd configuration omits users and auth is not enabled.

Severity
--------
Minor: System/Feature is usable with minor issue

Steps to Reproduce
------------------
1) Complete commissioning of AIO-SX from ISO. Verify user list and auth is enabled using the following method:

sudo bash
  port=2379
  bind_address="192.168.206.1"
  client_url="https://${bind_address}:${port},https://[127.0.0.1]:${port}"
  etcd_cert="/etc/etcd/etcd-client.crt"
  etcd_key="/etc/etcd/etcd-client.key"
  etcd_ca="/etc/etcd/ca.crt"

  etcdctl \
    --cert-file=${etcd_cert} \
    --key-file=${etcd_key} \
    --ca-file=${etcd_ca} \
    --endpoint=${client_url} \
    user list

# Output:
# apiserver-etcd-client
# root

  etcdctl \
    --cert-file=${etcd_cert} \
    --key-file=${etcd_key} \
    --ca-file=${etcd_ca} \
    --endpoint=${client_url} \
    -u root role get guest

# Press enter (no password)
# Output:
# Insufficient credentials

2) Perform Backup procedure (https://docs.starlingx.io/backup/kubernetes/running-ansible-backup-playbook-locally-on-the-controller.html#running-ansible-backup-playbook-locally-on-the-controller)

3) Poweroff the system and install the ISO again

4) Complete the restore operation (https://docs.starlingx.io/backup/kubernetes/restoring-starlingx-system-data-and-storage.html)

5) Reassert the etcd user list and auth enabled configuration per the method above.
# No users are listed for by the "user list" command
# And instead of "Insufficient credentials", the second command "role get guest" requiring authentication provides "auth: Role guest does not exist."

Expected Behavior
------------------
After restore of AIO-SX, etcd configuration includes users and auth is enabled.

Actual Behavior
----------------
After restore of AIO-SX, etcd configuration omits users and auth enabled.

Reproducibility
---------------
100% reproducible, AIO-SX

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
Starlingx master, October 27, 2021 (Cengn Stx monolithic build iso 20211027T043000Z)

Last Pass
---------
N/A

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Developer Testing

Workaround
----------
Manually restore users and auth enable:

sudo bash

  port=2379
  bind_address="192.168.206.1"
  client_url="https://${bind_address}:${port},https://[127.0.0.1]:${port}"
  etcd_cert="/etc/etcd/etcd-client.crt"
  etcd_key="/etc/etcd/etcd-client.key"
  etcd_ca="/etc/etcd/ca.crt"

  cmds="user add root:sysadmin
  user add apiserver-etcd-client:sysadmin
  auth enable"

  while read cmd; do
    etcdctl \
      --cert-file=${etcd_cert} \
      --key-file=${etcd_key} \
      --ca-file=${etcd_ca} \
      --endpoint=${client_url} \
      $cmd
  done <<<"$cmds"

# Output:
# User root created
# User apiserver-etcd-client created
# Authentication Enabled

[Michel Thebeau [WIND] (mthebeau)]

It seems to be that the omission of users/roles and auth enable over BnR is due to their being two data-stores - one each for the v2 and v3 API of etcdctl. The etcdctl user/role creation and "auth enable" is run against etcdctl v2 API, while the backup/restore is run using the etcdctl v3 API.  An external site explains that a different data-store is used when executing etcdctl using either v2 or v3 APIs.  So, we create users/roles and enable auth within the data-store behind v2 API, but we backup and restore the data-store behind the v3 API.

"An etcd 3.x server can understand both version 2 and version 3 APIs but, and it's a huge but, anything you create with clients using one API version will be invisible to clients using the other API version. That's because around the back end, each API routes to a separate data store - they are so different that they are isolated from each other inside the server."  - https://www.compose.com/articles/etcd2to3-new-apis-and-new-possibilities/

Kubernetes keys are stored in the data-store accessed through v3 API for example.

A BnR test with user/role added using v3 API confirms the explanation.

[Ghada Khalil (gkhalil)]

Scope of this change is much larger than a bug. There is no plan to address this gap in stx.6.0. We'll consider it for stx.7.0 and will likely need to open a story to track.

[Ghada Khalil (gkhalil)]

Currently there is no plan to pursue this further in the stx.7.0 release timeframe; removing the release tag.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/826661

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/826665

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/826667

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/826665
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/a68bfff483b48b873a7a9a0f118a8815a48d1adb
Submitter: "Zuul (22348)"
Branch: master

commit a68bfff483b48b873a7a9a0f118a8815a48d1adb
Author: Michel Thebeau <email address hidden>
Date: Tue Jan 18 16:59:02 2022 -0500

    Remove Etcd RBAC enabling code against the V2 API backend

    The use of Etcd API V2 is not supported. The kubernetes resources
    intended to be protected by Etcd RBAC are stored in the data-store
    backing the Etcd V3 API. These backend data-stores are independent.
    RBAC enabled through V2 API does not affect the data-store backing V3
    API.

    Clean up the RBAC enabling code against V2 API.

    Test plan:
    AIO-SX:
        bootstrap, confirm omitted RBAC configration: PASS
        BnR, confirm omitted RBAC configration: PASS
        upgrade, confirm omitted RBAC configuration: PASS

    AIO-DX:
        upgrade, confirm removed RBAC configuration: PASS

    Partial-Bug: 1949219

    Change-Id: I6d1c2fef41f212f0cb81639615ade4f0f4ccea4b
    Signed-off-by: Michel Thebeau <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/826661
Committed: https://opendev.org/starlingx/stx-puppet/commit/faa8c443a2313d9b5d439d4849c6dc3a8d845169
Submitter: "Zuul (22348)"
Branch: master

commit faa8c443a2313d9b5d439d4849c6dc3a8d845169
Author: Michel Thebeau <email address hidden>
Date: Tue Jan 18 17:05:35 2022 -0500

    Remove Etcd RBAC enabling code against the V2 API backend

    The use of Etcd API V2 is not supported. The kubernetes resources
    intended to be protected by Etcd RBAC are stored in the data-store
    backing the Etcd V3 API. These backend data-stores are independent.
    RBAC enabled through V2 API does not affect the data-store backing V3
    API.

    Clean up the RBAC enabling code against V2 API.

    Test plan:
    AIO-SX:
        bootstrap, confirm omitted RBAC configration: PASS
        BnR, confirm omitted RBAC configration: PASS
        upgrade, confirm omitted RBAC configuration: PASS

    AIO-DX:
        upgrade, confirm removed RBAC configuration: PASS

    Partial-Bug: 1949219

    Change-Id: I8c18b4f5e6247d2c23e068b24eab390c7a3fab5f
    Signed-off-by: Michel Thebeau <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/826667
Committed: https://opendev.org/starlingx/config/commit/49117ccaa7e1de8d8376bc0fbdb479ef95a362d0
Submitter: "Zuul (22348)"
Branch: master

commit 49117ccaa7e1de8d8376bc0fbdb479ef95a362d0
Author: Michel Thebeau <email address hidden>
Date: Wed Jan 26 15:59:54 2022 -0500

    Remove Etcd RBAC enabling code against the V2 API backend

    The use of Etcd API V2 is not supported. The kubernetes resources
    intended to be protected by Etcd RBAC are stored in the data-store
    backing the Etcd V3 API. These backend data-stores are independent.
    RBAC enabled through V2 API does not affect the data-store backing V3
    API.

    Remove the Etcd RBAC configuration against V2 API during
    upgrade-activate. The active controller commissioned with 6.0 release
    will have Etcd auth enabled and users root and apiserver-etcd-client
    created.

    Test plan:
    AIO-SX:
        bootstrap, confirm omitted RBAC configration: PASS
        BnR, confirm omitted RBAC configration: PASS
        upgrade, confirm omitted RBAC configuration: PASS

    AIO-DX:
        upgrade, confirm removed RBAC configuration: PASS

    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/826661
    Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/826665

    Partial-Bug: 1949219

    Change-Id: I12bbf4cd76cbda036fa4784c6d1cc2eefd39e1b1
    Signed-off-by: Michel Thebeau <email address hidden>