DC system controller admin endpoint is http, not https by default

[Peng Peng]

Brief Description
-----------------
In distributed cloud, system controller admin endpoint by default should be https, but it shows http

Severity
--------
Major

Steps to Reproduce
------------------
install DC system
check system controller admin endpoint

Expected Behavior
------------------
system controller admin endpoint should be https

Actual Behavior
----------------
system controller admin endpoint is http

Reproducibility
---------------
Reproducible
Intermittent
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
DC system

Lab-name: WCP_80-91

Branch/Pull Time/Commit
-----------------------
"2020-05-05_20-29-49"

Last Pass
---------
"2020-05-01_20-00-00"

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ openstack endpoint list | grep admin
| 76eb600b2e47401ebceb8356fbfe576a | RegionOne | fm | faultmanagement | True | admin | http://[fd01:1::2]:18002 |
| faf82ecb07c64cbeb747f47093a5ca0a | SystemController | dcmanager | dcmanager | True | admin | http://[fd01:1::2]:8119/v1.0 |
| 33696c92baa14ad8a58ce938670d74d4 | SystemController | patching | patching | True | admin | http://[fd01:1::2]:25491/ |
| be493cbd83c2498b9ca3ebef4092915d | RegionOne | patching | patching | True | admin | http://[fd01:1::2]:5491 |
| 97f45f5d58024973bad8c2d150d8cb86 | RegionOne | vim | nfv | True | admin | http://[fd01:1::2]:4545 |
| 82d9f1e227f1467ca11fb7b43cf671bc | RegionOne | barbican | key-manager | True | admin | http://[fd01:1::2]:9311 |
| 823d13e518864e8d95ae5947ab46b8a4 | RegionOne | smapi | smapi | True | admin | http://[fd01:1::2]:7777 |
| 9b517630ac714b3fb366590da05ff297 | RegionOne | keystone | identity | True | admin | http://[fd01:1::2]:5000/v3 |
| 8b2a5729fe1f497990f6dded31e445ec | SystemController | keystone | identity | True | admin | http://[fd01:1::2]:25000/v3 |
| e717b31f29624e9c84a1bf5e9a964ebc | RegionOne | sysinv | platform | True | admin | http://[fd01:1::2]:6385/v1 |
| 830b2e670e744be1a44674b44030168d | SystemController | sysinv | platform | True | admin | http://[fd01:1::2]:26385/v1 |
| fd4a00c96fa34733b35004e12b8d151a | RegionOne | dcdbsync | dcorch-dbsync | True | admin | http://[fd01:1::2]:8219/v1.0 |

Test Activity
-------------
Sanity

[Bin Qian (bqian20)]

the hieradata has the right admin_url, something after went wrong:
/opt/platform/puppet/20.04/hieradata/system.yaml:barbican::keystone::auth::admin_url: https://[fd01:1::2]:9312
/opt/platform/puppet/20.04/hieradata/system.yaml:dcdbsync::keystone::auth::admin_url: https://[fd01:1::2]:8220/v1.0
/opt/platform/puppet/20.04/hieradata/system.yaml:dcmanager::keystone::auth::admin_url: https://[fd01:1::2]:8120/v1.0
/opt/platform/puppet/20.04/hieradata/system.yaml:dcorch::keystone::auth::identity_proxy_admin_url: https://[fd01:1::2]:25001/v3
/opt/platform/puppet/20.04/hieradata/system.yaml:dcorch::keystone::auth::patching_proxy_admin_url: https://[fd01:1::2]:25492/
/opt/platform/puppet/20.04/hieradata/system.yaml:dcorch::keystone::auth::sysinv_proxy_admin_url: https://[fd01:1::2]:26386/v1
/opt/platform/puppet/20.04/hieradata/system.yaml:fm::keystone::auth::admin_url: https://[fd01:1::2]:18003
/opt/platform/puppet/20.04/hieradata/system.yaml:keystone::endpoint::admin_url: https://[fd01:1::2]:5001
/opt/platform/puppet/20.04/hieradata/system.yaml:nfv::keystone::auth::admin_url: https://[fd01:1::2]:4546
/opt/platform/puppet/20.04/hieradata/system.yaml:patching::keystone::auth::admin_url: https://[fd01:1::2]:5492
/opt/platform/puppet/20.04/hieradata/system.yaml:platform::smapi::params::admin_url: https://[fd01:1::2]:7778
/opt/platform/puppet/20.04/hieradata/system.yaml:smapi::keystone::auth::admin_url: https://[fd01:1::2]:7778
/opt/platform/puppet/20.04/hieradata/system.yaml:sysinv::keystone::auth::admin_url: https://[fd01:1::2]:6386/v1
Also /etc/haproxy/haproxy.cfg is configured correctly with admin endpoints as https
frontend barbican-restapi-admin
  bind fd01:1::2:9312 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend barbican-restapi-admin-internal
--
frontend dcdbsync-restapi-admin
  bind fd01:1::2:8220 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend dcdbsync-restapi-admin-internal
--
frontend dcmanager-restapi-admin
  bind fd01:1::2:8120 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend dcmanager-restapi-admin-internal
--
frontend dcorch-identity-api-proxy-admin
  bind fd01:1::2:25001 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend dcorch-identity-api-proxy-admin-internal
--
frontend dcorch-patch-api-proxy-admin
  bind fd01:1::2:25492 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend dcorch-patch-api-proxy-admin-internal
--
frontend dcorch-sysinv-api-proxy-admin
  bind fd01:1::2:26386 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend dcorch-sysinv-api-proxy-admin-internal
--
frontend fm-api-admin
  bind fd01:1::2:18003 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend fm-api-admin-internal
--
frontend keystone-restapi-admin
  bind fd01:1::2:5001 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend keystone-restapi-admin-internal
--
frontend patching-restapi-admin
  bind fd01:1::2:5492 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend patching-restapi-admin-internal
--
frontend sm-api-admin
  bind fd01:1::2:7778 ssl crt /etc/ssl/private/admin-ep-cert.pem
  default_backend sm-api-admin-internal
--
frontend sysinv-restapi-admin
  bind fd01:1::2:6386 ssl c...

[Peng Peng (ppeng)]

collect log failed by LP-1877142

tar /var/log/ logs and added at
https://files.starlingx.kube.cengn.ca/launchpad/1877138

[Yang Liu (yliu12)]

Was working on 2020-05-01-20-00-00 load:

[2020-05-04 14:05:33,218] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-interface internal --os-region-name SystemController endpoint list'
+----------------------------------+------------------+--------------+-----------------+---------+-----------+---------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+------------------+--------------+-----------------+---------+-----------+---------------------------------------------+
| a40e63f4919e421b80ecb1774190cfd2 | SystemController | dcmanager | dcmanager | True | admin | https://[fd01:11::2]:8120/v1.0 |

[Ghada Khalil (gkhalil)]

This issue appears to be introduced/exposed by a recent code change:
https://review.opendev.org/#/c/724741/

[Bin Qian (bqian20)]

As a side note, the admin endpoint are reconfigured to https after controller-0 1st unlock. This is triggered in sysinv conductor _controller_config_active_apply.

[Ghada Khalil (gkhalil)]

The commit that introduced this issue has been reverted:
https://review.opendev.org/#/c/725950/

Merged on 2020-05-06

[Peng Peng (ppeng)]

Verified on
BUILD_ID="2020-05-06_19-45-22"

[sysadmin@controller-0 ~(keystone_admin)]$ system show | grep https
| https_enabled | False |
[sysadmin@controller-0 ~(keystone_admin)]$ openstack endpoint list | grep admin
| fbf8c69e1ecd40cc8a61fb76f456c919 | SystemController | dcmanager | dcmanager | True | admin | https://[fd01:1::2]:8120/v1.0 |
| 67fe21704b244fffb0397a692c74b018 | subcloud6 | sysinv | platform | True | admin | https://[fd01:15::2]:6386/v1 |
| 20a079b7895e46828a0ef12a8bfb5d8b | subcloud6 | patching | patching | True | admin | https://[fd01:15::2]:5492 |
| 3c64365f3f524ce6aaa31e27d813e6d7 | subcloud6 | vim | nfv | True | admin | https://[fd01:15::2]:4546 |
| c2361f996b604ff69661434df9e5a4e4 | RegionOne | fm | faultmanagement | True | admin | https://[fd01:1::2]:18003 |
| d7bcfec52fbe47329809ed7766073293 | SystemController | patching | patching | True | admin | https://[fd01:1::2]:25492/ |