system https_enabled is True, but admin/public endpoint are http

[Peng Peng]

Brief Description
-----------------
System with https_enabled setting is True, both admin and public endpoint shows http. But Horizon could be accessed from https url.

Severity
--------
Major

Steps to Reproduce
------------------
install SX with https_enabled setting is True
check admin and public endpoint

TC-name: horizon/test_hosts.py::test_horizon_host_inventory_display

Expected Behavior
------------------
both admin and public endpoints should be aligned with https_enabled setting

Actual Behavior
----------------
both admin and public endpoints not aligned with https_enabled setting

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
One node system

Lab-name: wcp_112, wcp99-103

Branch/Pull Time/Commit
-----------------------
2020-05-05_20-29-49

Last Pass
---------
 2020-05-01_20-00-00

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ system show
...
| https_enabled | True |...

[sysadmin@controller-0 ~(keystone_admin)]$ openstack endpoint list
+----------------------------------+-----------+--------------+-----------------+---------+-----------+------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+-----------------+---------+-----------+------------------------------------------+
| 42eeae852dc94e2db08eccade75e07f3 | RegionOne | fm | faultmanagement | True | admin | http://[abcd:204::1]:18002 |
| b233f09809364354bba77fa4dd77d75a | RegionOne | fm | faultmanagement | True | internal | http://[abcd:204::1]:18002 |
| ea898dcd27284574b5f3fa548524b203 | RegionOne | fm | faultmanagement | True | public | http://[2620:10a:a001:a103::148]:18002 |
| 5ad2dd7c338e4355a619070f88fc30e6 | RegionOne | patching | patching | True | admin | http://[abcd:204::1]:5491 |
| 6123facad77e43efb57f410346e799ad | RegionOne | patching | patching | True | internal | http://[abcd:204::1]:5491 |
| 8cf0b70b00df4d7397d3290a00b30b48 | RegionOne | patching | patching | True | public | http://[2620:10a:a001:a103::148]:15491 |
| abe2e31d701443078109899b351834a5 | RegionOne | vim | nfv | True | admin | http://[abcd:204::1]:4545 |
| cb0b4debe6a747138f8cd688c1f8315d | RegionOne | vim | nfv | True | internal | http://[abcd:204::1]:4545 |
| 005d615554e740fb91a10a2608933eb0 | RegionOne | vim | nfv | True | public | http://[2620:10a:a001:a103::148]:4545 |
| 57a4395d0e7e4fc58ed373b9ac19fd65 | RegionOne | barbican | key-manager | True | admin | http://[abcd:204::1]:9311 |
| 1b4fe2a21f7341e890a2f951d0b4c30d | RegionOne | barbican | key-manager | True | internal | http://[abcd:204::1]:9311 |
| 0928a8de15a14186997abad8a6673af0 | RegionOne | barbican | key-manager | True | public | http://[2620:10a:a001:a103::148]:9311 |
| b08cf88849164a73b0cc2220db3c3c5e | RegionOne | smapi | smapi | True | admin | http://[abcd:204::1]:7777 |
| cb03302d4d5d40079fcf2811a1c2b1f6 | RegionOne | smapi | smapi | True | internal | http://[abcd:204::1]:7777 |
| 8d62d807ee38456e98b70c388d87cfa4 | RegionOne | smapi | smapi | True | public | http://[2620:10a:a001:a103::148]:7777 |
| 99b22eb1bf0c48a6aca6ad29280a205d | RegionOne | keystone | identity | True | admin | http://[abcd:204::1]:5000/v3 |
| b2ff888d627a41daa33d651142cfdaf1 | RegionOne | keystone | identity | True | internal | http://[abcd:204::1]:5000/v3 |
| 421ae5f8cb2842e19e4891e951099156 | RegionOne | keystone | identity | True | public | http://[2620:10a:a001:a103::148]:5000/v3 |
| 815dd17e5e774948b350ed4f0c21b9f6 | RegionOne | sysinv | platform | True | admin | http://[abcd:204::1]:6385/v1 |
| a74d1690eacc40e79d39d26a4f249ab6 | RegionOne | sysinv | platform | True | internal | http://[abcd:204::1]:6385/v1 |
| 0938147b885348f1a4da7e71c15bd95c | RegionOne | sysinv | platform | True | public | http://[2620:10a:a001:a103::148]:6385/v1 |

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

Log added at
https://files.starlingx.kube.cengn.ca/launchpad/1877125

[Yang Liu (yliu12)]

This also happened on the other non-SX system with https enabled - wcp99-103 (dx+worker).

[sysadmin@controller-0 ~(keystone_admin)]$ system show | grep https
| https_enabled | True |
[sysadmin@controller-0 ~(keystone_admin)]$ openstack endpoint list | grep public
| 824c806acda54304962656f255a25c83 | RegionOne | fm | faultmanagement | True | public | http://[2620:10a:a001:a103::1094]:18002 |
| a000d418c2c7409f8d583d7a6696821f | RegionOne | patching | patching | True | public | http://[2620:10a:a001:a103::1094]:15491 |
| 42f5f1670ebc4c6f907d3055c3bb2ba4 | RegionOne | vim | nfv | True | public | http://[2620:10a:a001:a103::1094]:4545 |
| 0e8aa0dc4ff544b69c605a9776a7eba0 | RegionOne | barbican | key-manager | True | public | http://[2620:10a:a001:a103::1094]:9311 |
| c31370ce439d49aba43d240266f61a87 | RegionOne | smapi | smapi | True | public | http://[2620:10a:a001:a103::1094]:7777 |
| 83a16ee08a2042a3ad496a105d129d0d | RegionOne | keystone | identity | True | public | http://[2620:10a:a001:a103::1094]:5000/v3 |
| c30ff52169444565b3263f82e728deb0 | RegionOne | sysinv | platform | True | public | http://[2620:10a:a001:a103::1094]:6385/v1 |

[Ghada Khalil (gkhalil)]

It appears that the root-cause of the system not being setup properly is a runtime manifest application failure.

From wcp99-103:
2020-05-06T05:00:17.295 ^[[1;31mError: 2020-05-06 05:00:15 +0000 Could not find resource 'Class[Platform::Sm::Ceph::Runtime]' for relationship from 'Class[Platform::Ceph::Runtime_base]' on node controller-0

[John Kung (john-kung)]

Related issue to the https-enable:

Since config_apply_runtime_manifest() checks for INITIAL_CONFIG_COMPLETE_FLAG prior to allowing a runtime manifest apply; the https update is not applied at runtime.

e.g. logs from the SX system:

# The runtime manifest apply for https is initiated at 15:06:
sysinv 2020-05-06 15:06:46.206 128486 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=77d4495a-d16b-4402-a168-d612616452ec, classes: ['platform::haproxy::runtime', 'openstack::keystone::endpoint::runtime', 'openstack::horizon::runtime', 'platform::firewall::runtime']
sysinv 2020-05-06 15:06:49.671 128486 INFO sysinv.agent.rpcapi [-] config_apply_runtime_manifest: fanout_cast: sending config 77d4495a-d16b-4402-a168-d612616452ec {'classes': ['platform::haproxy::runtime', 'openstack::keystone::endpoint::runtime', 'openstack::horizon::runtime', 'platform::firewall::runtime'], 'force': False, 'personalities': ['controller']} to agent

However, the .initial_config_complete isnt until 15:18
                                                       |
-rw-r--r-- 1 root root 0 May 6 15:18 .initial_config_complete

This is similar case in the the non-SX lab case mentioned above.

Potential solutions:
1) automation waits for /etc/platform/.initial_config_complete (initial puppet manifest applied) before performing the system modify https, or performs host-lock/unlock to update the config if its performed prior to .initial_config_complete .

2) investigate whether https can be applied at this point in the runtime manifest (e.g. via Force option). This alternative would require some investigation from https/networking to determine whether a force apply is possible and valid at this point as the manifests required to be applied are: ['platform::haproxy::runtime', 'openstack::keystone::endpoint::runtime', 'openstack::horizon::runtime', 'platform::firewall::runtime']

[Ghada Khalil (gkhalil)]

stx.4.0 / high priority - this appears to have been broken in the last few days

[Ghada Khalil (gkhalil)]

This issue appears to be introduced/exposed by a recent code change:
https://review.opendev.org/#/c/724741/

The plan is to revert the above commit and re-work it offline.

[Ghada Khalil (gkhalil)]

The commit that introduced this issue has been reverted:
https://review.opendev.org/#/c/725950/

Merged on 2020-05-06

[Peng Peng (ppeng)]

Verified on
"2020-05-06_19-45-22"

[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+------------------------------------------------------+
| Property | Value |
+----------------------+------------------------------------------------------+
| contact | None |
| created_at | 2020-05-07T14:07:30.193509+00:00 |
| description | yow-cgcs-wildcat-99_103: setup by deployment manager |
| https_enabled | True |
| location | None |
| name | yow-cgcs-wildcat-99-103 |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 20.04 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2020-05-07T14:22:59.595363+00:00 |
| uuid | 2b3d1e2a-258e-4b43-97ca-9ac7714d8af7 |
| vswitch_type | none |
+----------------------+------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ openstack endpoint list | grep admin
| de13121934e1434184f66b3118b96337 | RegionOne | fm | faultmanagement | True | admin | http://[face::1]:18002 |
| 57d79647f93b4e28ae3c9c638f891b2a | RegionOne | patching | patching | True | admin | http://[face::1]:5491 |
| 9a72ba8f7d514cd2b313555bcc71268c | RegionOne | vim | nfv | True | admin | http://[face::1]:4545 |
| f33764eb677f49a68ab21daa7a58a10b | RegionOne | barbican | key-manager | True | admin | http://[face::1]:9311 |
| 8a686ce8426348c3b1dcc46eb920cdad | RegionOne | smapi | smapi | True | admin | http://[face::1]:7777 |
| c143727d77ee4280b74b785f0df70695 | RegionOne | keystone | identity | True | admin | http://[face::1]:5000/v3 |
| 977cac995c5040629be5b1d8c024c219 | RegionOne | sysinv | platform | True | admin | http://[face::1]:6385/v1 |
[sysadmin@controller-0 ~(keystone_admin)]$ openstack endpoint list | grep public
| d60a9c2511924f77afa8355898605ef4 | RegionOne | fm | faultmanagement | True | public | https://[2620:10a:a001:a103::1094]:18002 |
| 34c9024e28914392ae3cec1445f51dc6 | RegionOne | patching | patching ...