Platform Horizon HTTPS can no longer be enabled

[Andy]

Brief Description
-----------------
10 Minutes after HTTPS is enabled by "system modify --https_enabled=True", platform Horizon existing session (logged in before HTTPS is enabled) is still working (ie, can still access different pages). Log out and log in at port 8443 (eg: https://128.224.151.241:8443/) takes forever and never present a login page.

Severity
--------
Provide the severity of the defect.
Critical

Steps to Reproduce
------------------
- Install a two node system. Openstack applications is NOT applied.
- Enable HTTPS by "system modify --https_enabled=True".
- Check HTTPS is enabled by "system show". eg:

  | https_enabled | True |

- Access platform Horizon by HTTPS from a browser, eg:
  https://128.224.151.241:8443

Expected Behavior
------------------
User should be presented with Platform Horizon login page, and login successfully after inputting user name and password.

Actual Behavior
----------------
- Existing session (logged in before https is enabled) is still working
- Access to HTTPS never present login page.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Two node system.

Branch/Pull Time/Commit
-----------------------

BUILD_ID="20190503T013000Z"
JOB="STX_build_master_master"
<email address hidden>"
(Lab alias: cgcs-wildcat-69_70)

Last Pass
---------

BUILD_ID="20190415T233001Z"
JOB="STX_build_master_master"
<email address hidden>"
(Lab alias: cgcs-wildcat-3-6)

As reported by:
https://bugs.launchpad.net/starlingx/+bug/1827641

Timestamp/Logs
--------------
     >>> [root@controller-0 certs(keystone_admin)]# system modify
     >>> --https_enabled=True
     >>> +----------------------+-----------------------------------------------+
     >>> | Property | Value |
     >>> +----------------------+-----------------------------------------------+
     >>> | contact | None |
     >>> | created_at | 2019-05-03T15:17:40.703235+00:00 |
     >>> | description | yow-cgcs-wildcat-69_70: setup by lab_setup.sh |
     >>> | https_enabled | True |
     >>> | location | None |
     >>> | name | yow-cgcs-wildcat-69_70 |
     >>> | region_name | RegionOne |
     >>> | sdn_enabled | False |
     >>> | security_feature | spectre_meltdown_v1 |
     >>> | service_project_name | services |
     >>> | software_version | 19.01 |
     >>> | system_mode | duplex |
     >>> | system_type | All-in-one |
     >>> | timezone | UTC |
     >>> | updated_at | 2019-05-03T18:51:09.955583+00:00 |
     >>> | uuid | abe85f35-9225-4d3f-8b4c-9c3c5610043b |
     >>> | vswitch_type | ovs-dpdk |
     >>> +----------------------+-----------------------------------------------+

     >> [root@controller-0 certs(keystone_admin)]# netstat -antp | grep LIST | grep 8080
     >> tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN
     >> 1932518/lighttpd
     >> tcp6 0 0 :::8080 :::* LISTEN
     >> 1932518/lighttpd
     >> [root@controller-0 certs(keystone_admin)]# netstat -antp | grep LIST | grep 8443
     >> tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN
     >> 1932518/lighttpd
     >> tcp6 0 0 :::8443 :::* LISTEN
     >> 1932518/lighttpd

System log is attached. Note that HTTPS were turned on/off multiple times.

Test Activity
-------------
Found the issue when working on another Launchpad issue.
https://bugs.launchpad.net/starlingx/+bug/1810329

[Andy (andy.wrs)]

Quoted from Matt Peters:
"The runtime puppet manifest should be re-applied following the system https reconfiguration, but it looks like it is missing from the "configure_system_https" RPC handler. The platform::firewall::runtime is missing from the list of runtime classes and needs to be added."

[Ghada Khalil (gkhalil)]

Marking as release gating / high priority -- issue recently introduced by Calico firewall feature.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/657534

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657534
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=7a75f75d87efce78cf0d09f0a1f28f59125d7c08
Submitter: Zuul
Branch: master

commit 7a75f75d87efce78cf0d09f0a1f28f59125d7c08
Author: Yi Wang <email address hidden>
Date: Wed May 8 10:04:48 2019 +0800

    Fix Horizon access bug

    Re-apply puppet firewall runtime class when the setting https_enabled
    is changed expose correct port on OAM.

    Please note that the firewall change takes a while (dozens of seconds)
    to take effect.

    Change-Id: I82f89e9ce913021d193f77087bf5cdda3a56a56a
    Closes-Bug: #1827903
    Signed-off-by: Yi Wang <email address hidden>

[Chris Winnicki (chriswinnicki)]

Retested
Verdict: Passed

Build info:

###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190515T220331Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="102"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-05-15 22:03:31 +0000"