Failed to apply run-time manifest after the http port was modified

[Chris Winnicki]

Title
-----
Failed to apply run-time manifest after the http port was modified

Brief Description
-----------------
same as the title

Severity
--------
Provide the severity of the defect.
Minor

Steps to Reproduce
------------------
# Modify the http serviec paramter

[wrsroot@controller-0 ~(keystone_admin)]$ system service-parameter-modify http config http_port=6666 https_port=6666
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| uuid | 64ad337b-6783-476a-abcb-f0e92ac1b8ab |
| service | http |
| section | config |
| name | http_port |
| value | 6666 |
| personality | None |
| resource | None |
+-------------+--------------------------------------+
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| uuid | bf5ed9f5-3cdd-4e91-8a52-24a293fcfea7 |
| service | http |
| section | config |
| name | https_port |
| value | 6666 |
| personality | None |
| resource | None |
+-------------+--------------------------------------+

# Wait for alarms to clear
[wrsroot@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+--------------------------------------------+-------------------+----------+------------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+--------------------------------------------+-------------------+----------+------------------+
| 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2019-02-27T21:58 |
| | | | | :41.938133 |
| | | | | |
+----------+--------------------------------------------+-------------------+----------+------------------+
[wrsroot@controller-0 ~(keystone_admin)]$ system service-parameter-apply http
Applying http service parameters
[wrsroot@controller-0 ~(keystone_admin)]$
[wrsroot@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+--------------------------------------------+-------------------+----------+------------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+--------------------------------------------+-------------------+----------+------------------+
| 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2019-02-27T21:58 |
| | | | | :57.344490 |
| | | | | |
+----------+--------------------------------------------+-------------------+----------+------------------+
[wrsroot@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+--------------------------------------------+-------------------+----------+------------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+--------------------------------------------+-------------------+----------+------------------+
| 250.001 | controller-0 Configuration is out-of-date. | host=controller-0 | major | 2019-02-27T21:58 |
| | | | | :57.344490 |
| | | | | |
+----------+--------------------------------------------+-------------------+----------+------------------+
[wrsroot@controller-0 ~(keystone_admin)]$ fm alarm-list

[wrsroot@controller-0 ~(keystone_admin)]$ system service-parameter-list
+--------------------------------------+----------+-------------+-----------------------------+-------+-------------+----------+
| uuid | service | section | name | value | personality | resource |
+--------------------------------------+----------+-------------+-----------------------------+-------+-------------+----------+
| daaeb592-51b1-4c66-ae5a-62faca399fa2 | identity | assignment | driver | sql | None | None |
| 75d7600e-0d4e-4e63-9f35-207b569cda38 | horizon | auth | lockout_retries | 3 | None | None |
| a65d8b19-a83f-4f8b-a72b-1fd3c74d71bc | horizon | auth | lockout_seconds | 300 | None | None |
| bcf734ae-30b0-4f17-8259-1f8ebe83c06e | swift | config | fs_size_mb | 25 | None | None |
| 64ad337b-6783-476a-abcb-f0e92ac1b8ab | http | config | http_port | 6666 | None | None |
| bf5ed9f5-3cdd-4e91-8a52-24a293fcfea7 | http | config | https_port | 6666 | None | None |
| 7fd45d36-3a44-4db0-bf51-8c7b8da53e7b | swift | config | service_enabled | false | None | None |
| af4e2d97-910d-48dc-8445-b12a3dfd7100 | identity | config | token_expiration | 3600 | None | None |
| db2fbb5c-007a-441d-be10-3d1a722989f1 | aodh | database | alarm_history_time_to_live | 86400 | None | None |
| f91d4ca4-b36f-414d-96e3-dd832f6d98a0 | panko | database | event_time_to_live | 86400 | None | None |
| 9420b9d5-a630-4749-8769-8fdce2bd8eb3 | cinder | emc_vnx | enabled | false | None | None |
| ef04439c-0b08-4ea2-a0da-9f83c7151448 | cinder | hpe3par | enabled | false | None | None |
| 261a348d-7649-4105-aba6-4074e7365820 | cinder | hpe3par10 | enabled | false | None | None |
| ec145c7f-26c1-45bf-bf0a-6f405b347ec9 | cinder | hpe3par11 | enabled | false | None | None |
| 4f55ebdf-8bf8-498b-93a8-e95874ca5d23 | cinder | hpe3par12 | enabled | false | None | None |
| cba101be-bfd3-4314-ba49-db372923302f | cinder | hpe3par2 | enabled | false | None | None |
| 26415921-8878-4a5b-8e7a-8b506092e66a | cinder | hpe3par3 | enabled | false | None | None |
| 0262b1b1-de77-4be3-a070-fcb4def0bd77 | cinder | hpe3par4 | enabled | false | None | None |
| 69e77b4d-5d73-40c1-ab3f-355908e716bf | cinder | hpe3par5 | enabled | false | None | None |
| a0902b69-d30c-464b-a304-8efff3d1092f | cinder | hpe3par6 | enabled | false | None | None |
| 597ac7c4-a98a-4358-9213-8d0acb99497e | cinder | hpe3par7 | enabled | false | None | None |
| 469b7070-e479-4bb5-8fd9-5531efcac53f | cinder | hpe3par8 | enabled | false | None | None |
| 74591ab1-ab0c-4a31-9552-d2b130057897 | cinder | hpe3par9 | enabled | false | None | None |
| ae9bc8fd-73d8-4ac7-8bfc-95c48267fbc7 | cinder | hpelefthand | enabled | false | None | None |
| 0d2396bb-a8ce-4bc2-bb1d-ea3dbae0dc3f | identity | identity | driver | sql | None | None |
| 5100394b-c910-41b3-9a66-c4fe517a5fcc | platform | maintenance | controller_boot_timeout | 1200 | None | None |
| a5db8279-6c6c-44a4-b3bd-b89cc6839d86 | platform | maintenance | heartbeat_degrade_threshold | 6 | None | None |
| e1270f2b-34ff-49ec-bf24-d5b317f040e5 | platform | maintenance | heartbeat_failure_action | fail | None | None |
| b413a6b7-7c71-49c9-88e5-075e92381cd7 | platform | maintenance | heartbeat_failure_threshold | 10 | None | None |
| dd0eba5b-288c-419c-bf76-8a6f8b8aa1b5 | platform | maintenance | heartbeat_period | 100 | None | None |
| 20864874-5cc7-4dff-9b3e-c0f87359663a | platform | maintenance | mnfa_threshold | 2 | None | None |
| ebf34627-2d15-4d99-977f-34eb73f4a3cf | platform | maintenance | mnfa_timeout | 0 | None | None |
| 5219e853-ec40-4f9c-b089-3fc2543288af | platform | maintenance | worker_boot_timeout | 720 | None | None |
+--------------------------------------+----------+-------------+-----------------------------+-------+-------------+----------+

# Attampt to access Horizon dashboard on port 6666

Expected Behavior
------------------
After modifying http service-parameter Horizon dashboard should be accessbile on http port 6666

Actual Behavior
----------------
After modifying http service-parameter to a port other than 6666 renders Horizon dashboard inaccessible, apply run-time manifest failed to apply:

2019-03-04T19:01:14.342 Debug: 2019-03-04 19:01:14 +0000 Exec[update local starlingx helm repo](provider=posix): Executing 'helm repo add starlingx http://127.0.0.1:6666/helm_charts' 2019-03-04T19:01:14.351 Debug: 2019-03-04 19:01:14 +0000 Executing with uid=wrsroot gid=wrs: 'helm repo add starlingx http://127.0.0.1:6666/helm_charts' 2019-03-04T19:01:14.413 ^[[mNotice: 2019-03-04 19:01:14 +0000 /Stage[main]/Platform::Helm::Runtime/Exec[update local starlingx helm repo]/returns: Error: Looks like "http://127.0.0.1:6666/helm_charts" is not a valid chart repository or cannot be reached: Get http://127.0.0.1:6666/helm_charts/index.yaml: dial tcp 127.0.0.1:6666: connect: connection refused

Reproducibility
---------------
1/10

System Configuration
--------------------
One node system, http, IPv4

Branch/Pull Time/Commit
-----------------------
2019-02-26

Timestamp/Logs
--------------
2019-03-04T19:01:14.342
yow-cgcs-wolfpass-15-controller-0_20190304.193510.tar

[Chris Winnicki (chriswinnicki)]

Logs attached: yow-cgcs-wolfpass-15-controller-0_20190304.193510.tar

[Ghada Khalil (gkhalil)]

Marking as release gating until further investigation. Low priority given the issue is not reproducible 10% of the time.

[Bill Zvonar (billzvonar)]

Assigned to Bruce for re-assignment.

[Abraham Arce (xe1gyq)]

I am taking the ownership of this bug to solve, after an initial review I was able to reproduce. I will provide more details and questions in the next couple of days looking to have an estimated time of completion.

[Abraham Arce (xe1gyq)]

My research (mental map) will be written down here: https://github.com/xe1gyq/starlingx/blob/master/bugs/1818887.md

[Abraham Arce (xe1gyq)]

The first time service-parameter-apply is applied, we have the process error and the fault alarm kept, the second time it executes with out error and the fault alarm is cleared:

system service-parameter-modify http config http_port=8085
system service-parameter-apply http

  sysinv.agent.manager _apply_runtime_manifest with hieradata_path
  Failed to execute runtime manifest for host 192.168.204.3
  CalledProcessError: Command '['/usr/local/bin/puppet-manifest-apply.sh']

system service-parameter-apply http

  Runtime manifest apply completed for classes...
  Agent config applied
  SYS_I Clear system config alarm

[Abraham Arce (xe1gyq)]

Email sent to mailing list:
http://lists.starlingx.io/pipermail/starlingx-discuss/2019-August/005764.html

[Ghada Khalil (gkhalil)]

As per agreement with the community, moving all unresolved medium/low priority bugs from stx.2.0 to stx.3.0

[Ghada Khalil (gkhalil)]

I asked Chris Winnicki to re-test this scenario in a recent load and he confirmed that he is not able to reproduce this issue.

Tested on an AIO-SX system
Load: build from master - Build Timestamp: 2019-12-11_21-20-16