cleanup nova services after lock/unlock of controller

[Gerry Kopec]

Brief Description
-----------------
Each time a controller is locked/unlocked, we end up with leftover nova services for the deleted pods.
controller-1:~# openstack compute service list
+-----+------------------+-----------------------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+-----+------------------+-----------------------------------+----------+---------+-------+----------------------------+
| 7 | nova-conductor | nova-conductor-64c878c9fc-vmxv4 | internal | enabled | down | 2019-02-26T21:17:59.000000 |
| 10 | nova-consoleauth | nova-consoleauth-5fcdfd4797-928th | internal | enabled | down | 2019-02-26T20:48:04.000000 |
| 13 | nova-consoleauth | nova-consoleauth-5fcdfd4797-7p99c | internal | enabled | down | 2019-02-26T21:17:53.000000 |
| 19 | nova-scheduler | nova-scheduler-79fdf5bb69-cqvcn | internal | enabled | down | 2019-02-26T21:18:11.000000 |
| 37 | nova-conductor | nova-conductor-64c878c9fc-gldqg | internal | enabled | down | 2019-02-26T20:47:50.000000 |
| 40 | nova-scheduler | nova-scheduler-79fdf5bb69-4jdbg | internal | enabled | down | 2019-02-26T20:48:01.000000 |
| 73 | nova-compute | compute-1 | nova | enabled | up | 2019-02-27T02:28:57.000000 |
| 76 | nova-compute | compute-0 | nova | enabled | up | 2019-02-27T02:28:56.000000 |
| 81 | nova-consoleauth | nova-consoleauth-5fcdfd4797-8xjlg | internal | enabled | down | 2019-02-26T21:44:35.000000 |
| 85 | nova-scheduler | nova-scheduler-79fdf5bb69-vjlb2 | internal | enabled | down | 2019-02-26T21:44:29.000000 |
| 115 | nova-conductor | nova-conductor-64c878c9fc-696qc | internal | enabled | down | 2019-02-26T21:44:35.000000 |
| 119 | nova-conductor | nova-conductor-64c878c9fc-n5gfm | internal | enabled | down | 2019-02-26T22:14:05.000000 |
| 121 | nova-consoleauth | nova-consoleauth-5fcdfd4797-r7mp9 | internal | enabled | down | 2019-02-26T22:14:05.000000 |
| 123 | nova-scheduler | nova-scheduler-79fdf5bb69-5tkmq | internal | enabled | down | 2019-02-26T22:14:14.000000 |
| 143 | nova-scheduler | nova-scheduler-79fdf5bb69-hng69 | internal | enabled | down | 2019-02-26T23:47:45.000000 |
| 169 | nova-conductor | nova-conductor-64c878c9fc-lfnns | internal | enabled | down | 2019-02-26T23:47:33.000000 |
| 171 | nova-consoleauth | nova-consoleauth-5fcdfd4797-f97l8 | internal | enabled | down | 2019-02-26T23:47:38.000000 |
| 175 | nova-scheduler | nova-scheduler-79fdf5bb69-wq8pc | internal | enabled | down | 2019-02-26T22:34:08.000000 |
| 195 | nova-consoleauth | nova-consoleauth-5fcdfd4797-ht28z | internal | enabled | down | 2019-02-26T22:34:03.000000 |
| 197 | nova-conductor | nova-conductor-64c878c9fc-l9cf8 | internal | enabled | down | 2019-02-26T22:33:45.000000 |
| 201 | nova-conductor | nova-conductor-64c878c9fc-k46c7 | internal | enabled | down | 2019-02-27T01:54:11.000000 |
| 203 | nova-consoleauth | nova-consoleauth-5fcdfd4797-m2c4r | internal | enabled | down | 2019-02-27T01:54:06.000000 |
| 207 | nova-scheduler | nova-scheduler-79fdf5bb69-wb89l | internal | enabled | down | 2019-02-27T01:54:12.000000 |
| 225 | nova-conductor | nova-conductor-64c878c9fc-fndtf | internal | enabled | down | 2019-02-27T01:32:28.000000 |
| 227 | nova-consoleauth | nova-consoleauth-5fcdfd4797-z4n5w | internal | enabled | down | 2019-02-27T01:32:41.000000 |
| 229 | nova-scheduler | nova-scheduler-79fdf5bb69-2mmnf | internal | enabled | down | 2019-02-27T01:32:31.000000 |
| 259 | nova-conductor | nova-conductor-64c878c9fc-mhfr4 | internal | enabled | up | 2019-02-27T02:28:55.000000 |
| 261 | nova-consoleauth | nova-consoleauth-5fcdfd4797-wtlkf | internal | enabled | up | 2019-02-27T02:28:57.000000 |
| 265 | nova-scheduler | nova-scheduler-79fdf5bb69-s6pd6 | internal | enabled | up | 2019-02-27T02:28:57.000000 |
| 268 | nova-scheduler | nova-scheduler-79fdf5bb69-dfb6x | internal | enabled | up | 2019-02-27T02:28:51.000000 |
| 288 | nova-conductor | nova-conductor-64c878c9fc-rb4bf | internal | enabled | up | 2019-02-27T02:28:54.000000 |
| 292 | nova-consoleauth | nova-consoleauth-5fcdfd4797-2tt69 | internal | enabled | up | 2019-02-27T02:28:56.000000 |
+-----+------------------+-----------------------------------+----------+---------+-------+----------------------------+

Issue is likely due to override/turning off of nova tasks that should clean this up in openstack-helm.

Severity
--------
Major

Steps to Reproduce
------------------
Lock/unlock controller

Expected Behavior
------------------
Resources should be cleaned up

Actual Behavior
----------------
nova controller services are not cleaned up

Reproducibility
---------------
Reproducible

System Configuration
--------------------
All

Branch/Pull Time/Commit
-----------------------
day one issue with nova overrides

Last Pass
---------
none

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Developer testing

[Ghada Khalil (gkhalil)]

Marking as stx.2.0 release gating - this issue results in a resource leak

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/672757

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672757
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=d36e444bba86790a7302c1e7a7898f478764f10f
Submitter: Zuul
Branch: master

commit d36e444bba86790a7302c1e7a7898f478764f10f
Author: Gerry Kopec <email address hidden>
Date: Wed Jul 24 22:37:51 2019 -0400

    Enable nova service cleaner in helm chart

    Enable service cleaner cronjob in nova helm chart. This will run hourly
    and delete any nova services that are no longer up (e.g. conductor,
    scheduler & consoleauth). These will be left over after controller
    lock/unlock or application-update.

    Change-Id: I001bf79b497eb1924b4252612c5ead6e992e8196
    Closes-Bug: 1835565
    Signed-off-by: Gerry Kopec <email address hidden>