Comment 0 for bug 1816842

Brief Description
-----------------
Openstack pods in bad states affer force rebooting active controller

Severity
--------
Major

Steps to Reproduce
------------------
- Install and configure system, apply stx-openstack application
- 'sudo reboot -f' from active controller

Expected Behavior
------------------
- system swacted to the other controller, pods are in good states. i.e., Running or Completed.

Actual Behavior
----------------
- After force rebooting the controller, a number of openstack pods stuck in Pending/Init state.

[wrsroot@controller-0 ~(keystone_admin)]$ kubectl get pods --all-namespaces | grep -v -e Completed -e Running
NAMESPACE NAME READY STATUS RESTARTS AGE
openstack aodh-api-65848ddb9f-d26zf 0/1 Init:0/1 0 17h
openstack aodh-evaluator-6f4597fb4-bsrh9 0/1 Init:0/1 0 17h
openstack aodh-listener-687d674c5b-ccscl 0/1 Init:0/1 0 17h
openstack aodh-notifier-6c69c55f49-xzsk6 0/1 Init:0/1 0 17h
openstack ceilometer-notification-5c89bcc85c-ckdvd 0/1 Init:0/1 0 17h
openstack cinder-api-588778fd88-47gbj 0/1 Init:0/2 0 17h
openstack cinder-backup-7f48f88bc8-tc88r 0/1 Init:0/3 0 17h
openstack cinder-scheduler-d46f6df84-xxpqm 0/1 Init:0/2 0 17h
openstack cinder-volume-857d996cc8-nqcg4 0/1 Init:0/3 0 17h
openstack cinder-volume-usage-audit-1550627100-jjllv 0/1 Init:0/1 0 17h
openstack glance-api-7bf78b9fd9-wcdpp 0/1 Init:0/3 0 17h
openstack gnocchi-api-7d69779896-468fs 0/1 Init:0/2 0 17h
openstack gnocchi-api-7d69779896-m2bh8 0/1 Init:0/2 0 17h
openstack gnocchi-metricd-7j6xw 0/1 Init:0/2 1 18h
openstack heat-api-694ddb96b5-clcr2 0/1 Init:0/1 0 17h
openstack heat-cfn-5c74bc6f69-nt2wj 0/1 Init:0/1 0 17h
openstack heat-engine-6b9c8dcf54-4s774 0/1 Init:0/1 0 17h
openstack heat-engine-cleaner-1550627100-xfw2l 0/1 Init:0/1 0 17h
openstack horizon-cb98c5576-dppqq 0/1 Init:0/1 0 17h
openstack keystone-api-778f89467b-8qlt5 0/1 Init:0/1 0 17h
openstack keystone-api-778f89467b-w96zh 0/1 Init:0/1 0 17h
openstack neutron-server-df798b75f-hwml7 0/1 Init:0/1 0 17h
openstack nova-api-metadata-6f7db984fc-4b64k 0/1 Pending 0 16h
openstack nova-api-metadata-6f7db984fc-v79qn 0/1 Init:0/2 0 16h
openstack nova-api-osapi-66d6b6778c-gpdhq 0/1 Pending 0 16h
openstack nova-api-osapi-66d6b6778c-l52ck 0/1 Init:0/1 0 16h
openstack nova-cell-setup-6j5hr 0/1 Init:0/2 0 16h
openstack nova-compute-compute-0-75ea0372-j7z48 0/2 Init:0/5 0 16h
openstack nova-compute-compute-1-3dfb81d6-zwtsd 0/2 Init:0/5 0 16h
openstack nova-conductor-6b5cf94dfd-flsbr 0/1 Init:0/1 0 16h
openstack nova-conductor-6b5cf94dfd-vcmmt 0/1 Pending 0 16h
openstack nova-consoleauth-86d46c65cd-5k8kf 0/1 Init:0/1 0 16h
openstack nova-consoleauth-86d46c65cd-n8xg7 0/1 Pending 0 16h
openstack nova-db-init-m7jvd 0/3 Init:0/1 0 16h
openstack nova-db-sync-g75rz 0/1 Init:0/1 0 16h
openstack nova-ks-service-vjmgm 0/1 Init:0/1 0 16h
openstack nova-ks-user-6rsd5 0/1 Init:0/1 0 16h
openstack nova-novncproxy-56c4b5dfd7-mvg94 0/1 Init:0/3 0 16h
openstack nova-placement-api-57447fd476-kvp42 0/1 Init:0/1 0 16h
openstack nova-placement-api-57447fd476-xd4kd 0/1 Pending 0 16h
openstack nova-scheduler-7f56659bc4-4kpj6 0/1 Init:0/1 0 16h
openstack nova-scheduler-7f56659bc4-gpfsr 0/1 Pending 0 16h
openstack panko-api-84b564cbfc-nr777 0/1 Init:0/1 0 17h
openstack panko-events-cleaner-1550628600-q55l8 0/1 Init:0/1 0 16h
openstack placement-ks-endpoints-fczxd 0/3 Init:0/1 0 16h
openstack placement-ks-service-6jl87 0/1 Init:0/1 0 16h
openstack placement-ks-user-9kng7 0/1 Init:0/1 0 16h
openstack prometheus-mysql-exporter-5fb97d88c6-lq6qj 0/1 Init:0/1 0 17h

Reproducibility
---------------
Intermittent

System Configuration
--------------------
Multi-node system

Branch/Pull Time/Commit
-----------------------
master as of 2019-02-15

Timestamp/Logs
--------------
[2019-02-20 01:40:28,952] 139 INFO MainThread host_helper.reboot_hosts:: Rebooting active controller: controller-0
[2019-02-20 01:40:28,952] 262 DEBUG MainThread ssh.send :: Send 'sudo reboot -f'