Brief Description ----------------- Openstack pods in bad states affer force rebooting active controller
Severity -------- Major
Steps to Reproduce ------------------ - Install and configure system, apply stx-openstack application - 'sudo reboot -f' from active controller
Expected Behavior ------------------ - system swacted to the other controller, pods are in good states. i.e., Running or Completed.
Actual Behavior ---------------- - After force rebooting the controller, a number of openstack pods stuck in Pending/Init state.
[wrsroot@controller-0 ~(keystone_admin)]$ kubectl get pods --all-namespaces | grep -v -e Completed -e Running NAMESPACE NAME READY STATUS RESTARTS AGE openstack aodh-api-65848ddb9f-d26zf 0/1 Init:0/1 0 17h openstack aodh-evaluator-6f4597fb4-bsrh9 0/1 Init:0/1 0 17h openstack aodh-listener-687d674c5b-ccscl 0/1 Init:0/1 0 17h openstack aodh-notifier-6c69c55f49-xzsk6 0/1 Init:0/1 0 17h openstack ceilometer-notification-5c89bcc85c-ckdvd 0/1 Init:0/1 0 17h openstack cinder-api-588778fd88-47gbj 0/1 Init:0/2 0 17h openstack cinder-backup-7f48f88bc8-tc88r 0/1 Init:0/3 0 17h openstack cinder-scheduler-d46f6df84-xxpqm 0/1 Init:0/2 0 17h openstack cinder-volume-857d996cc8-nqcg4 0/1 Init:0/3 0 17h openstack cinder-volume-usage-audit-1550627100-jjllv 0/1 Init:0/1 0 17h openstack glance-api-7bf78b9fd9-wcdpp 0/1 Init:0/3 0 17h openstack gnocchi-api-7d69779896-468fs 0/1 Init:0/2 0 17h openstack gnocchi-api-7d69779896-m2bh8 0/1 Init:0/2 0 17h openstack gnocchi-metricd-7j6xw 0/1 Init:0/2 1 18h openstack heat-api-694ddb96b5-clcr2 0/1 Init:0/1 0 17h openstack heat-cfn-5c74bc6f69-nt2wj 0/1 Init:0/1 0 17h openstack heat-engine-6b9c8dcf54-4s774 0/1 Init:0/1 0 17h openstack heat-engine-cleaner-1550627100-xfw2l 0/1 Init:0/1 0 17h openstack horizon-cb98c5576-dppqq 0/1 Init:0/1 0 17h openstack keystone-api-778f89467b-8qlt5 0/1 Init:0/1 0 17h openstack keystone-api-778f89467b-w96zh 0/1 Init:0/1 0 17h openstack neutron-server-df798b75f-hwml7 0/1 Init:0/1 0 17h openstack nova-api-metadata-6f7db984fc-4b64k 0/1 Pending 0 16h openstack nova-api-metadata-6f7db984fc-v79qn 0/1 Init:0/2 0 16h openstack nova-api-osapi-66d6b6778c-gpdhq 0/1 Pending 0 16h openstack nova-api-osapi-66d6b6778c-l52ck 0/1 Init:0/1 0 16h openstack nova-cell-setup-6j5hr 0/1 Init:0/2 0 16h openstack nova-compute-compute-0-75ea0372-j7z48 0/2 Init:0/5 0 16h openstack nova-compute-compute-1-3dfb81d6-zwtsd 0/2 Init:0/5 0 16h openstack nova-conductor-6b5cf94dfd-flsbr 0/1 Init:0/1 0 16h openstack nova-conductor-6b5cf94dfd-vcmmt 0/1 Pending 0 16h openstack nova-consoleauth-86d46c65cd-5k8kf 0/1 Init:0/1 0 16h openstack nova-consoleauth-86d46c65cd-n8xg7 0/1 Pending 0 16h openstack nova-db-init-m7jvd 0/3 Init:0/1 0 16h openstack nova-db-sync-g75rz 0/1 Init:0/1 0 16h openstack nova-ks-service-vjmgm 0/1 Init:0/1 0 16h openstack nova-ks-user-6rsd5 0/1 Init:0/1 0 16h openstack nova-novncproxy-56c4b5dfd7-mvg94 0/1 Init:0/3 0 16h openstack nova-placement-api-57447fd476-kvp42 0/1 Init:0/1 0 16h openstack nova-placement-api-57447fd476-xd4kd 0/1 Pending 0 16h openstack nova-scheduler-7f56659bc4-4kpj6 0/1 Init:0/1 0 16h openstack nova-scheduler-7f56659bc4-gpfsr 0/1 Pending 0 16h openstack panko-api-84b564cbfc-nr777 0/1 Init:0/1 0 17h openstack panko-events-cleaner-1550628600-q55l8 0/1 Init:0/1 0 16h openstack placement-ks-endpoints-fczxd 0/3 Init:0/1 0 16h openstack placement-ks-service-6jl87 0/1 Init:0/1 0 16h openstack placement-ks-user-9kng7 0/1 Init:0/1 0 16h openstack prometheus-mysql-exporter-5fb97d88c6-lq6qj 0/1 Init:0/1 0 17h
Reproducibility --------------- Intermittent
System Configuration -------------------- Multi-node system
Branch/Pull Time/Commit ----------------------- master as of 2019-02-15
Timestamp/Logs -------------- [2019-02-20 01:40:28,952] 139 INFO MainThread host_helper.reboot_hosts:: Rebooting active controller: controller-0 [2019-02-20 01:40:28,952] 262 DEBUG MainThread ssh.send :: Send 'sudo reboot -f'
Brief Description
-----------------
Openstack pods in bad states affer force rebooting active controller
Severity
--------
Major
Steps to Reproduce
------------------
- Install and configure system, apply stx-openstack application
- 'sudo reboot -f' from active controller
Expected Behavior
------------------
- system swacted to the other controller, pods are in good states. i.e., Running or Completed.
Actual Behavior
----------------
- After force rebooting the controller, a number of openstack pods stuck in Pending/Init state.
[wrsroot@ controller- 0 ~(keystone_admin)]$ kubectl get pods --all-namespaces | grep -v -e Completed -e Running 65848ddb9f- d26zf 0/1 Init:0/1 0 17h 6f4597fb4- bsrh9 0/1 Init:0/1 0 17h 687d674c5b- ccscl 0/1 Init:0/1 0 17h 6c69c55f49- xzsk6 0/1 Init:0/1 0 17h notification- 5c89bcc85c- ckdvd 0/1 Init:0/1 0 17h api-588778fd88- 47gbj 0/1 Init:0/2 0 17h backup- 7f48f88bc8- tc88r 0/1 Init:0/3 0 17h scheduler- d46f6df84- xxpqm 0/1 Init:0/2 0 17h volume- 857d996cc8- nqcg4 0/1 Init:0/3 0 17h volume- usage-audit- 1550627100- jjllv 0/1 Init:0/1 0 17h api-7bf78b9fd9- wcdpp 0/1 Init:0/3 0 17h api-7d69779896- 468fs 0/1 Init:0/2 0 17h api-7d69779896- m2bh8 0/1 Init:0/2 0 17h metricd- 7j6xw 0/1 Init:0/2 1 18h 694ddb96b5- clcr2 0/1 Init:0/1 0 17h 5c74bc6f69- nt2wj 0/1 Init:0/1 0 17h 6b9c8dcf54- 4s774 0/1 Init:0/1 0 17h cleaner- 1550627100- xfw2l 0/1 Init:0/1 0 17h cb98c5576- dppqq 0/1 Init:0/1 0 17h api-778f89467b- 8qlt5 0/1 Init:0/1 0 17h api-778f89467b- w96zh 0/1 Init:0/1 0 17h server- df798b75f- hwml7 0/1 Init:0/1 0 17h metadata- 6f7db984fc- 4b64k 0/1 Pending 0 16h metadata- 6f7db984fc- v79qn 0/1 Init:0/2 0 16h osapi-66d6b6778 c-gpdhq 0/1 Pending 0 16h osapi-66d6b6778 c-l52ck 0/1 Init:0/1 0 16h setup-6j5hr 0/1 Init:0/2 0 16h compute- 0-75ea0372- j7z48 0/2 Init:0/5 0 16h compute- 1-3dfb81d6- zwtsd 0/2 Init:0/5 0 16h 6b5cf94dfd- flsbr 0/1 Init:0/1 0 16h 6b5cf94dfd- vcmmt 0/1 Pending 0 16h h-86d46c65cd- 5k8kf 0/1 Init:0/1 0 16h h-86d46c65cd- n8xg7 0/1 Pending 0 16h service- vjmgm 0/1 Init:0/1 0 16h -56c4b5dfd7- mvg94 0/1 Init:0/3 0 16h api-57447fd476- kvp42 0/1 Init:0/1 0 16h api-57447fd476- xd4kd 0/1 Pending 0 16h 7f56659bc4- 4kpj6 0/1 Init:0/1 0 16h 7f56659bc4- gpfsr 0/1 Pending 0 16h 84b564cbfc- nr777 0/1 Init:0/1 0 17h cleaner- 1550628600- q55l8 0/1 Init:0/1 0 16h ks-endpoints- fczxd 0/3 Init:0/1 0 16h ks-service- 6jl87 0/1 Init:0/1 0 16h ks-user- 9kng7 0/1 Init:0/1 0 16h mysql-exporter- 5fb97d88c6- lq6qj 0/1 Init:0/1 0 17h
NAMESPACE NAME READY STATUS RESTARTS AGE
openstack aodh-api-
openstack aodh-evaluator-
openstack aodh-listener-
openstack aodh-notifier-
openstack ceilometer-
openstack cinder-
openstack cinder-
openstack cinder-
openstack cinder-
openstack cinder-
openstack glance-
openstack gnocchi-
openstack gnocchi-
openstack gnocchi-
openstack heat-api-
openstack heat-cfn-
openstack heat-engine-
openstack heat-engine-
openstack horizon-
openstack keystone-
openstack keystone-
openstack neutron-
openstack nova-api-
openstack nova-api-
openstack nova-api-
openstack nova-api-
openstack nova-cell-
openstack nova-compute-
openstack nova-compute-
openstack nova-conductor-
openstack nova-conductor-
openstack nova-consoleaut
openstack nova-consoleaut
openstack nova-db-init-m7jvd 0/3 Init:0/1 0 16h
openstack nova-db-sync-g75rz 0/1 Init:0/1 0 16h
openstack nova-ks-
openstack nova-ks-user-6rsd5 0/1 Init:0/1 0 16h
openstack nova-novncproxy
openstack nova-placement-
openstack nova-placement-
openstack nova-scheduler-
openstack nova-scheduler-
openstack panko-api-
openstack panko-events-
openstack placement-
openstack placement-
openstack placement-
openstack prometheus-
Reproducibility
---------------
Intermittent
System Configuration ------- ------
-------
Multi-node system
Branch/Pull Time/Commit ------- ------- --
-------
master as of 2019-02-15
Timestamp/Logs reboot_ hosts:: Rebooting active controller: controller-0
--------------
[2019-02-20 01:40:28,952] 139 INFO MainThread host_helper.
[2019-02-20 01:40:28,952] 262 DEBUG MainThread ssh.send :: Send 'sudo reboot -f'