| 2019-12-06 18:36:34 |
Yosief Gebremariam |
bug |
|
|
added bug |
| 2019-12-06 18:36:34 |
Yosief Gebremariam |
attachment added |
|
ALL_NODES_20191206.173815.tar https://bugs.launchpad.net/bugs/1855474/+attachment/5310283/+files/ALL_NODES_20191206.173815.tar |
|
| 2019-12-06 18:37:16 |
Yosief Gebremariam |
summary |
openstack pods were not recovered after force reboot active controller |
OpenStack pods were not recovered after force reboot active controller |
|
| 2019-12-06 19:49:07 |
Ghada Khalil |
description |
Brief Description
-----------------
Many OpenStack pods in fail to recover or were slow to recover after force rebooting the active controller
Severity
--------
Major
Steps to Reproduce
------------------
- Install and configure system, apply stx-openstack application
- 'sudo reboot -f' from active controller
Expected Behavior
------------------
- system swacts to the standby controller and all OpenStack pods recover to Running or Completed states.
Actual Behavior
----------------
- After force rebooting the controller, a number of OpenStack pods stuck in Init state. The keystone API and cinder-volume pods crushed.
controller-0:~$ kubectl get pods --all-namespaces | grep -v -e Completed -e Running
NAMESPACE NAME READY STATUS RESTARTS AGE
openstack cinder-api-59fd9c7c6f-86h2d 0/1 Init:0/2 0 3h
openstack cinder-volume-654bcb6569-lsjxt 0/1 Init:CrashLoopBackOff 22 3h
openstack fm-rest-api-78f97cc864-fqkhj 0/1 Init:0/1 0 3h
openstack glance-api-54777c6d45-gxrdc 0/1 Init:0/3 0 3h
openstack heat-api-69b8487b88-g4tc2 0/1 Init:0/1 0 3h
openstack heat-cfn-6b4b6b74f8-w7f78 0/1 Init:0/1 0 3h
openstack heat-engine-8458cf778f-xbbd4 0/1 Init:0/1 0 3h
openstack heat-engine-cleaner-1575645900-pd697 0/1 Init:0/1 0 178m
openstack horizon-5545469f58-j4bf6 0/1 Init:0/1 0 175m
openstack keystone-api-6c45dc9dbb-2v8h5 0/1 CrashLoopBackOff 43 3h39m
openstack keystone-api-6c45dc9dbb-pch72 0/1 Init:0/1 0 3h
openstack neutron-server-79c6fdf585-lwpb7 0/1 Init:0/1 0 3h
openstack nova-api-metadata-855ccf8fc4-mk446 0/1 Init:0/2 0 3h
openstack nova-api-osapi-58b7ffbf-zjv8l 0/1 Init:0/1 0 3h
openstack nova-conductor-6bbf89bf4c-7bhvg 0/1 Init:0/1 0 3h
openstack nova-novncproxy-58779744bd-szx4m 0/1 Init:0/3 0 3h
openstack nova-scheduler-67c986b5c8-rgt8x 0/1 Init:0/1 0 3h
openstack nova-service-cleaner-1575648000-kdln5 0/1 Init:0/1 0 143m
Reproducibility
---------------
Intermittent
System Configuration
--------------------
Multi-node system
Branch/Pull Time/Commit
-----------------------
r/stx.3.0 as of 2019-12-05 02:30:00
Timestamp/Logs
--------------
2019-12-06 15:21:50,338] 181 INFO MainThread host_helper.reboot_hosts:: Rebooting active controller: controller-0
[2019-12-06 15:21:50,338] 311 DEBUG MainThread ssh.send :: Send 'sudo reboot -f' |
Brief Description
-----------------
Many OpenStack pods in fail to recover or were slow to recover after force rebooting the active controller
Severity
--------
Major
Steps to Reproduce
------------------
- Install and configure system, apply stx-openstack application
- 'sudo reboot -f' from active controller
Expected Behavior
------------------
- system swacts to the standby controller and all OpenStack pods recover to Running or Completed states.
Actual Behavior
----------------
- After force rebooting the controller, a number of OpenStack pods stuck in Init state. The keystone API and cinder-volume pods crushed.
controller-0:~$ kubectl get pods --all-namespaces | grep -v -e Completed -e Running
NAMESPACE NAME READY STATUS RESTARTS AGE
openstack cinder-api-59fd9c7c6f-86h2d 0/1 Init:0/2 0 3h
openstack cinder-volume-654bcb6569-lsjxt 0/1 Init:CrashLoopBackOff 22 3h
openstack fm-rest-api-78f97cc864-fqkhj 0/1 Init:0/1 0 3h
openstack glance-api-54777c6d45-gxrdc 0/1 Init:0/3 0 3h
openstack heat-api-69b8487b88-g4tc2 0/1 Init:0/1 0 3h
openstack heat-cfn-6b4b6b74f8-w7f78 0/1 Init:0/1 0 3h
openstack heat-engine-8458cf778f-xbbd4 0/1 Init:0/1 0 3h
openstack heat-engine-cleaner-1575645900-pd697 0/1 Init:0/1 0 178m
openstack horizon-5545469f58-j4bf6 0/1 Init:0/1 0 175m
openstack keystone-api-6c45dc9dbb-2v8h5 0/1 CrashLoopBackOff 43 3h39m
openstack keystone-api-6c45dc9dbb-pch72 0/1 Init:0/1 0 3h
openstack neutron-server-79c6fdf585-lwpb7 0/1 Init:0/1 0 3h
openstack nova-api-metadata-855ccf8fc4-mk446 0/1 Init:0/2 0 3h
openstack nova-api-osapi-58b7ffbf-zjv8l 0/1 Init:0/1 0 3h
openstack nova-conductor-6bbf89bf4c-7bhvg 0/1 Init:0/1 0 3h
openstack nova-novncproxy-58779744bd-szx4m 0/1 Init:0/3 0 3h
openstack nova-scheduler-67c986b5c8-rgt8x 0/1 Init:0/1 0 3h
openstack nova-service-cleaner-1575648000-kdln5 0/1 Init:0/1 0 143m
Reproducibility
---------------
Intermittent (2 out of 3)
System Configuration
--------------------
Multi-node system
Branch/Pull Time/Commit
-----------------------
r/stx.3.0 as of 2019-12-05 02:30:00
Timestamp/Logs
--------------
2019-12-06 15:21:50,338] 181 INFO MainThread host_helper.reboot_hosts:: Rebooting active controller: controller-0
[2019-12-06 15:21:50,338] 311 DEBUG MainThread ssh.send :: Send 'sudo reboot -f' |
|
| 2019-12-06 19:49:20 |
Ghada Khalil |
tags |
|
stx.containers stx.distro.openstack |
|
| 2019-12-06 19:49:42 |
Ghada Khalil |
starlingx: assignee |
|
yong hu (yhu6) |
|
| 2019-12-06 20:02:46 |
Yosief Gebremariam |
description |
Brief Description
-----------------
Many OpenStack pods in fail to recover or were slow to recover after force rebooting the active controller
Severity
--------
Major
Steps to Reproduce
------------------
- Install and configure system, apply stx-openstack application
- 'sudo reboot -f' from active controller
Expected Behavior
------------------
- system swacts to the standby controller and all OpenStack pods recover to Running or Completed states.
Actual Behavior
----------------
- After force rebooting the controller, a number of OpenStack pods stuck in Init state. The keystone API and cinder-volume pods crushed.
controller-0:~$ kubectl get pods --all-namespaces | grep -v -e Completed -e Running
NAMESPACE NAME READY STATUS RESTARTS AGE
openstack cinder-api-59fd9c7c6f-86h2d 0/1 Init:0/2 0 3h
openstack cinder-volume-654bcb6569-lsjxt 0/1 Init:CrashLoopBackOff 22 3h
openstack fm-rest-api-78f97cc864-fqkhj 0/1 Init:0/1 0 3h
openstack glance-api-54777c6d45-gxrdc 0/1 Init:0/3 0 3h
openstack heat-api-69b8487b88-g4tc2 0/1 Init:0/1 0 3h
openstack heat-cfn-6b4b6b74f8-w7f78 0/1 Init:0/1 0 3h
openstack heat-engine-8458cf778f-xbbd4 0/1 Init:0/1 0 3h
openstack heat-engine-cleaner-1575645900-pd697 0/1 Init:0/1 0 178m
openstack horizon-5545469f58-j4bf6 0/1 Init:0/1 0 175m
openstack keystone-api-6c45dc9dbb-2v8h5 0/1 CrashLoopBackOff 43 3h39m
openstack keystone-api-6c45dc9dbb-pch72 0/1 Init:0/1 0 3h
openstack neutron-server-79c6fdf585-lwpb7 0/1 Init:0/1 0 3h
openstack nova-api-metadata-855ccf8fc4-mk446 0/1 Init:0/2 0 3h
openstack nova-api-osapi-58b7ffbf-zjv8l 0/1 Init:0/1 0 3h
openstack nova-conductor-6bbf89bf4c-7bhvg 0/1 Init:0/1 0 3h
openstack nova-novncproxy-58779744bd-szx4m 0/1 Init:0/3 0 3h
openstack nova-scheduler-67c986b5c8-rgt8x 0/1 Init:0/1 0 3h
openstack nova-service-cleaner-1575648000-kdln5 0/1 Init:0/1 0 143m
Reproducibility
---------------
Intermittent (2 out of 3)
System Configuration
--------------------
Multi-node system
Branch/Pull Time/Commit
-----------------------
r/stx.3.0 as of 2019-12-05 02:30:00
Timestamp/Logs
--------------
2019-12-06 15:21:50,338] 181 INFO MainThread host_helper.reboot_hosts:: Rebooting active controller: controller-0
[2019-12-06 15:21:50,338] 311 DEBUG MainThread ssh.send :: Send 'sudo reboot -f' |
Brief Description
-----------------
Many OpenStack pods fail to recover or were slow to recover after force rebooting the active controller
Severity
--------
Major
Steps to Reproduce
------------------
- Install and configure system, apply stx-openstack application
- 'sudo reboot -f' from active controller
Expected Behavior
------------------
- system swacts to the standby controller and all OpenStack pods recover to Running or Completed states.
Actual Behavior
----------------
- After force rebooting the controller, a number of OpenStack pods stuck in Init state. The keystone API and cinder-volume pods crushed.
controller-0:~$ kubectl get pods --all-namespaces | grep -v -e Completed -e Running
NAMESPACE NAME READY STATUS RESTARTS AGE
openstack cinder-api-59fd9c7c6f-86h2d 0/1 Init:0/2 0 3h
openstack cinder-volume-654bcb6569-lsjxt 0/1 Init:CrashLoopBackOff 22 3h
openstack fm-rest-api-78f97cc864-fqkhj 0/1 Init:0/1 0 3h
openstack glance-api-54777c6d45-gxrdc 0/1 Init:0/3 0 3h
openstack heat-api-69b8487b88-g4tc2 0/1 Init:0/1 0 3h
openstack heat-cfn-6b4b6b74f8-w7f78 0/1 Init:0/1 0 3h
openstack heat-engine-8458cf778f-xbbd4 0/1 Init:0/1 0 3h
openstack heat-engine-cleaner-1575645900-pd697 0/1 Init:0/1 0 178m
openstack horizon-5545469f58-j4bf6 0/1 Init:0/1 0 175m
openstack keystone-api-6c45dc9dbb-2v8h5 0/1 CrashLoopBackOff 43 3h39m
openstack keystone-api-6c45dc9dbb-pch72 0/1 Init:0/1 0 3h
openstack neutron-server-79c6fdf585-lwpb7 0/1 Init:0/1 0 3h
openstack nova-api-metadata-855ccf8fc4-mk446 0/1 Init:0/2 0 3h
openstack nova-api-osapi-58b7ffbf-zjv8l 0/1 Init:0/1 0 3h
openstack nova-conductor-6bbf89bf4c-7bhvg 0/1 Init:0/1 0 3h
openstack nova-novncproxy-58779744bd-szx4m 0/1 Init:0/3 0 3h
openstack nova-scheduler-67c986b5c8-rgt8x 0/1 Init:0/1 0 3h
openstack nova-service-cleaner-1575648000-kdln5 0/1 Init:0/1 0 143m
Reproducibility
---------------
Intermittent (2 out of 3)
System Configuration
--------------------
Multi-node system
Branch/Pull Time/Commit
-----------------------
r/stx.3.0 as of 2019-12-05 02:30:00
Timestamp/Logs
--------------
2019-12-06 15:21:50,338] 181 INFO MainThread host_helper.reboot_hosts:: Rebooting active controller: controller-0
[2019-12-06 15:21:50,338] 311 DEBUG MainThread ssh.send :: Send 'sudo reboot -f' |
|
| 2019-12-09 00:29:36 |
yong hu |
starlingx: assignee |
yong hu (yhu6) |
zhipeng liu (zhipengs) |
|
| 2019-12-09 01:12:08 |
yong hu |
tags |
stx.containers stx.distro.openstack |
stx.3.0 stx.containers stx.distro.openstack |
|
| 2019-12-09 01:12:16 |
yong hu |
starlingx: importance |
Undecided |
Medium |
|
| 2019-12-10 15:16:17 |
Peng Peng |
attachment added |
|
ALL_NODES_20191210.144348.tar https://bugs.launchpad.net/starlingx/+bug/1855474/+attachment/5311476/+files/ALL_NODES_20191210.144348.tar |
|
| 2019-12-11 02:10:16 |
Cristopher Lemus |
attachment added |
|
ALL_NODES_20191210.184122.tar https://bugs.launchpad.net/starlingx/+bug/1855474/+attachment/5311586/+files/ALL_NODES_20191210.184122.tar |
|
| 2019-12-11 08:28:41 |
yong hu |
starlingx: assignee |
zhipeng liu (zhipengs) |
yong hu (yhu6) |
|
| 2019-12-11 15:24:24 |
Ghada Khalil |
starlingx: importance |
Medium |
High |
|
| 2019-12-11 15:24:40 |
Ghada Khalil |
starlingx: status |
New |
Triaged |
|
| 2019-12-16 05:24:29 |
yong hu |
starlingx: status |
Triaged |
In Progress |
|
| 2019-12-18 13:15:40 |
Bart Wensley |
bug |
|
|
added subscriber Bart Wensley |
| 2019-12-24 15:31:41 |
Peng Peng |
attachment added |
|
ALL_NODES_20191224.145835.tar https://bugs.launchpad.net/starlingx/+bug/1855474/+attachment/5315101/+files/ALL_NODES_20191224.145835.tar |
|
| 2020-02-18 02:17:00 |
zhipeng liu |
bug |
|
|
added subscriber zhipeng liu |
| 2020-02-28 14:54:47 |
Ghada Khalil |
tags |
stx.3.0 stx.containers stx.distro.openstack |
stx.3.0 stx.4.0 stx.containers stx.distro.openstack |
|
| 2020-03-26 12:32:50 |
Yatindra |
attachment added |
|
stx-openstack-apply_2020-03-25-16-44-41.log https://bugs.launchpad.net/starlingx/+bug/1855474/+attachment/5341810/+files/stx-openstack-apply_2020-03-25-16-44-41.log |
|
| 2020-04-21 14:59:47 |
OpenStack Infra |
starlingx: status |
In Progress |
Fix Released |
|
| 2020-04-28 13:13:25 |
Bill Zvonar |
bug |
|
|
added subscriber Bill Zvonar |
| 2020-08-13 20:55:56 |
Bill Zvonar |
tags |
stx.3.0 stx.4.0 stx.containers stx.distro.openstack |
stx.3.0 stx.4.0 stx.cherrypickneeded stx.containers stx.distro.openstack |
|
| 2020-08-27 20:35:37 |
Bill Zvonar |
tags |
stx.3.0 stx.4.0 stx.cherrypickneeded stx.containers stx.distro.openstack |
stx.3.0 stx.4.0 stx.containers stx.distro.openstack |
|
