After cluster reboot, nova-cloud-controller stays blocked with "Services not running that should be: nova-consoleauth"

[Jason Hobbs]

I rebooted my entire fcb cluster, then recovered mysql. However, nova-cloud-controller units remained in Blocked state, with status "Services not running that should be: nova-consoleauth"

I was able to get past the blocked state by sshing into each nova-cloud-controller unit and running "sudo service start nova-consoleauth"

http://paste.ubuntu.com/p/mG8hp8hr7F/

crashdump: http://people.canonical.com/~jhobbs/juju-crashdump-openstack-2019-05-14-11.57.47.tar.gz

[Aurelien Lourot (aurelien-lourot)]

Cannot reproduce by restarting all nodes at the same time of a bionic-rocky deployment with the current development charms. [1] The entire deployment always nicely recovers: all units get back to green eventually and the nova-consoleauth service is running. I attempted three times in a row.

App Version Status Scale Charm Store Rev OS Notes
ceph-mon 13.2.8 active 3 ceph-mon jujucharms 485 ubuntu
ceph-osd 13.2.8 active 3 ceph-osd jujucharms 505 ubuntu
ceph-radosgw 13.2.8 active 1 ceph-radosgw jujucharms 414 ubuntu
cinder 13.0.9 active 1 cinder jujucharms 452 ubuntu
cinder-ceph 13.0.9 active 1 cinder-ceph jujucharms 362 ubuntu
glance 17.0.0 active 1 glance jujucharms 486 ubuntu
keystone 14.2.0 active 1 keystone jujucharms 582 ubuntu
mysql 5.7.20 active 1 percona-cluster jujucharms 405 ubuntu
neutron-api 13.0.7 active 1 neutron-api jujucharms 488 ubuntu
neutron-gateway 13.0.7 active 1 neutron-gateway jujucharms 469 ubuntu
neutron-openvswitch 13.0.7 active 2 neutron-openvswitch jujucharms 456 ubuntu
nova-cloud-controller 18.3.0 active 1 nova-cloud-controller jujucharms 559 ubuntu
nova-compute 18.3.0 active 2 nova-compute jujucharms 586 ubuntu
ntp 3.2 active 3 ntp jujucharms 47 ubuntu
openstack-dashboard 14.1.0 active 1 openstack-dashboard jujucharms 519 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 449 ubuntu

I'll now try with an older release of the charms, as close as possible with the OP's used charm versions.

[1]: https://github.com/AurelienLourot/openstack-bundles/blob/bug/1829001/development/openstack-base-bionic-rocky/bundle.yaml

[Aurelien Lourot (aurelien-lourot)]

Same with the release 18.11 of the charms: [1] can't reproduce. I believe this wasn't a bug in the charm but in the nova-consoleauth service itself (possibly a wrong dependency or start condition), which must have been fixed meanwhile.

App Version Status Scale Charm Store Rev OS Notes
ceph-mon 13.2.8 active 3 ceph-mon jujucharms 31 ubuntu
ceph-osd 13.2.8 active 3 ceph-osd jujucharms 273 ubuntu
ceph-radosgw 13.2.8 active 1 ceph-radosgw jujucharms 262 ubuntu
cinder 13.0.9 active 1 cinder jujucharms 276 ubuntu
cinder-ceph 13.0.9 active 1 cinder-ceph jujucharms 238 ubuntu
glance 17.0.0 active 1 glance jujucharms 271 ubuntu
keystone 14.2.0 active 1 keystone jujucharms 288 ubuntu
mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 272 ubuntu
neutron-api 13.0.7 active 1 neutron-api jujucharms 266 ubuntu
neutron-gateway 13.0.7 active 1 neutron-gateway jujucharms 256 ubuntu
neutron-openvswitch 13.0.7 active 2 neutron-openvswitch jujucharms 255 ubuntu
nova-cloud-controller 18.3.0 active 1 nova-cloud-controller jujucharms 316 ubuntu
nova-compute 18.3.0 active 2 nova-compute jujucharms 290 ubuntu
ntp 3.2 active 3 ntp jujucharms 31 ubuntu
openstack-dashboard 14.1.0 active 1 openstack-dashboard jujucharms 271 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 82 ubuntu

[1]: https://github.com/AurelienLourot/openstack-bundles/blob/bug/1829001/development/openstack-base-bionic-rocky/bundle-18.11.yaml