nova-consoleauth is down in compute service list after Openstack upgrade

[Gábor Mészáros]

While doing Openstack upgrades (queens -> ussuri) the nova-cloud-controller removes the deprecated nova-consoleauth service from the units (as per https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1848478)

But when doing an openstack compute service list, it shows the endpoints there as down, instead of them being removed.
$ openstack compute service list | grep down
| 7 | nova-consoleauth | juju-2bff36-21-lxd-11 | internal | enabled | down | 2021-11-12T15:46:29.000000 |
| 28 | nova-consoleauth | juju-2bff36-19-lxd-10 | internal | enabled | down | 2021-11-12T15:46:18.000000 |
| 130 | nova-consoleauth | juju-2bff36-20-lxd-10 | internal | enabled | down | 2021-11-12T15:46:19.000000 |

These services should be removed when doing the Charmed Openstack Train upgrade.

[Aurelien Lourot (aurelien-lourot)]

Can be reproduced with the latest charms by running the OpenStack upgrade tests from Stein to Train: [1]

$ tox -e openstack-upgrade kitchen-sink-bionic-stein

App Version Status Scale Charm Store Rev OS Notes
ceilometer 13.1.2 active 1 ceilometer jujucharms 412 ubuntu
ceilometer-agent 13.1.2 active 3 ceilometer-agent jujucharms 366 ubuntu
ceph-mon 14.2.22 active 3 ceph-mon jujucharms 487 ubuntu
ceph-osd 14.2.22 active 3 ceph-osd jujucharms 506 ubuntu
cinder 15.6.0 active 1 cinder jujucharms 453 ubuntu
cinder-ceph 15.6.0 active 1 cinder-ceph jujucharms 362 ubuntu
glance 19.0.4 active 1 glance jujucharms 487 ubuntu
gnocchi 4.3.4 active 1 gnocchi jujucharms 148 ubuntu
keystone 16.0.2 active 1 keystone jujucharms 585 ubuntu
memcached active 1 memcached jujucharms 34 ubuntu
mysql 5.7.20 active 1 percona-cluster jujucharms 302 ubuntu
neutron-api 15.3.4 active 1 neutron-api jujucharms 490 ubuntu
neutron-gateway 15.3.4 active 1 neutron-gateway jujucharms 469 ubuntu
neutron-openvswitch 15.3.4 active 3 neutron-openvswitch jujucharms 457 ubuntu
nova-cloud-controller 20.6.1 active 1 nova-cloud-controller jujucharms 560 ubuntu
nova-compute 20.6.1 active 3 nova-compute jujucharms 587 ubuntu
placement 2.0.1 active 1 placement jujucharms 67 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 450 ubuntu
swift-proxy 2.23.3 active 1 swift-proxy jujucharms 239 ubuntu
swift-storage-z1 2.23.3 active 1 swift-storage jujucharms 395 ubuntu
swift-storage-z2 2.23.3 active 1 swift-storage jujucharms 395 ubuntu
swift-storage-z3 2.23.3 active 1 swift-storage jujucharms 395 ubuntu

$ openstack compute service list
+----+------------------+------------------------------------------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+------------------------------------------------------+----------+---------+-------+----------------------------+
| 1 | nova-consoleauth | juju-7c141d-zaza-21897d1d9780-15 | internal | enabled | down | 2021-12-17T10:27:19.000000 |
| 2 | nova-conductor | juju-7c141d-zaza-21897d1d9780-15 | internal | enabled | up | 2021-12-17T14:14:17.000000 |
| 4 | nova-scheduler | juju-7c141d-zaza-21897d1d9780-15 | internal | enabled | up | 2021-12-17T14:14:16.000000 |
| ...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/822105

[Aurelien Lourot (aurelien-lourot)]

I can't find an easy way to make the service disappear from the list. It seems like purging the nova-consoleauth package and restarting all nova services from all nova-cloud-controller and nova-compute units doesn't help. I'm wondering if some cleanup is missing in Nova and if this should be a Nova bug?

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Aurelien Lourot <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/822105
Reason: This doesn't fix the original issue, see https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1951863/comments/3

[Billy Olsen (billy-olsen)]

I think issuing a `nova service delete` for the appropriate services after the upgrade would suffice here.