Stein->Train Upgrade : Old nova services not cleaned

[Keith Plant]

**This appears to be entirely superficial, I haven't been any to find any functional fallout from this behavior**

When upgrading from stable/stein to stable/train, the old nova-consoleauth and nova-scheduler services are not cleaned up:

Before upgrade: http://paste.openstack.org/show/82jvGkP67YZkzUl3dCJS/
After upgrade: http://paste.openstack.org/show/fNwE00IDdvzX4On8ZWyc/

**Environment**:
OS: CentOS 7.7.1908
Kernel: 3.10.0-1062.9.1.el7.x86_64
Docker version: 18.09.7, build 2d0083d
kolla-ansible version: stable/train (9.0.1-8-g4358395)
docker image type: source
docker image dist: centos7

[Mark Goddard (mgoddard)]

The nova-consoleauth container was removed in Train, so we should delete those services. I don't know why you have new scheduler services though. Did something change in the configuration? Perhaps something to do with the cells refactor?

[Keith Plant (kplant)]

Not to my knowledge, I may have missed some necessary configuration steps. I merged the stable/train globals.yml with the stable/stein globals.yml by doing a stare-and-compare with vimdiff.

[Radosław Piliszek (yoctozepto)]

I confirm this happens despite the same config. Wonder what happens with those schedulers.

[xinliang (xin3liang)]

I also met the same issue, when do a stein->train upgrade.

[Oliver Horecny (horecoli)]

I hit same issue during rolling upgrade Rocky->Stein->Train.

Since this change: https://github.com/openstack/kolla-ansible/commit/78a828ef42160d161f944308ec51af0303778ca7
nova-scheduler was moved to different DB than it was before. And after upgrade it is duplicated in openstack:

openstack compute service list --service nova-scheduler

+----+----------------+----------------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+----------------+----------------------------+----------+---------+-------+----------------------------+
| 1 | nova-scheduler | test-upgrade | internal | enabled | up | 2020-12-01T14:32:09.000000 |
| 4 | nova-scheduler | test-upgrade | internal | enabled | down | 2020-10-26T16:58:35.000000 |
+----+----------------+----------------------------+----------+---------+-------+----------------------------+

It is also visible in DB, where nova-scheduler is stored on two places:

MariaDB [(none)]> select created_at,updated_at,deleted_at,id,host,`binary` from nova.services where `binary`='nova-scheduler';
+---------------------+---------------------+------------+----+----------------------------+----------------+
| created_at | updated_at | deleted_at | id | host | binary |
+---------------------+---------------------+------------+----+----------------------------+----------------+
| 2020-09-29 13:19:35 | 2020-10-26 16:58:35 | NULL | 4 | test-upgrade | nova-scheduler |
+---------------------+---------------------+------------+----+----------------------------+----------------+

MariaDB [(none)]> select created_at,updated_at,deleted_at,id,host,`binary` from nova_cell0.services where `binary`='nova-scheduler';
+---------------------+---------------------+------------+----+----------------------------+----------------+
| created_at | updated_at | deleted_at | id | host | binary |
+---------------------+---------------------+------------+----+----------------------------+----------------+
| 2020-10-26 16:55:20 | 2020-12-01 13:35:39 | NULL | 1 | test-upgrade | nova-scheduler |
+---------------------+---------------------+------------+----+----------------------------+----------------+

After upgrade it seems, that openstack is fully functional, but that nova-scheduler which is in state "down" shouldn't be present in openstack. I think kolla-ansible should handle this situation by removing that old unused service from DB.
There is also additional unused service nova-consoleauth which was used in previous releases and need to be removed from DB after upgrade. So it can be done in same step.

[Radosław Piliszek (yoctozepto)]

Patch in gerrit: https://review.opendev.org/c/openstack/kolla-ansible/+/765347