Comment 5 for bug 1863107

I hit same issue during rolling upgrade Rocky->Stein->Train.

Since this change: https://github.com/openstack/kolla-ansible/commit/78a828ef42160d161f944308ec51af0303778ca7
nova-scheduler was moved to different DB than it was before. And after upgrade it is duplicated in openstack:

openstack compute service list --service nova-scheduler

+----+----------------+----------------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+----------------+----------------------------+----------+---------+-------+----------------------------+
| 1 | nova-scheduler | test-upgrade | internal | enabled | up | 2020-12-01T14:32:09.000000 |
| 4 | nova-scheduler | test-upgrade | internal | enabled | down | 2020-10-26T16:58:35.000000 |
+----+----------------+----------------------------+----------+---------+-------+----------------------------+

It is also visible in DB, where nova-scheduler is stored on two places:

MariaDB [(none)]> select created_at,updated_at,deleted_at,id,host,`binary` from nova.services where `binary`='nova-scheduler';
+---------------------+---------------------+------------+----+----------------------------+----------------+
| created_at | updated_at | deleted_at | id | host | binary |
+---------------------+---------------------+------------+----+----------------------------+----------------+
| 2020-09-29 13:19:35 | 2020-10-26 16:58:35 | NULL | 4 | test-upgrade | nova-scheduler |
+---------------------+---------------------+------------+----+----------------------------+----------------+

MariaDB [(none)]> select created_at,updated_at,deleted_at,id,host,`binary` from nova_cell0.services where `binary`='nova-scheduler';
+---------------------+---------------------+------------+----+----------------------------+----------------+
| created_at | updated_at | deleted_at | id | host | binary |
+---------------------+---------------------+------------+----+----------------------------+----------------+
| 2020-10-26 16:55:20 | 2020-12-01 13:35:39 | NULL | 1 | test-upgrade | nova-scheduler |
+---------------------+---------------------+------------+----+----------------------------+----------------+

After upgrade it seems, that openstack is fully functional, but that nova-scheduler which is in state "down" shouldn't be present in openstack. I think kolla-ansible should handle this situation by removing that old unused service from DB.
There is also additional unused service nova-consoleauth which was used in previous releases and need to be removed from DB after upgrade. So it can be done in same step.