Comment 4 for bug 1778305

Thanks for including the debug log.

This reminded me of another bug related to upgrading from Ocata => Pike and getting UUIDs generated for services for hosts on which instances are residing [1]. We can see the UUID generation happening in your nova-api log "Generated UUID 2dd86bf1-a805-42ef-9fb8-c94c9c3408fe for service 104 _from_db_object /usr/lib/python2.7/site-packages/nova/objects/service.py:245".

Having a look around the code, it looks like you have a situation where some of your instances are still associated in the database with the old, deleted=0 service record. In the link from Matt [2] about the table join, it's only filtering out Instance.deleted=0, *not* Service.deleted=0. So it looks like there's nothing preventing a join of the 'instances' and 'services' tables including deleted services.

During a 'nova list', the services associated with the hosts on which the instances reside will be queried and if they lack a UUID, one will be generated and saved. During the save [3] is where we hit the ServiceTooOld exception (which is logged as "This service is older (v9) than the minimum (v16) version of the rest of the deployment. Unable to continue.").

So, it seems like we need to do something to avoid pulling deleted services when we join the 'instances' and 'services' tables.

I'm also reminded of another bug related to this where we made a change to block deletion of a service that is still hosting instances [4]. That means it will force you to have to migrate or delete your instances associated with that compute host before allowing the service to be deleted. That patch landed in stable/pike a few days ago and has *not* yet been released.

[1] https://bugs.launchpad.net/nova/pike/+bug/1746509
[2] https://github.com/openstack/nova/blob/stable/pike/nova/db/sqlalchemy/models.py#L103
[3] https://github.com/openstack/nova/blob/16.1.4/nova/objects/service.py#L244-L246
[4] https://bugs.launchpad.net/nova/+bug/1763183