Upgrade from Ocata to Pike breaks nova-api-metadata service due to old nova-cert service entries

[Drew Freiberger]

Because nova-cert service was deprecated and finally removed in Pike an later versions of Openstack [1], the nova-cloud-controller charm upgrade should remove the previously-registered, now-dead service from the nova database.

It was discovered during investigation of metadata issues on a post-upgraded Pike cloud that the nova-api-metadata service running on the neutron-gateway nodes was showing an error:

ServiceTooOld_Remote: This service is older (v15) than the minimum (v22) version of the rest of the deployment. Unable to continue.

This error seems to be a sanity check done in nova-api-metadata that is checking all services for version equal to current version.

In querying the database for services of that version 15 (select * from services where version = '15';) it returns the nova-cert binary services.

Workaround:

openstack compute service list -f csv -c ID -c Binary| \
   grep nova-cert| \
   cut -d, -f1 | \
   xargs -I@ openstack compute service delete @

[1] - https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/remove-nova-cert.html

[Drew Freiberger (afreiberger)]

This may be a semi-red-herring. the service needs to be removed from the database to not confuse operators and monitoring tools, but the reasoning behind it affecting nova-api-metadata on the neutron-gateway nodes are incorrect. It turns out my issue with the ServiceTooOld message is that my cloud-archive on the neutron-gateway is stuck at newton when the charm is set to xenial-pike.

[Andrea Ieri (aieri)]

Here's a slightly easier deletion workaround:

openstack compute service list --service nova-cert -cId -fvalue | xargs -n1 openstack compute service delete