Upgrade from Ocata to Pike caused instance listing to fail

[Andrea Ieri]

This really seems awfully similar to LP#1809210 (and it happened in the same site): after the upgrade listing all instances yielded the following failure:

This service is older (v9) than the minimum (v16) version of the rest of the deployment. Unable to continue. (HTTP 400)

Looking at the nova database, indeed there were some stale entries with version=9:

https://pastebin.canonical.com/p/DDs4GFrNxw/

The juju-machine-<n>-lxc-<m> hosts did not even exist anymore in the model, but setting them to disabled (manually, `openstack host set --disable <hostname>` did not work) did not help.

Instance listing resumed working only after all version=9 entries were deleted.

[James Page (james-page)]

@aieri

I'm assuming that the v9 based services and the machines hosting them where around at some point in time; the 'Service' db code checks the version and raises this exception for everything that does not meet the minimum requirement hence why deleting the entries resolved the issue.

I think we at a minimum need to add some pre-flight check notes to the charm deployment guide to purge any down/old service entries from nova prior to upgrades. This can probably be done in advance of an upgrade but obviously can't be done post upgrade as the API will continue to throw these errors.

[James Page (james-page)]

a "openstack compute service delete" might be possible to clean up the old, versioned services.

[James Page (james-page)]

deployment guide update: https://review.openstack.org/631209

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/631209
Committed: https://git.openstack.org/cgit/openstack/charm-deployment-guide/commit/?id=5b82aa481447bdd033d8250f851422642ee46db3
Submitter: Zuul
Branch: master

commit 5b82aa481447bdd033d8250f851422642ee46db3
Author: James Page <email address hidden>
Date: Wed Jan 16 14:42:21 2019 +0200

    upgrades: purge old compute service entries

    Add pre-upgrade step to discover and purge and old compute service
    entries; in later openstack releases, the DB model will throw
    exceptions for service entries with versions less than the minimum
    requirement, causing API requests to error post upgrade.

    Change-Id: Id739f0f23aaadf08dff216bb9f17552deee1b0a4
    Closes-Bug: 1809260