key distribution for live migration scales poorly

[Paul Collins]

I recently upgraded an OpenStack deployment of the ~15.04 charms to 17.11. During the upgrade we noticed that nova-cloud-controller was spending a lot of time in "configuring live migration".

This turned out to be because it was spending approximately 40 seconds on each of the 60 units in our nova-compute-cache service and, for each unit, it was running relation-set over 1110 times (increasing by 4 for each subsequent unit, strangely).

This nova-cloud-controller is also related to a nova-compute service with only four units, and when handling keys for those units it only ran relation-set 84 times for the first unit, again increasing by 4 for each subsequent unit. This all added up to an additional 30 or 40 minutes spent upgrading this one charm.

Here's a breakdown of the relation-get/relation-set calls during the first half or so of this upgrade: https://pastebin.canonical.com/p/mznDWrZqQK/ (sorry, link is Canonical-only).

[Simon Monette (simon-monette)]

This issue can also be reproduced during a deployment.

With cs:nova-cloud-controller-302 on xenial-newton
 and cs:nova-compute-279 on xenial-newton and enable-live-migration=true

It can add-up to 4h of wait-time on stack with ~100 computes.

[James Page (james-page)]

We might be better to review TLS configuration of libvirt and change the way we do live migration rather than tackle this specific issue.

With the new certificate management work being done with Vault, configuring the libvirt daemons directly with TLS via a trusted CA should be much easier, requiring a 1 -> N set of hook executions rather than the N x N that we have today.

References: https://wiki.libvirt.org/page/TLSSetup

[Alex Kavanagh (ajkavanagh)]

This is effectively solved by the 19.07 release of the nova-cloud-controller which has optimisations to reduce the time taken for upgrades (it doesn't re-evaluate all of the knownhosts during an upgrade).