controller migration is very hard when dealing with large deployments

[Aymen Frikha]

Hello trying to do the controller migration from xenial controller to bionic one with a big openstack environment.
The migration keep failing because there is some units are in executing mode.

Is it possible to add a feature for juju to prevent all the agents from executing to be able to do the migration smoothly ?

We are using juju 2.8.9

[Joseph Phillips (manadart)]

There is something odd happening here. We should be allowing migration to proceed if the agent status is "idle" or "executing".

It is others that we stop for: allocating/rebooting/failed/lost.

If we encounter this again, I would very much like to see what the status actually is.

[Joseph Phillips (manadart)]

See here:
https://pastebin.ubuntu.com/p/YPqrVzrgwB/

When the migration-inactive-flag drops, a graceful shut-down should ensue.

Instead, we are throwing an error, changing the status to "failed" as seen here:
https://pastebin.ubuntu.com/p/d7XSfmQycN/

This means the QUIESC stage fails and the migration is aborted.

Fixing this will mean that we don't have to keep retrying migrations until we get lucky and pass the first validation step.

[Joseph Phillips (manadart)]

The juju/mutex package has an error type defined, ErrCancelled.

We just need to handle this specifically instead of throwing out with it.

[Joseph Phillips (manadart)]

Should be addressed by https://github.com/juju/juju/pull/13084

[Joseph Phillips (manadart)]

2.9 under https://github.com/juju/juju/pull/13085