model migration from 2.2.4 to 2.2.4 fails

[Paul Gear]

I tested model migration from a 2.2.2 to 2.2.4 controller today. It failed twice with the same status message: "migrating: aborted, removing model from target controller".

The controllers are both in the same tenant on an OpenStack cloud.

[Paul Gear (paulgear)]

Logs for the source and target controllers, along with a yaml status of the model which I attempted to migrate are at https://private-fileshare.canonical.com/~paulgear/lp1717860/

[Christian Muirhead (2-xtian)]

Looking in the log reveals:
2017-09-18 06:39:40 ERROR juju.worker.migrationmaster.2780ee worker.go:279 model data transfer failed, model export failed: failed to read status history collection: Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.

The status history collection didn't have an index that was needed by migration - {"model-uuid", "-updated", "-_id"}
This was fixed in https://github.com/juju/juju/pull/7722 (so not in 2.2.2, but in 2.2.3+).

[Christian Muirhead (2-xtian)]

I think Paul's problem has been solved by upgrading to 2.2.4 - a workaround would have been to add the index in the mongo shell.

[Paul Gear (paulgear)]

I upgraded to 2.2.4 on the source controller and tried again. This failed also; logs uploaded to https://private-fileshare.canonical.com/~paulgear/lp1717860/1717860-source-try2.tar.gz

[Paul Gear (paulgear)]

Try 3 failed; logs at https://private-fileshare.canonical.com/~paulgear/lp1717860/migration-3.tar.gz

[Paul Gear (paulgear)]

After the 3rd failed attempt at migration, the source controller became much less responsive - 260 seconds to run "juju models" on a controller with only 2 models: https://pastebin.canonical.com/199228/ Rebooting that controller got the time down to around 5 seconds.

[Paul Gear (paulgear)]

Dug into this today with @2-xtian; new logs uploaded to https://private-fileshare.canonical.com/~paulgear/lp1717860/ - he found an issue with the OpenStack provider which resulted in the target controller not able to fully adopt the migrated model.

[Christian Muirhead (2-xtian)]

PR for 2.3-beta2, hopefully.
https://github.com/juju/juju/pull/7962