`juju migrate` fails with source prechecks failed ERROR

[james beedy]

Migrating model fails with http://paste.ubuntu.com/25139932/

$ juju debug-log -m controller --replay < http://paste.ubuntu.com/25139943/

[John A Meinel (jameinel)]

That is saying that it is missing a resource for one of your applications.
Migrate wants to make sure that all the objects are complete locally
(machines aren't pending, application resources are available, etc)

is there a reason you expect not to have that resource?

John
=:->

On Jul 21, 2017 17:50, "james beedy" <email address hidden> wrote:

> Public bug reported:
>
> Migrating model fails with http://paste.ubuntu.com/25139932/
>
> $ juju debug-log -m controller --replay <
> http://paste.ubuntu.com/25139943/
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1705730
>
> Title:
> `juju migrate` fails with source prechecks failed ERROR
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1705730/+subscriptions
>

[james beedy (jamesbeedy)]

@jameinel everything around that resource seems to be as working as expected http://paste.ubuntu.com/25155437/

[james beedy (jamesbeedy)]

just realized that model is on 2.0.2. I'm going to give it an upgrade and see what gives.

[james beedy (jamesbeedy)]

something to note here, upgrading the model forced the 'config-changed' hook to fire it seems

[james beedy (jamesbeedy)]

The following hooks fire on model upgrade, both from 2.0.2 -> 2.1.3, and from 2.1.3 to 2.2.2.2:

leader-elected, leader-settings-changed, config-changed

http://paste.ubuntu.com/25155490/

I would expect I'm able to migrate this model now its > 2.0.2.

Update: Even with the source model and controller on 2.2.2.2, and the target controller on 2.2.2.2, I still get the source precheck failure for feed-web-integration :(

[Christian Muirhead (2-xtian)]

I tried migrating a model using locally-provided resources but I can't reproduce this. It seems like that resource is stuck pending, but I'm not sure how that would happen (or how it wouldn't get retried). Could you show the output from running db.resources.find().pretty() in Mongo?

[james beedy (jamesbeedy)]

@2-xtian http://paste.ubuntu.com/25166028/

[Christian Muirhead (2-xtian)]

Hi James - sorry, I missed this update. Unfortunately that doesn't match what we thought might be happening. We'd guessed the part that had failed was a unit downloading a resource, but here the pending-ids are on application-level records.

[Christian Muirhead (2-xtian)]

I think the workaround here is to delete the 2 pending-id resources records, and the needed code change is that adding the resource manually (using attach) should remove any pending resource record as if the pending operation had completed.

[Christian Muirhead (2-xtian)]

None of which really explains why the pending records are stuck there in the first place - maybe a network problem or something while they were being uploaded?

[Christian Muirhead (2-xtian)]

Reading the code for resource handling more I think I understand what's probably happened here. Pending resources for an application are created during deployment before the application itself is created, and then as part of the application creation they are "resolved" - the pending resource record is deleted and replaced with a corresponding non-pending record.

See the call to resources.NewResolvePendingResourcesOps in State.AddApplication. https://github.com/juju/juju/blob/develop/state/state.go#L1294

It looks like if the AddApplication call fails for some reason there is nothing to clean these pending records up. I'll add code to the deploy command to remove them if creating the application fails. I'm not sure what the best resolution is if the deploy command is prevented from doing (say by a network error). Maybe there's a worker that runs periodic tasks for this kind of cleanup (not to be confused with specific cleanups).

[Christian Muirhead (2-xtian)]

I won't bother with the cleanup task. I'll change the migration precheck not to complain about pending application-level resources for an application that exists (they're obviously remnants of this bug). I think that and adding cleanup in the deploy command will be enough.

[Christian Muirhead (2-xtian)]

PR with the fix for 2.2.3: https://github.com/juju/juju/pull/7759

Rather than cleanup in the deploy command, the cleanup's done in the Deploy API method (so the client doesn't need to make another API call to remove the resources).

[james beedy (jamesbeedy)]

@wallyworld @2-xtian @jameinel many thanks!