Migration fails with stack trace

[Peter Sabaini]

When running juju migrate I'm getting an abort, "migrating: aborted, removing model from target controller: model data transfer failed, failed to import model into target controller: assignment to entry in nil map". This seems reproduceable, I've tried a few times with same result.

On the target controller I can see a stack trace

9 10 32 32 105 110 102 111 58 32 34 34 10] Charms:[] Tools:[] Resources:[]}: assignment to entry in nil map
goroutine 2085609 [running]:
runtime/debug.Stack(0xc00bd88a68, 0x3ada580, 0x4d82ab0)
        /snap/go/5365/src/runtime/debug/stack.go:24 +0x9d
github.com/juju/juju/rpc.(*Conn).runRequest.func1(0xc00bd89f28, 0x3e62c80, 0xc0120f59e0, 0x199, 0xc0019641e0, 0x4e63ec0, 0xc01e0632e0)
        /workspace/_build/src/github.com/juju/juju/rpc/server.go:558 +0xe2
panic(0x3ada580, 0x4d82ab0)
        /snap/go/5365/src/runtime/panic.go:522 +0x1b5
github.com/juju/juju/state.(*importer).parseBindings(0xc00be05620, 0x0, 0xc00ddb50e0, 0xc0045de380, 0x0)
        /workspace/_build/src/github.com/juju/juju/state/migration_import.go:933 +0x1b1
github.com/juju/juju/state.(*importer).application(0xc00be05620, 0x4f7c1a0, 0xc04b9f29c0, 0x0, 0x0)
        /workspace/_build/src/github.com/juju/juju/state/migration_import.go:853 +0x560
github.com/juju/juju/state.(*importer).applications(0xc00be05620, 0x0, 0x0)
        /workspace/_build/src/github.com/juju/juju/state/migration_import.go:756 +0x29c
github.com/juju/juju/state.(*Controller).Import(0xc006752270, 0x4f874e0, 0xc004af8700, 0x0, 0x0, 0x0, 0x0)
        /workspace/_build/src/github.com/juju/juju/state/migration_import.go:188 +0xbb5
github.com/juju/juju/migration.ImportModel(0x4e1c4c0, 0xc006752270, 0xc001c32700, 0xc058116000, 0x324dbcb, 0x324dbcc, 0x40e5c9, 0xc000484480, 0x80, 0xc001fa1a60)
        /workspace/_build/src/github.com/juju/juju/migration/migration.go:71 +0xab
github.com/juju/juju/apiserver/facades/controller/migrationtarget.(*API).Import(0xc000898180, 0xc058116000, 0x324dbcb, 0x324dbcc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /workspace/_build/src/github.com/juju/juju/apiserver/facades/controller/migrationtarget/migrationtarget.go:115 +0xaf
reflect.Value.call(0x3f23c20, 0xc000898180, 0x1613, 0x429b811, 0x4, 0xc001c32720, 0x1, 0x1, 0xa31901, 0x3946d80, ...)
        /snap/go/5365/src/reflect/value.go:447 +0x461
reflect.Value.Call(0x3f23c20, 0xc000898180, 0x1613, 0xc001c32720, 0x1, 0x1, 0x1, 0x1, 0xc000972e00)
        /snap/go/5365/src/reflect/value.go:308 +0xa4
github.com/juju/juju/vendor/github.com/juju/rpcreflect.newMethod.func6(0x4ebc300, 0xc00210c9c0, 0x3f23c20, 0xc000898180, 0x16, 0x3e62c80, 0xc0120f59e0, 0x199, 0x42f00f, 0xc000000000, ...)
        /workspace/_build/src/github.com/juju/juju/vendor/github.com/juju/rpcreflect/type.go:327 +0x106
github.com/juju/juju/apiserver.(*srvCaller).Call(0xc00210c900, 0x4ebc300, 0xc00210c9c0, 0x0, 0x0, 0x3e62c80, 0xc0120f59e0, 0x199, 0xc01e0632e0, 0x0, ...)
        /workspace/_build/src/github.com/juju/juju/apiserver/root.go:171 +0xd6
github.com/juju/juju/rpc.(*Conn).runRequest(0xc0019641e0, 0x4e85dc0, 0xc00210c900, 0x4433ab8, 0x2, 0xc002294f50, 0xf, 0x1, 0x0, 0x0, ...)
        /workspace/_build/src/github.com/juju/juju/rpc/server.go:571 +0x1ab
created by github.com/juju/juju/rpc.(*Conn).handleRequest
        /workspace/_build/src/github.com/juju/juju/rpc/server.go:475 +0x999

Both source and target controller are on 2.7.5, non-HA

Context: trying to upgrade the controller machine, source is on Xenial, target controller Bionic.

[Peter Sabaini (peter-sabaini)]

Subscribing field-high as this blocks a prod upgrade.

[Peter Sabaini (peter-sabaini)]

I've uploaded machine-0.log of the target controller here (Canonical-only, sorry):

https://private-fileshare.canonical.com/~sabaini/lp1871066/f02e4bc3-6095-4da4-9621-458f28482aa4/target-machine-0.log.bz2

[Richard Harding (rharding)]

Looks like the part of the code hitting it was parseBindings which means there's something with the endpoint bindings that the code isn't happy with. Joe can you peek at this when you get to a good point please?

Peter, can you note anything of interest regarding network bindings and the applications in the model that is failing?

[Peter Sabaini (peter-sabaini)]

Hey,

The model is an Openstack deploy with MAAS provider.

I've placed a sanitized bundle here if this helps:

https://private-fileshare.canonical.com/~sabaini/lp1871066/f02e4bc3-6095-4da4-9621-458f28482aa4/bundle-sanitized.yaml

It was originally deployed with Juju 2.5.1, has been upgraded recently.

Anything else of interest please ping me here or in IRC

[Joseph Phillips (manadart)]

Proposed for 2.7:
https://github.com/juju/juju/pull/11421

[Peter Sabaini (peter-sabaini)]

Hey, I see a fix has been merged -- much appreciated!
Do you have an ETA for a 2.7.6 release?

[Richard Harding (rharding)]

As soon as we can get it. Hopefully we can get something by the end of the
week.

On Tue, Apr 7, 2020 at 3:05 PM Peter Sabaini <email address hidden>
wrote:

> Hey, I see a fix has been merged -- much appreciated!
> Do you have an ETA for a 2.7.6 release?
>
> --
> You received this bug notification because you are subscribed to juju.
> https://bugs.launchpad.net/bugs/1871066
>
> Title:
> Migration fails with stack trace
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1871066/+subscriptions
>

[Joseph Phillips (manadart)]

Edge: https://github.com/juju/juju/pull/11429