after upgrading model from 1.25 to v2.x, juju migrate fails on lxd containers

Bug #1834404 reported by Xav Paice on 2019-06-27
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju
Medium
Unassigned

Bug Description

Juju client 2.6.4, current model (and controller) 2.6.4 all using Trusty.

The original install was 1.2x, maas 1.9, with multiple units using LXC containers.

This particular model was upgraded to 2.x, and the containers migrated to lxd as part of that process.

When I try to migrate the model to a new controller (Xenial), I get the following:

:~/bootstack-backups$ juju migrate --debug mymodel newcontroller
00:17:21 INFO juju.cmd supercommand.go:57 running juju [2.6.4 gc go1.10.4]
00:17:21 DEBUG juju.cmd supercommand.go:58 args: []string{"/snap/juju/8241/bin/juju", "migrate", "--debug", "mymodel", "newcontroller"}
00:17:21 INFO juju.juju api.go:67 connecting to API addresses: [10.101.52.156:17070]
00:17:21 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.101.52.156:17070/api"
00:17:21 INFO juju.api apiclient.go:624 connection established to "wss://10.101.52.156:17070/api"
00:17:21 DEBUG juju.api monitor.go:35 RPC connection died
00:17:21 INFO juju.juju api.go:67 connecting to API addresses: [10.101.52.169:17070]
00:17:21 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.101.52.169:17070/api"
00:17:21 INFO juju.api apiclient.go:624 connection established to "wss://10.101.52.169:17070/api"
00:17:21 DEBUG juju.api monitor.go:35 RPC connection died
00:17:21 INFO juju.juju api.go:67 connecting to API addresses: [10.101.52.156:17070]
00:17:21 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.101.52.156:17070/api"
00:17:21 INFO juju.api apiclient.go:624 connection established to "wss://10.101.52.156:17070/api"
00:17:21 DEBUG juju.api monitor.go:35 RPC connection died
ERROR source prechecks failed: machine 0/lxd/0 not running
00:17:21 DEBUG cmd supercommand.go:496 error stack:
source prechecks failed: machine 0/lxd/0 not running
/build/juju/parts/juju/go/src/github.com/juju/juju/api/controller/controller.go:333:

The container mentioned is one of the containers that was migrated from lxc.

Digging in the db using db.statuses.find( {_id: { $regex: /0\/lxd\// }}).pretty() I see that the 'status' and 'statusinfo' fields aren't populated on containers that were migrated. Restarting the machine agent doesn't seem to affect this.

Good status:

{
        "_id" : "43759692-2e64-4550-8dd4-59d1572213e6:m#0/lxd/17#instance",
        "model-uuid" : "43759692-2e64-4550-8dd4-59d1572213e6",
        "neverset" : false,
        "status" : "running",
        "statusdata" : {

        },
        "statusinfo" : "Container started",
        "txn-queue" : [ ],
        "txn-revno" : NumberLong(52),
        "updated" : NumberLong("1556832174380495695")
}

Not so good:

{
        "_id" : "43759692-2e64-4550-8dd4-59d1572213e6:m#0/lxd/14#instance",
        "model-uuid" : "43759692-2e64-4550-8dd4-59d1572213e6",
        "neverset" : false,
        "status" : "",
        "statusdata" : {

        },
        "statusinfo" : "",
        "txn-queue" : [ ],
        "txn-revno" : NumberLong(2),
        "updated" : NumberLong("1538373101870144741")
}

juju show-machine shows something very similar:

Good:
      0/lxd/17:
        juju-status:
          current: started
          since: 26 Jun 2019 04:30:57Z
          version: 2.6.4
        dns-name: 10.101.52.166
        ip-addresses:
        - 10.101.52.166
        - 10.101.60.1
        instance-id: juju-2213e6-0-lxd-17
        machine-status:
          current: running
          message: Container started
          since: 02 May 2019 21:22:54Z
        modification-status:
          current: applied
          since: 26 Jun 2019 05:32:03Z
        series: trusty
        network-interfaces:
          eth0:
            ip-addresses:
            - 10.101.52.166
            mac-address: 00:16:3e:35:93:03
            gateway: 10.101.52.1
            dns-nameservers:
            - 10.101.52.6
            space: space-0
            is-up: true
          eth1:
            ip-addresses:
            - 10.101.60.1
            mac-address: 00:16:3e:d7:79:3e
            space: space-0
            is-up: true

Not so good:
      0/lxd/0:
        juju-status:
          current: started
          since: 26 Jun 2019 04:30:05Z
          version: 2.6.4
        dns-name: 10.101.52.90
        ip-addresses:
        - 10.101.52.90
        instance-id: juju-2213e6-0-lxd-0
        machine-status:
          since: 01 Oct 2018 05:51:41Z
        modification-status:
          current: applied
          since: 26 Jun 2019 05:32:03Z
        series: trusty
        hardware: arch=amd64

      0/lxd/14:
        juju-status:
          current: started
          since: 26 Jun 2019 04:30:07Z
          version: 2.6.4
        dns-name: 10.101.52.96
        ip-addresses:
        - 10.101.52.241
        - 10.101.52.96
        instance-id: juju-2213e6-0-lxd-14
        machine-status:
          since: 01 Oct 2018 05:51:41Z
        modification-status:
          current: applied
          since: 26 Jun 2019 05:32:03Z
        series: trusty
        hardware: arch=amd64

(note also the lack of network-interfaces)

Tim Penhey (thumper) wrote :

We need to find out why the machine agent on the lxd machines aren't updating the link-layer devices or the provider status.

Changed in juju:
status: New → Triaged
importance: Undecided → Medium
tags: added: lxd model-migration
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers