model upgrade tries to upgrade the lxd profile and fails

[Xav Paice]

Client 2.6.4-trusty-amd64 (snap). Controller is one (trusty) machine.

Tried to upgrade a model from 2.5.7 to 2.6.4, and all the lxd units are now reporting their machine status as 'error' with the message "cannot upgrade machine's lxd profile: 0/lxd/0: not found".

The machine-0.log on the host shows:

2019-06-26 04:29:49 INFO juju.cmd supercommand.go:57 running jujud [2.6.4 gc go1.11.11]
2019-06-26 04:29:49 DEBUG juju.cmd supercommand.go:58 args: []string{"/var/lib/juju/tools/machine-0/jujud", "machine", "--data-dir", "/var/lib/juju", "--machine-id", "0", "--debug"}
2019-06-26 04:29:49 DEBUG juju.utils gomaxprocs.go:24 setting GOMAXPROCS to 40
2019-06-26 04:29:49 DEBUG juju.agent agent.go:545 read agent config, format "2.0"
2019-06-26 04:29:49 INFO juju.cmd.jujud agent.go:133 setting logging config to "<root>=WARNING; unit=INFO"
2019-06-26 04:29:57 WARNING juju.worker.proxyupdater proxyupdater.go:231 unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
2019-06-26 04:30:06 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/4 lxd profiles: 0/lxd/4: not found
2019-06-26 04:30:06 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/0 lxd profiles: 0/lxd/0: not found
2019-06-26 04:30:07 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/14 lxd profiles: 0/lxd/14: not found
2019-06-26 04:30:08 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/1 lxd profiles: 0/lxd/1: not found
2019-06-26 04:30:09 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/5 lxd profiles: 0/lxd/5: not found
2019-06-26 04:30:09 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/6 lxd profiles: 0/lxd/6: not found
2019-06-26 04:30:10 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/3 lxd profiles: 0/lxd/3: not found
2019-06-26 04:30:10 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/9 lxd profiles: 0/lxd/9: not found
2019-06-26 04:30:10 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/10 lxd profiles: 0/lxd/10: not found
2019-06-26 04:30:10 ERROR juju.worker.dependency engine.go:663 "instance-mutater" manifold worker returned unexpected error: 0/lxd/4: not found
2019-06-26 04:30:11 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/8 lxd profiles: 0/lxd/8: not found
2019-06-26 04:30:11 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/11 lxd profiles: 0/lxd/11: not found
2019-06-26 04:30:12 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/12 lxd profiles: 0/lxd/12: not found
2019-06-26 04:30:12 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/7 lxd profiles: 0/lxd/7: not found
2019-06-26 04:30:21 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/5 lxd profiles: 0/lxd/5: not found
2019-06-26 04:30:25 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/1 lxd profiles: 0/lxd/1: not found
2019-06-26 04:30:26 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/3 lxd profiles: 0/lxd/3: not found
2019-06-26 04:30:27 ERROR juju.worker.instancemutater mutater.go:192 cannot upgrade machine-0/lxd/6 lxd profiles: 0/lxd/6: not found
2019-06-26 04:30:27 ERROR juju.worker.dependency engine.go:663 "instance-mutater" manifold worker returned unexpected error: 0/lxd/5: not found

... etc.

[Xav Paice (xavpaice)]

Note: this is due to a failure in the upgrade of the model previously from Juju 1 to Juju 2.

The machine IDs are listed as juju-machine-x and not juju-abc123-x (the model suffix). That breaks the lxd integration.

The fix is in the db:

db.instanceData.find( {'instanceid': { $regex: /juju-machine.*/ }}).forEach(
    function(doc) {
        var modelSuffix = doc["model-uuid"].substr(30,6);
        var newID = "juju-" + modelSuffix + doc["instanceid"].substr(12);
        print("updating "+ doc._id + " instanceid to " + newID);
        db.instanceData.update({_id: doc._id}, {$set: {"instanceid": newID}})
    })