Comment 4 for bug 1808034

When upgrading from Juju 1 to Juju 2 on a cloud that hosts LXD containers, the upgrade process updates the instanceid field in the instanceData collection incorrectly. You need to fix this if you want to be able to remove a unit that is an LXD container. This can be corrected as follows:

Get a connection to the Juju database, this is a handy script to do so:

##################

#!/bin/bash

machine=${1:-0}
model=${2:-controller}

read -d '' -r cmds <<'EOF'
conf=/var/lib/juju/agents/machine-*/agent.conf
user=`sudo grep tag $conf | cut -d' ' -f2`
password=`sudo grep statepassword $conf | cut -d' ' -f2`
/usr/lib/juju/mongo*/bin/mongo 127.0.0.1:37017/juju --authenticationDatabase admin --ssl --sslAllowInvalidCertificates --username "$user" --password "$password"
EOF

juju ssh -m $model $machine "$cmds"

##################

Once in the database:

juju:PRIMARY> db.instanceData.find( {'instanceid': { $regex: /juju-machine.*/ }})

If that returns a bunch of records like the below, that confirms that you have this issue:
{ "_id" : "371812c9-f9e2-4da0-8eec-326a53958f87:0/lxd/0", "machineid" : "0/lxd/0", "instanceid" : "juju-machine-0-lxd-0", "model-uuid" : "371812c9-f9e2-4da0-8eec-326a53958f87", "arch" : "amd64", "txn-revno" : NumberLong(2), "txn-queue" : [ ] }

Confirm that the instanceid's can be changed correctly:

juju:PRIMARY> db.instanceData.find( {'instanceid': { $regex: /juju-machine.*/ }}).forEach( function(doc) { var modelSuffix = doc["model-uuid"].substr(30,6); var newID = "juju-" + modelSuffix + doc["instanceid"].substr(12); print(newID); })

juju-958f87-5-lxd-5
juju-958f87-5-lxd-6
juju-958f87-12-lxd-0

Check that the names output match the names in 'lxc list' on the host machines, e.g. "juju ssh 5 'sudo lxc list'". If we're all good:

Make a Juju backup. Store it locally, rather than just within Juju.

Stop the controller process on all the controller machines (i.e. service named jujud-machine-0 on the controller model machine 0).

Run the following to run the update:

juju:PRIMARY> db.instanceData.find( {'instanceid': { $regex: /juju-machine.*/ }}).forEach(
    function(doc) {
        var modelSuffix = doc["model-uuid"].substr(30,6);
        var newID = "juju-" + modelSuffix + doc["instanceid"].substr(12);
        print("updating "+ doc._id + " instanceid to " + newID);
        db.instanceData.update({_id: doc._id}, {$set: {"instanceid": newID}})
    })

Start controller services again.