juju-export bundle gives error after upgrade

[Ponnuvel Palaniyappan]

After upgrading from 2.8.x, exporting bundle results in the following error:

$ juju export-bundle --debug
20:47:51 INFO juju.cmd supercommand.go:56 running juju [2.9.9 0 a426bc84a75e0cd18440d773991281766d803443 gc go1.14.15]
20:47:51 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/16808/bin/juju", "export-bundle", "--debug"}
20:47:51 INFO juju.juju api.go:78 connecting to API addresses: [172.31.242.101:17070 172.31.242.103:17070 172.31.243.14:17070]
20:47:51 DEBUG juju.api apiclient.go:1149 successfully dialed "wss://172.31.243.14:17070/model/affe5d4c-6099-4998-80cd-f5041fcf9afe/api"
20:47:51 INFO juju.api apiclient.go:680 connection established to "wss://172.31.243.14:17070/model/affe5d4c-6099-4998-80cd-f5041fcf9afe/api"
20:47:53 DEBUG juju.api monitor.go:35 RPC connection died
ERROR charm origin: risk "" not valid
20:47:53 DEBUG cmd supercommand.go:537 error stack:
charm origin: risk "" not valid
/build/snapcraft-juju-35d6cf/parts/juju/src/rpc/client.go:178:
/build/snapcraft-juju-35d6cf/parts/juju/src/api/apiclient.go:1248:
/build/snapcraft-juju-35d6cf/parts/juju/src/api/bundle/client.go:73:
$

Otherwise the deployment generally looks healthy with no other issues.

This identical problem has been observed by at least two different user
with different cloud deployments.

[Shunde Zhang (shunde-zhang)]

model version is Juju 2.9.8

[Ian Booth (wallyworld)]

One possible cause is the upgrade to 2.9 did not fill in all the required new charm metadata that is now modelled in 2.9 with charmhub. Needs to be verified.

Marking as Critical initially, as it seems like a regression. We can adjust if needed.

[Simon Richardson (simonrichardson)]

I can't replicate this. I'll probably need more input about the Juju version they're currently running and also the Juju version they started using for 2.9 (2.9.0, 2.9.1, etc).

The commands I tried to replicate this with:

    $ /snap/bin/juju bootstrap lxd test
    $ /snap/bin/juju deploy juju deploy cs:ubuntu --channel="edge" ubuntuz
    $ /snap/bin/juju deploy cs:ubuntu ubuntux
    $ juju upgrade-controller --build-agent
    $ juju export-bundle
    series: bionic
    applications:
    ubuntux:
        charm: cs:ubuntu-18
        channel: stable
        num_units: 1
        to:
        - "1"
    ubuntuz:
        charm: cs:ubuntu-18
        channel: edge
        num_units: 1
        to:
        - "0"
    machines:
    "0": {}
    "1": {}

Mongo was correctly updated as well:

    db.applications.find().pretty()
    {
            "_id" : "62e93afa-0115-451a-847c-2cc9d6002993:ubuntuz",
            "name" : "ubuntuz",
            "model-uuid" : "62e93afa-0115-451a-847c-2cc9d6002993",
            "series" : "bionic",
            "subordinate" : false,
            "charmurl" : "cs:ubuntu-18",
            "cs-channel" : "edge",
            "charmmodifiedversion" : 0,
            "forcecharm" : false,
            "life" : 0,
            "unitcount" : 1,
            "relationcount" : 0,
            "exposed" : false,
            "minunits" : 0,
            "txn-revno" : NumberLong(4),
            "metric-credentials" : BinData(0,""),
            "scale" : 0,
            "passwordhash" : "",
            "txn-queue" : [
                    "611b7e6ca838473fbc7fda20_d01282d1"
            ],
            "charm-origin" : {
                    "source" : "charm-store",
                    "type" : "charm",
                    "id" : "cs:ubuntu-18",
                    "hash" : "",
                    "revision" : 18,
                    "channel" : {
                            "risk" : "edge"
                    },
                    "platform" : {

                    }
            }
    }
    {
            "_id" : "62e93afa-0115-451a-847c-2cc9d6002993:ubuntux",
            "name" : "ubuntux",
            "model-uuid" : "62e93afa-0115-451a-847c-2cc9d6002993",
            "series" : "bionic",
            "subordinate" : false,
            "charmurl" : "cs:ubuntu-18",
            "cs-channel" : "stable",
            "charmmodifiedversion" : 0,
            "forcecharm" : false,
            "life" : 0,
            "unitcount" : 1,
            "relationcount" : 0,
            "exposed" : false,
            "minunits" : 0,
            "txn-revno" : NumberLong(4),
            "metric-credentials" : BinData(0,""),
            "scale" : 0,
            "passwordhash" : "",
            "txn-queue" : [
                    "611b7e6ca838473fbc7fda20_d01282d1"
            ],
            "charm-origin" : {
                    "source" : "charm-store",
                    "type" : "charm",
                    "id" : "cs:ubuntu-18",
                    "hash" : "",
                    "revision" : 18,
                    "channel" : {
                            "risk" : "stable"
                    },
                    "platform" : {

                    }
        ...

[Simon Richardson (simonrichardson)]

Juju PR: https://github.com/juju/juju/pull/13247

The workaround is to use the following setup:

 1. Login to mongo
 2. Run the following command:

    var malformedCount = db.applications.find({$and: [{"cs-channel": {$ne:""}}, {"charm-origin.channel.risk": ""}]}).count()
    if (malformedCount > 0) {
        print("expected application channel and charm-origin channel risk to be empty.")
        quit(1);
    }
    db.applications.updateMany({"charm-origin.channel.risk": ""}, {$set:{"charm-origin.channel.risk":"stable"}})

[Paul Collins (pjdc)]

Just a note for future travellers and search engines that this appears to also affect juju dump-model and the equivalent DumpModel API, which is used by some monitoring and reporting infrastructure.

jenkins@juju-59050e-is-mojo-ci-0:~$ juju export-bundle --debug
01:32:50 INFO juju.cmd supercommand.go:56 running juju [2.9.11 0 7fcbdb3115b295c1610287d0db7323dfa72e8f21 gc go1.14.15]
01:32:50 DEBUG juju.cmd supercommand.go:57 args: []string{"juju", "export-bundle", "--debug"}
01:32:50 INFO juju.juju api.go:78 connecting to API addresses: [10.25.62.244:17070 10.25.62.246:17070 10.25.62.245:17070]
01:32:50 DEBUG juju.api apiclient.go:1149 successfully dialed "wss://10.25.62.245:17070/model/e013765c-2101-472f-814a-52c161511925/api"
01:32:50 INFO juju.api apiclient.go:680 connection established to "wss://10.25.62.245:17070/model/e013765c-2101-472f-814a-52c161511925/api"
01:32:50 INFO juju.juju api.go:330 API endpoints changed from [10.25.62.245:17070 10.25.62.244:17070 10.25.62.246:17070] to [10.25.62.245:17070 10.25.62.246:17070 10.25.62.244:17070]
01:32:50 DEBUG juju.api monitor.go:35 RPC connection died
ERROR charm origin: risk "" not valid
01:32:50 DEBUG cmd supercommand.go:537 error stack:
charm origin: risk "" not valid
/build/snapcraft-juju-35d6cf/parts/juju/src/rpc/client.go:178:
/build/snapcraft-juju-35d6cf/parts/juju/src/api/apiclient.go:1248:
/build/snapcraft-juju-35d6cf/parts/juju/src/api/bundle/client.go:73:
jenkins@juju-59050e-is-mojo-ci-0:~$ JUJU_DEV_FEATURE_FLAGS=developer-mode juju dump-model
{}
ERROR charm origin: risk "" not valid
jenkins@juju-59050e-is-mojo-ci-0:~$ _