ERROR cannot deploy exported bundle: bad metadata: resource missing filename OR invalid storage: count must be greater than zero

[Nobuto Murata]

Exported bundle cannot be used for another deployment with an error:

invalid storage "data" in application "etcd": cannot parse count: count must be greater than zero, got "0"

After deleting the storage section by hand, then I get the following next:

ERROR cannot deploy bundle: while adding pending resource info for "core": bad resource metadata: bad info: bad metadata: resource missing filename

$ juju version
2.9.29-ubuntu-amd64

How to reproduce:
1. juju bootstrap aws/ap-northeast-1
2. juju deploy etcd -n3
3. juju export-bundle > exported_bundle.yaml
4. juju add-model another-model
5. juju deploy ./exported_bundle.yaml

ERROR cannot deploy bundle: the provided bundle has the following errors:
invalid storage "data" in application "etcd": cannot parse count: count must be greater than zero, got "0"

After deleting the "storage:" from the bundle, then:

Located charm "etcd" in charm-hub, channel stable
Executing changes:
- upload charm etcd from charm-hub for series focal with revision 691 with architecture=amd64
- deploy application etcd from charm-hub on focal with stable
ERROR cannot deploy bundle: while adding pending resource info for "core": bad resource metadata: bad info: bad metadata: resource missing filename

[exported_bundle.yaml]
series: focal
applications:
  etcd:
    charm: etcd
    channel: stable
    revision: 691
    resources:
      core: 0
      etcd: 3
      snapshot: 0
    num_units: 3
    to:
    - "0"
    - "1"
    - "2"
    constraints: arch=amd64
    storage:
      data: loop,0,1024
machines:
  "0":
    constraints: arch=amd64
  "1":
    constraints: arch=amd64
  "2":
    constraints: arch=amd64

[Peter Jose De Sousa (pjds)]

Experiencing this issue on 2.9 and 3.0 edge on two different machines.

I can deploy graylog fine on its own, but once in a bundle it does not work and will throw the above error. See stacktrace:

Located charm "telegraf" in charm-store, channel stable
Located charm "thruk-agent" in charm-store, channel stable
Executing changes:
- add new machine 0
- add new machine 1
- add new machine 2
- add new machine 3
- add new machine 4
- add new machine 5
- add new machine 6
- add new machine 7
- add new machine 8
- add new machine 9
- add new machine 10
- add new machine 11
- add new machine 12
- add new machine 13
- add new machine 14
- add new machine 15 (bundle machine 18)
- add new machine 16 (bundle machine 19)
- upload charm landscape-server from charm-store for series bionic with architecture=amd64
- upload charm rabbitmq-server from charm-store for series focal with architecture=amd64
- upload charm mongodb from charm-store for series focal with architecture=amd64
- add new machine 17 (bundle machine 20)
- upload charm graylog from charm-store for series focal with architecture=amd64
- upload charm elasticsearch from charm-store for series focal with architecture=amd64
- deploy application landscape-server from charm-store on bionic
- deploy application landscape-rabbitmq-server from charm-store on focal using rabbitmq-server
- upload charm postgresql from charm-store for series bionic with architecture=amd64
- deploy application graylog-mongodb from charm-store on focal using mongodb
- add lxd container 10/lxd/0 on new machine 10
- deploy application graylog from charm-store on focal
ERROR cannot deploy bundle: cs:graylog-55 resource "core": bad metadata: resource missing filename
➜ ~ juju deploy ./lma.yaml
➜ ~ JUJU juju
➜ ~ juju deploy graylog
Located charm "graylog" in charm-hub, revision 55
Deploying "graylog" from charm-hub charm "graylog", revision 55 in channel stable on focal

Issue bundle:

series: focal
variables:

  # This is Management network, unrelated to OpenStack and other applications
  # OAM - Operations, Administration and Maintenance
  oam-space: &oam-space oam-space

  # nagios-context should be bootstack-customerA-locationB-cloudname
  nagios-context: &nagios-context TODO

  # NTP configuration
  ntp-source: &ntp-source "ntp.ubuntu.com"

machines:
  # KVMs
  "0":
    constraints: tags=nagios
    series: bionic
  "1":
    constraints: tags=grafana
  "2":
    constraints: tags=landscapeha
  "3":
    constraints: tags=landscapesql
    series: bionic
  "4":
    constraints: tags=landscapeamqp
  "5":
    constraints: tags=elastic
  "6":
    constraints: tags=landscape
    series: bionic
  "7":
    constraints: tags=landscapeamqp
  "8":
    constraints: tags=landscapesql
    series: bionic
  "9":
    constraints: tags=prometheus
  "10":
    constraints: tags=graylog
  "11":
    constraints: tags=landscape
    series: bionic
  "12":
    constraints: tags=landscapeamqp
  "13":
    constraints: tags=elastic
  "14":
    constraints: tags=landscape
    series: bionic
  "18":
    constraints: tags=elastic
  "19":
    constraints: tags=graylog
  "20":
    constraints: tags=graylog

...

[Peter Jose De Sousa (pjds)]

subscribing field high as bundles are currently broken

[Peter Jose De Sousa (pjds)]

Moving to field medium. Team has found a workaround:

Remove `cs:` from affected applications.

[Peter Jose De Sousa (pjds)]

removing field medium as this needs to be fixed elsewhere.

[Heather Lanigan (hmlanigan)]

Interesting, I cannot deploy graylog on its own.

$ juju deploy cs:graylog
Located charm "graylog" in charm-store, revision 55
ERROR cs:graylog-55 resource "core": bad metadata: resource missing filename

[Daniel Manrique (roadmr)]

It's likely that revision of graylog was released without resources, i.e.

charmcraft release graylog --revision whatever --channel whatever

instead of

charmcraft release graylog --revision whatever --channel whatever --resources yadda yadda yadda

the same revision can be rereleased as above, specifying the resources explicitly, and it should make things work as intended.

Also of note, if you are using Juju 2.9 you don't need to use the cs: prefix, and the new API does not have this problem. (`juju deploy graylog` for example)

https://bugs.launchpad.net/snapstore-server/+bug/1971920 has more details as well (and as you can see, we are tracking it on charmhub's side).

[Nobuto Murata (nobuto)]

> https://bugs.launchpad.net/snapstore-server/+bug/1971920 has more details as well (and as you can see, we are tracking it on charmhub's side).

This totally makes sense to me. But just to be clear this bug report I opened is nothing to do with cs: vs ch: thing because it's about how Juju exports a bundle and imports it. And it's reproducible with "deploy application etcd from charm-hub".

$ juju deploy ./exported_bundle.yaml
Located charm "etcd" in charm-hub, channel stable
Executing changes:
- upload charm etcd from charm-hub for series focal with revision 691 with architecture=amd64
- deploy application etcd from charm-hub on focal with stable
ERROR cannot deploy bundle: while adding pending resource info for "core": bad resource metadata: bad info: bad metadata: resource missing filename

[Graeme Moss (graememoss)]

We are also getting this error not only in deploying but when trying to upgrade the charm

~/deploy$ juju upgrade-charm -m lma --channel edge graylog --debug
16:25:04 INFO juju.cmd supercommand.go:56 running juju [2.9.32 917a8f1033561ce28a73ff81d71da75aec6e0785 gc go1.18.3]
16:25:04 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/19681/bin/juju", "upgrade-charm", "-m", "lma", "--channel", "edge", "graylog", "--debug"}
16:25:04 INFO juju.juju api.go:78 connecting to API addresses: [*:17070 *:17070 *:17070]
16:25:04 DEBUG juju.api apiclient.go:1153 successfully dialed "wss://*:17070/model/e4264e50-4fd6-4228-8763-b03795044929/api"
16:25:04 INFO juju.api apiclient.go:688 connection established to "wss://*:17070/model/e4264e50-4fd6-4228-8763-b03795044929/api"
16:25:05 INFO juju.juju api.go:78 connecting to API addresses: [*:17070 *:17070 *:17070]
16:25:05 DEBUG juju.api apiclient.go:1153 successfully dialed "wss://*:17070/api"
16:25:05 INFO juju.api apiclient.go:688 connection established to "wss://*:17070/api"
16:25:05 INFO cmd refresher.go:252 Original channel "stable"
16:25:05 INFO cmd refresher.go:254 Requested channel "edge"
16:25:07 INFO cmd refresher.go:309 Using channel "edge"
16:25:08 INFO cmd refresh.go:425 Added charm-hub charm "graylog", revision 57 in channel edge, to the model
16:25:08 DEBUG juju.api monitor.go:35 RPC connection died
ERROR got bad data from server: unsupported resource type ""
16:25:08 DEBUG cmd supercommand.go:537 error stack:
/build/snapcraft-juju-a284566302ade03f36071a6fe755224b/parts/juju/src/vendor/github.com/juju/charm/v8/resource/type.go:33: unsupported resource type ""
/build/snapcraft-juju-a284566302ade03f36071a6fe755224b/parts/juju/src/api/client/resources/helpers.go:167:
/build/snapcraft-juju-a284566302ade03f36071a6fe755224b/parts/juju/src/api/client/resources/helpers.go:81: got bad data from server
/build/snapcraft-juju-a284566302ade03f36071a6fe755224b/parts/juju/src/api/client/resources/client.go:73: ...

[Ian Booth (wallyworld)]

With the scenario in comment #8, can we confirm if this was a new deployment of graylog on a fresh 2.9.32 model, or was the controller initially something like 2.9.29 and upgraded to 2.9.32?

Can we get a dump of the "resources" collection?

[Ian Booth (wallyworld)]

The bundle issue is easily reproducible.

export-bundle gives

series: focal
applications:
  etcd:
    charm: etcd
    channel: stable
    revision: 691
    resources:
      core: 0
      etcd: 3
      snapshot: 0
    num_units: 1
    to:
    - "1"
    constraints: arch=amd64
    storage:
      data: loop,1024M
machines:
  "1":
    constraints: arch=amd64

When the bundle is redeployed, charmhub is asked to provide metadata and comes back with an "empty" record which juju tries to deploy and fails.

[Ian Booth (wallyworld)]

So I added some logging and it seems we're getting incomplete resource metadata back from charmhub when making a query. The metadata is missing path, description. For the etcd resources, we make an api call to charmhub and get back:

For etcd

Meta: resource.Meta{Name:"etcd", Type:1, Path:"", Description:""}
Revision: 3
Size: 0

Interestingly, for core and snapsort resources we do get more complete metadata, eg

snapshot:

resource.Meta{Name:"snapshot", Type:1, Path:"snapshot.tar.gz", Description:"Tarball snapshot of an etcd clusters data."},

Does this indicate that charmhub is sending incomplete / bad info back to juju?

[Daniel Andrzejewski (sys-ops)]

I just had encountered the same problem deploying charmed-kubernetes bundle. To redeploy exported bundle I had to remove resources key from every application, including etcd.

So, the following yaml should be deployed with success:

[exported_bundle.yaml]
series: focal
applications:
  etcd:
    charm: etcd
    channel: stable
    revision: 691
    num_units: 3
    to:
    - "0"
    - "1"
    - "2"
    constraints: arch=amd64
    storage:
      data: loop,0,1024
machines:
  "0":
    constraints: arch=amd64
  "1":
    constraints: arch=amd64
  "2":
    constraints: arch=amd64

[Will Fitch (wrfitch)]

Ran into this issue with the new postgres VM bundle - we managed a workaround by deleting `storage:` and `resource:` fields from our bundle.yaml, and letting the relevant charm handle them on its own.

[Eric Chen (eric-chen)]

From the description of Ian Booth, this issue is not realted to graylog.
I will mark "invalid" for graylog.

[Juan M. Tirado (tiradojm)]

Can we confirm this is happening with juju 2.9.38?

[Nobuto Murata (nobuto)]

> Can we confirm this is happening with juju 2.9.38?

Yes.

$ juju version
2.9.38-ubuntu-amd64

1. juju bootstrap aws/ap-northeast-1
2. juju deploy etcd -n3
3. juju export-bundle > exported_bundle.yaml
4. juju add-model another-model
5. juju deploy ./exported_bundle.yaml

Located charm "etcd" in charm-hub, channel stable
Executing changes:
- upload charm etcd from charm-hub for series jammy with revision 722 with architecture=amd64
- deploy application etcd from charm-hub on jammy with stable
ERROR cannot deploy bundle: while adding pending resource info for "snapshot": bad resource metadata: bad info: bad metadata: resource missing filename

[Ancheng Liu (anchengliu)]

I'm trying to upgrade project from bionic, I got same issue
as the upgrade purpose, I have to use old juju version

```
$ juju deploy ./config/bundle.yaml

- deploy application neutron-api on bionic using cs:neutron-api-304
ERROR cannot deploy bundle: cs:neutron-api-304 resource "policyd-override": bad metadata: resource missing filename

$ juju --version
2.8.13-bionic-amd64

# bundle.yaml
series: bionic

machines:
  "1001":
    constraints: tags=ctrl001
  "1005":
    constraints: tags=hci
  "1006":
    constraints: tags=ctrl002
  "1007":
    constraints: tags=hci
  "1008":
    constraints: tags=ctrl003

applications:
  neutron-api:
    charm: cs:neutron-api
    num_units: 3
    bindings:
      "": *oam-space
      public: *public-space
      admin: *admin-space
      internal: *internal-space
      shared-db: *internal-space
    options:
      worker-multiplier: *worker-multiplier
      openstack-origin: *openstack-origin
      region: *openstack-region
      flat-network-providers: physnet1
      vlan-ranges:
      neutron-security-groups: True
      overlay-network-type: vxlan
      use-internal-endpoints: True
      vip: *neutron-api-vip
      enable-l3ha: False
      enable-dvr: True
      dhcp-agents-per-network: 2
      enable-ml2-port-security: True
      default-tenant-network-type: vxlan
      l2-population: True
      enable-ml2-dns: True
      dns-domain: *dns-domain
      reverse-dns-lookup: True
      ipv4-ptr-zone-prefix-size: *dns-cidr
      global-physnet-mtu: 1550
      path-mtu: 1550
      physical-network-mtus: physnet1:1500
    to:
    - lxd:1001
    - lxd:1006
    - lxd:1008

```
not working while change `cs` to `ch`
FROM: charm: cs:neutron-api
TO: charm: ch:neutron-api

```
$ juju deploy ./config/bundle.yaml
ERROR cannot deploy bundle: the provided bundle has the following errors:
invalid charm URL in application "neutron-api": cannot parse URL "ch:neutron-api": schema "ch" not valid
```

how to fix this issue?

[Ian Marsh (drulgaard)]

For the record, with Juju 3.2-beta1, I have the same issue. An export-bundle produces various entries along these lines:

  cinder:
    charm: cinder
    channel: zed/stable
    revision: 594
    resources:
      policyd-override: 0
    num_units: 3
    to:
    - lxd:0
    - lxd:1
    - lxd:2
    options:
      block-device: None
      glance-api-version: 2
      openstack-origin: cloud:jammy-zed
      vip: [REDACTED]
    constraints: arch=amd64
    storage:
      block-devices: loop,10240M

I have to remove all the 'storage' and 'resources' sections to get a bundle I can then deploy.

Example errors whilst trying to deploy exported bundle:

- deploy application ceph-mon from charm-hub on jammy with quincy/stable
ERROR cannot deploy bundle: while adding pending resource info for "alert-rules": bad resource metadata: bad info: bad metadata: resource missing filename

ERROR cannot deploy bundle: cannot add unit for application "cinder": acquiring machine to host unit "cinder/0": cannot assign unit "cinder/0" to machine 0/lxd/2: adding storage to lxd container not supported (not supported)

[Heather Lanigan (hmlanigan)]

@anchengliu, related to #18.

Charms with ch: are not supported before juju 2.9.x.

[Heather Lanigan (hmlanigan)]

I can reproduce it outside of an exported bundle:

$ juju deploy etcd --revision 742 --channel stable --resource core=0 --resource etcd=3 --resource snapshot=0
Located charm "etcd" in charm-hub, revision 742
ERROR while adding pending resource info for "core": bad resource metadata: bad info: bad metadata: resource missing filename

It seems to be specific to some charms.

juju deploy juju-qa-test --resource foo-file=3
Located charm "juju-qa-test" in charm-hub, revision 19
Deploying "juju-qa-test" from charm-hub charm "juju-qa-test", revision 19 in channel stable on ubuntu@20.04/stable

[Trent Lloyd (lathiat)]

For future travellers, on the off chance like us you're trying to deploy juju 2.8 + openstack from charmstore to reproduce upgrades.

To work around this issue, first deploy a juju 2.8.13 controller using the 2.8.13 client. But then switch back to 2.9/stable (2.9.43) before deploying the model/bundle - then it will deploy fine to the 2.8 controller without the "bad metadata: resource missing filename" error for policyd-override on the various charms (cinder/glance/etc). That part seems fixed on the client side and 2.9.43 client works fine against a 2.8.13 controller.

"juju deploy glance" also works fine, it only fails when using a bundle/bundle including overlays. Manually doing 'juju deploy' steps is another possible workaround.

[Ian Booth (wallyworld)]

The next 2.9.46 candidate release will not include a fix for this bug and we don't plan on any more 2.9 releases. As such it is being removed from its 2.9 milestone.

If the bug is still important to you, let us know and we can consider it for inclusion on a 3.x milestone.