ERROR cannot deploy exported bundle: bad metadata: resource missing filename OR invalid storage: count must be greater than zero

Bug #1951182 reported by Nobuto Murata
40
This bug affects 9 people
Affects Status Importance Assigned to Milestone
Graylog Charm
New
Undecided
Unassigned
juju
Triaged
High
Ben Hoyt

Bug Description

Exported bundle cannot be used for another deployment with an error:

invalid storage "data" in application "etcd": cannot parse count: count must be greater than zero, got "0"

After deleting the storage section by hand, then I get the following next:

ERROR cannot deploy bundle: while adding pending resource info for "core": bad resource metadata: bad info: bad metadata: resource missing filename

$ juju version
2.9.29-ubuntu-amd64

How to reproduce:
1. juju bootstrap aws/ap-northeast-1
2. juju deploy etcd -n3
3. juju export-bundle > exported_bundle.yaml
4. juju add-model another-model
5. juju deploy ./exported_bundle.yaml

ERROR cannot deploy bundle: the provided bundle has the following errors:
invalid storage "data" in application "etcd": cannot parse count: count must be greater than zero, got "0"

After deleting the "storage:" from the bundle, then:

Located charm "etcd" in charm-hub, channel stable
Executing changes:
- upload charm etcd from charm-hub for series focal with revision 691 with architecture=amd64
- deploy application etcd from charm-hub on focal with stable
ERROR cannot deploy bundle: while adding pending resource info for "core": bad resource metadata: bad info: bad metadata: resource missing filename

[exported_bundle.yaml]
series: focal
applications:
  etcd:
    charm: etcd
    channel: stable
    revision: 691
    resources:
      core: 0
      etcd: 3
      snapshot: 0
    num_units: 3
    to:
    - "0"
    - "1"
    - "2"
    constraints: arch=amd64
    storage:
      data: loop,0,1024
machines:
  "0":
    constraints: arch=amd64
  "1":
    constraints: arch=amd64
  "2":
    constraints: arch=amd64

Revision history for this message
Peter De Sousa (pjds) wrote (last edit ):
Download full text (17.8 KiB)

Experiencing this issue on 2.9 and 3.0 edge on two different machines.

I can deploy graylog fine on its own, but once in a bundle it does not work and will throw the above error. See stacktrace:

Located charm "telegraf" in charm-store, channel stable
Located charm "thruk-agent" in charm-store, channel stable
Executing changes:
- add new machine 0
- add new machine 1
- add new machine 2
- add new machine 3
- add new machine 4
- add new machine 5
- add new machine 6
- add new machine 7
- add new machine 8
- add new machine 9
- add new machine 10
- add new machine 11
- add new machine 12
- add new machine 13
- add new machine 14
- add new machine 15 (bundle machine 18)
- add new machine 16 (bundle machine 19)
- upload charm landscape-server from charm-store for series bionic with architecture=amd64
- upload charm rabbitmq-server from charm-store for series focal with architecture=amd64
- upload charm mongodb from charm-store for series focal with architecture=amd64
- add new machine 17 (bundle machine 20)
- upload charm graylog from charm-store for series focal with architecture=amd64
- upload charm elasticsearch from charm-store for series focal with architecture=amd64
- deploy application landscape-server from charm-store on bionic
- deploy application landscape-rabbitmq-server from charm-store on focal using rabbitmq-server
- upload charm postgresql from charm-store for series bionic with architecture=amd64
- deploy application graylog-mongodb from charm-store on focal using mongodb
- add lxd container 10/lxd/0 on new machine 10
- deploy application graylog from charm-store on focal
ERROR cannot deploy bundle: cs:graylog-55 resource "core": bad metadata: resource missing filename
➜ ~ juju deploy ./lma.yaml
➜ ~ JUJU juju
➜ ~ juju deploy graylog
Located charm "graylog" in charm-hub, revision 55
Deploying "graylog" from charm-hub charm "graylog", revision 55 in channel stable on focal

Issue bundle:

series: focal
variables:

  # This is Management network, unrelated to OpenStack and other applications
  # OAM - Operations, Administration and Maintenance
  oam-space: &oam-space oam-space

  # nagios-context should be bootstack-customerA-locationB-cloudname
  nagios-context: &nagios-context TODO

  # NTP configuration
  ntp-source: &ntp-source "ntp.ubuntu.com"

machines:
  # KVMs
  "0":
    constraints: tags=nagios
    series: bionic
  "1":
    constraints: tags=grafana
  "2":
    constraints: tags=landscapeha
  "3":
    constraints: tags=landscapesql
    series: bionic
  "4":
    constraints: tags=landscapeamqp
  "5":
    constraints: tags=elastic
  "6":
    constraints: tags=landscape
    series: bionic
  "7":
    constraints: tags=landscapeamqp
  "8":
    constraints: tags=landscapesql
    series: bionic
  "9":
    constraints: tags=prometheus
  "10":
    constraints: tags=graylog
  "11":
    constraints: tags=landscape
    series: bionic
  "12":
    constraints: tags=landscapeamqp
  "13":
    constraints: tags=elastic
  "14":
    constraints: tags=landscape
    series: bionic
  "18":
    constraints: tags=elastic
  "19":
    constraints: tags=graylog
  "20":
    constraints: tags=graylog

...

Revision history for this message
Peter De Sousa (pjds) wrote :

subscribing field high as bundles are currently broken

Revision history for this message
Peter De Sousa (pjds) wrote :

Moving to field medium. Team has found a workaround:

Remove `cs:` from affected applications.

Revision history for this message
Peter De Sousa (pjds) wrote :

removing field medium as this needs to be fixed elsewhere.

Revision history for this message
Heather Lanigan (hmlanigan) wrote :

Interesting, I cannot deploy graylog on its own.

$ juju deploy cs:graylog
Located charm "graylog" in charm-store, revision 55
ERROR cs:graylog-55 resource "core": bad metadata: resource missing filename

Revision history for this message
Daniel Manrique (roadmr) wrote :

It's likely that revision of graylog was released without resources, i.e.

charmcraft release graylog --revision whatever --channel whatever

instead of

charmcraft release graylog --revision whatever --channel whatever --resources yadda yadda yadda

the same revision can be rereleased as above, specifying the resources explicitly, and it should make things work as intended.

Also of note, if you are using Juju 2.9 you don't need to use the cs: prefix, and the new API does not have this problem. (`juju deploy graylog` for example)

https://bugs.launchpad.net/snapstore-server/+bug/1971920 has more details as well (and as you can see, we are tracking it on charmhub's side).

Revision history for this message
Nobuto Murata (nobuto) wrote :

> https://bugs.launchpad.net/snapstore-server/+bug/1971920 has more details as well (and as you can see, we are tracking it on charmhub's side).

This totally makes sense to me. But just to be clear this bug report I opened is nothing to do with cs: vs ch: thing because it's about how Juju exports a bundle and imports it. And it's reproducible with "deploy application etcd from charm-hub".

$ juju deploy ./exported_bundle.yaml
Located charm "etcd" in charm-hub, channel stable
Executing changes:
- upload charm etcd from charm-hub for series focal with revision 691 with architecture=amd64
- deploy application etcd from charm-hub on focal with stable
ERROR cannot deploy bundle: while adding pending resource info for "core": bad resource metadata: bad info: bad metadata: resource missing filename

summary: - ERROR cannot deploy bundle: while adding pending resource info for
- "core": bad resource metadata: bad info: bad metadata: resource missing
- filename
+ ERROR cannot deploy exported bundle: bad metadata: resource missing
+ filename OR invalid storage: count must be greater than zero
description: updated
Revision history for this message
Graeme Moss (graememoss) wrote :
Download full text (3.9 KiB)

We are also getting this error not only in deploying but when trying to upgrade the charm

~/deploy$ juju upgrade-charm -m lma --channel edge graylog --debug
16:25:04 INFO juju.cmd supercommand.go:56 running juju [2.9.32 917a8f1033561ce28a73ff81d71da75aec6e0785 gc go1.18.3]
16:25:04 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/19681/bin/juju", "upgrade-charm", "-m", "lma", "--channel", "edge", "graylog", "--debug"}
16:25:04 INFO juju.juju api.go:78 connecting to API addresses: [*:17070 *:17070 *:17070]
16:25:04 DEBUG juju.api apiclient.go:1153 successfully dialed "wss://*:17070/model/e4264e50-4fd6-4228-8763-b03795044929/api"
16:25:04 INFO juju.api apiclient.go:688 connection established to "wss://*:17070/model/e4264e50-4fd6-4228-8763-b03795044929/api"
16:25:05 INFO juju.juju api.go:78 connecting to API addresses: [*:17070 *:17070 *:17070]
16:25:05 DEBUG juju.api apiclient.go:1153 successfully dialed "wss://*:17070/api"
16:25:05 INFO juju.api apiclient.go:688 connection established to "wss://*:17070/api"
16:25:05 INFO cmd refresher.go:252 Original channel "stable"
16:25:05 INFO cmd refresher.go:254 Requested channel "edge"
16:25:07 INFO cmd refresher.go:309 Using channel "edge"
16:25:08 INFO cmd refresh.go:425 Added charm-hub charm "graylog", revision 57 in channel edge, to the model
16:25:08 DEBUG juju.api monitor.go:35 RPC connection died
ERROR got bad data from server: unsupported resource type ""
16:25:08 DEBUG cmd supercommand.go:537 error stack:
/build/snapcraft-juju-a284566302ade03f36071a6fe755224b/parts/juju/src/vendor/github.com/juju/charm/v8/resource/type.go:33: unsupported resource type ""
/build/snapcraft-juju-a284566302ade03f36071a6fe755224b/parts/juju/src/api/client/resources/helpers.go:167:
/build/snapcraft-juju-a284566302ade03f36071a6fe755224b/parts/juju/src/api/client/resources/helpers.go:81: got bad data from server
/build/snapcraft-juju-a284566302ade03f36071a6fe755224b/parts/juju/src/api/client/resources/client.go:73: ...

Read more...

Revision history for this message
Ian Booth (wallyworld) wrote :

With the scenario in comment #8, can we confirm if this was a new deployment of graylog on a fresh 2.9.32 model, or was the controller initially something like 2.9.29 and upgraded to 2.9.32?

Can we get a dump of the "resources" collection?

Ian Booth (wallyworld)
Changed in juju:
milestone: none → 2.9.33
importance: Undecided → High
status: New → Triaged
Revision history for this message
Ian Booth (wallyworld) wrote (last edit ):

The bundle issue is easily reproducible.

export-bundle gives

series: focal
applications:
  etcd:
    charm: etcd
    channel: stable
    revision: 691
    resources:
      core: 0
      etcd: 3
      snapshot: 0
    num_units: 1
    to:
    - "1"
    constraints: arch=amd64
    storage:
      data: loop,1024M
machines:
  "1":
    constraints: arch=amd64

When the bundle is redeployed, charmhub is asked to provide metadata and comes back with an "empty" record which juju tries to deploy and fails.

Revision history for this message
Ian Booth (wallyworld) wrote :

So I added some logging and it seems we're getting incomplete resource metadata back from charmhub when making a query. The metadata is missing path, description. For the etcd resources, we make an api call to charmhub and get back:

For etcd

Meta: resource.Meta{Name:"etcd", Type:1, Path:"", Description:""}
Revision: 3
Size: 0

Interestingly, for core and snapsort resources we do get more complete metadata, eg

snapshot:

resource.Meta{Name:"snapshot", Type:1, Path:"snapshot.tar.gz", Description:"Tarball snapshot of an etcd clusters data."},

Does this indicate that charmhub is sending incomplete / bad info back to juju?

Revision history for this message
Daniel Andrzejewski (sys-ops) wrote :

I just had encountered the same problem deploying charmed-kubernetes bundle. To redeploy exported bundle I had to remove resources key from every application, including etcd.

So, the following yaml should be deployed with success:

[exported_bundle.yaml]
series: focal
applications:
  etcd:
    charm: etcd
    channel: stable
    revision: 691
    num_units: 3
    to:
    - "0"
    - "1"
    - "2"
    constraints: arch=amd64
    storage:
      data: loop,0,1024
machines:
  "0":
    constraints: arch=amd64
  "1":
    constraints: arch=amd64
  "2":
    constraints: arch=amd64

Changed in juju:
milestone: 2.9.33 → 2.9.34
Ben Hoyt (benhoyt)
Changed in juju:
assignee: nobody → Ben Hoyt (benhoyt)
Changed in juju:
milestone: 2.9.34 → 2.9.35
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers