[2.7.0][caas] removing an application with block storage fails

[Dmitrii Shcherbakov]

juju remove-application --force test
removing application test failed: BlockStorage instance not implemented

storage:
  database:
    type: filesystem
    location: /srv/mysql
  backup:
    type: block
    multiple:
      range: 0-
  stash:
    type: block
    multiple:
      range: 0-

# from bash history

juju deploy ./ test0 --storage 'database=tmpfs,100M backup=loop,100M stash=loop,100M'
juju run --unit test/0 'storage-list'
juju run --unit test/0 'storage-add backup count=1'
juju run --unit test/0 'storage-list'
juju remove-application --force test

juju remove-application --force test
removing application test failed: BlockStorage instance not implemented

---------------

The root cause seems to be that Juju allows this situation to happen in the first place by not properly validating storage constraints:

1) When backup=loop,100M is used to deploy a k8s charm without any file system storage the error is present at the deploy time;

2) when both file system storage and block storage constraints are present, the error is not thrown:

'database-a=tmpfs,100M database-b=tmpfs,100M database-c=tmpfs,100M backup=loop,100M stash=loop,100M'

1)
juju deploy ./ test --storage 'backup=loop,100M stash=loop,100M' --debug
12:28:37 INFO juju.cmd supercommand.go:80 running juju [2.7.0 gc go1.10.4]
12:28:37 DEBUG juju.cmd supercommand.go:81 args: []string{"/snap/juju/9876/bin/juju", "deploy", "./", "test", "--storage", "backup=loop,100M stash=loop,100M", "--debug"}
12:28:37 INFO juju.juju api.go:67 connecting to API addresses: [10.152.183.208:17070]
12:28:37 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.152.183.208:17070/model/63509cbd-79bb-4138-87c0-1f3faa56d528/api"
12:28:37 INFO juju.api apiclient.go:624 connection established to "wss://10.152.183.208:17070/model/63509cbd-79bb-4138-87c0-1f3faa56d528/api"
12:28:37 INFO juju.juju api.go:67 connecting to API addresses: [10.152.183.208:17070]
12:28:37 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.152.183.208:17070/api"
12:28:37 INFO juju.api apiclient.go:624 connection established to "wss://10.152.183.208:17070/api"
12:28:37 DEBUG httpbakery client.go:243 client do POST https://10.152.183.208:17070/model/63509cbd-79bb-4138-87c0-1f3faa56d528/charms?revision=0&schema=local&series=kubernetes {
12:28:38 DEBUG httpbakery client.go:245 } -> error <nil>
12:28:38 INFO cmd deploy.go:1460 Deploying charm "local:kubernetes/wordpress-0".
12:28:38 DEBUG juju.api monitor.go:35 RPC connection died
ERROR getting workload storage params for "test": storage provider "loop" not valid
12:28:38 DEBUG cmd supercommand.go:519 error stack:
getting workload storage params for "test": storage provider "loop" not valid
/build/juju/parts/juju/go/src/github.com/juju/juju/api/application/client.go:161:
/build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/application/deploy.go:258:
/build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/application/deploy.go:1149:
/build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/application/deploy.go:1461:

2)
juju deploy ./ test --storage 'database-a=tmpfs,100M database-b=tmpfs,100M database-c=tmpfs,100M backup=loop,100M stash=loop,100M' --debug
12:28:54 INFO juju.cmd supercommand.go:80 running juju [2.7.0 gc go1.10.4]
12:28:54 DEBUG juju.cmd supercommand.go:81 args: []string{"/snap/juju/9876/bin/juju", "deploy", "./", "test", "--storage", "database-a=tmpfs,100M database-b=tmpfs,100M database-c=tmpfs,100M backup=loop,100M stash=loop,100M", "--debug"}
12:28:54 INFO juju.juju api.go:67 connecting to API addresses: [10.152.183.208:17070]
12:28:54 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.152.183.208:17070/model/63509cbd-79bb-4138-87c0-1f3faa56d528/api"
12:28:54 INFO juju.api apiclient.go:624 connection established to "wss://10.152.183.208:17070/model/63509cbd-79bb-4138-87c0-1f3faa56d528/api"
12:28:54 INFO juju.juju api.go:67 connecting to API addresses: [10.152.183.208:17070]
12:28:54 DEBUG juju.api apiclient.go:1092 successfully dialed "wss://10.152.183.208:17070/api"
12:28:54 INFO juju.api apiclient.go:624 connection established to "wss://10.152.183.208:17070/api"
12:28:54 DEBUG httpbakery client.go:243 client do POST https://10.152.183.208:17070/model/63509cbd-79bb-4138-87c0-1f3faa56d528/charms?revision=0&schema=local&series=kubernetes {
12:28:55 DEBUG httpbakery client.go:245 } -> error <nil>
12:28:55 INFO cmd deploy.go:1460 Deploying charm "local:kubernetes/wordpress-1".
12:28:56 DEBUG juju.api monitor.go:35 RPC connection died
12:28:56 INFO cmd supercommand.go:525 command finished

cat metadata.yaml
name: wordpress
summary: WordPress
maintainer: Operator Framework Developers https://github.com/canonical/operator/graphs/contributors
description: |
    WordPress CMS charm.
tags:
    - blog
series:
    - kubernetes
requires:
    db:
        interface: mysql
storage:
  database-a:
    type: filesystem
    location: /srv/mysqla
  database-b:
    type: filesystem
    location: /srv/mysqlb
  database-c:
    type: filesystem
    location: /srv/mysqlc
  backup:
    type: block
    multiple:
      range: 0-
  stash:
    type: block
    multiple:
      range: 0-

An error message is present in the log after the deployment time:

application-test: 12:29:15 ERROR juju.worker.dependency "unit-init-worker" manifold worker returned unexpected error: initializing unit "test/0": caas-unit-init failed: : command terminated with exit code 137

[Thomas Miller (tlmiller)]

Hi Dmitrii,

I have reviewed the bug and it looks to me like an error with the command line args for storage constraints. Reviewing the code and multiple storage constraints cannot be provided in the one argument to --storage. I have done a PR here https://github.com/juju/juju/pull/10998 that should more explicitly error on the above conditions.

[Dmitrii Shcherbakov (dmitriis)]

Hi Thomas,

Thank you for looking into it.

Is there any way to specify multiple storage constraints, e.g. via multiple --storage arguments, at all?

This logic seems a bit different to how network spaces are handled.

For example, network spaces are possible to specify via --constraints [0][1], but there is also the --bind syntax [2] for them which allows space-separated endpoint bindings:

juju deploy --bind "default-space db=db-space db-admin=admin-space" mysql

Since applications may have storage endpoints it seems worthwhile to support something like this for storage.

It seems like a bigger chunk of work than just error handling so I can create a separate bug or discourse topic if needed.

[0] https://jaas.ai/docs/constraints#heading--setting-constraints-for-the-initial-controller-and-default-models
[1] https://jaas.ai/docs/constraints-reference
[2] https://jaas.ai/docs/deploying-advanced-applications#heading--deploying-to-network-spaces

[Ian Booth (wallyworld)]

--storage is like --config
If you want more than one, you need to repeat, eg
juju deploy foo --config a=b --config b=c --storage data=1G --storage logs=2G

[Dmitrii Shcherbakov (dmitriis)]

I see, thanks for clarifying.

[Ian Booth (wallyworld)]

The issue here I think is that juju should reject charms which want block storage as that's not supported yet.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/11574