[AWS] not able to spin LXD containers with extra storage

[Pedro Guimarães]

Original issue: https://github.com/canonical/postgresql-operator/issues/354

It seems that Juju has an exclusion that disallows a given non-LXD cloud to deploy LXD containers on a VM with extra storage volumes.

I can see that error explicitly presented in: https://github.com/juju/juju/blob/9a37403396ac5524400932ebfeb8fa8e0d0adb56/cmd/juju/application/deployer/charm.go#L74

I believe what is happening is the following: (1) we deploy a bundle; (2) juju controller processes the bundle and issues separate requests for each part of that bundle; (3) the "0/lxd/0" fails with storage - hence we are seeing an error from juju/cmd in the juju status instead as a return from CLI.

In any case, this is a complicated limitation for us. It means we cannot run a given DB on top of LXDs on any cloud.

Would be better instead if Juju controller could process LXD-type storage requests: receive the request, issue a new volume just like any other workload; then if this is going to LXD, setup a new storage pool in the LXD of that said machine and pass it as a volume to the container.

[John A Meinel (jameinel)]

As noted, Juju doesn't support this. You're right that we *could* support mounting the EBS volume on the host, and then work throug the mechanics to provide it to LXD, and get it mounted into a container.
We don't have any of that logic today, and it is not exactly trivial to get right, especially with edge cases around when storage shows up, how to do clean setup/teardown/etc.

In general, we haven't prioritized this work because when you are deploying into clouds, you already can size VMs however much you would like, so there isn't much *need* to take a VM and then carve it up into LXD containers.
(Note that you also will have a host of issues around networking and accessibility, etc, so storage is only one of the concerns around containers inside of VMs on arbitrary clouds.)

If there are hard constraints as to why things need to be spun up into containers-inside-VMs, you're welcome to raise it as a request, but you are definitely in "this has never been supported, so cannot be considered a bug" territory.

[Alex Lutay (taurus)]

Thank you John for the quick answer.

Just for the history only. Simplified steps to reproduce (inside VM or just on your laptop):

```
juju add-model teststorage # juju 3.4.1 (rev 26895), lxd 5.20-f3dd836 (rev 27049)

juju deploy postgresql psql-ok1 --channel 14/stable # ALL OK, our default testcase

juju add-machine
juju deploy postgresql psql-ok2 --channel 14/stable --to 1 # ALL OK, if container manually specified

juju deploy postgresql psql-fail --channel 14/stable --to lxd # ERROR (for container `2/lxd/0` inside the container `2`?)

juju status
...
App Version Status Scale Charm Channel Rev Exposed Message
psql-fail error 0/1 postgresql 14/stable 363 no cannot assign unit "psql-fail/0" to machine 2/lxd/0: adding storage to lxd container not supported

Unit Workload Agent Machine Public address Ports Message
psql-fail/0 error lost cannot assign unit "psql-fail/0" to machine 2/lxd/0: adding storage to lxd container not supported
...

```

I believe `--to lxd` confused SQA Team... and they should use `add-machine && deploy --to $id` as above.
Maybe we should document this somewhere.