Juju agent stuck in pending if charm storage's minimum-size exceeds available disk space on machine

[Adam Israel]

Version(s) affected: Juju 1.25.5, 2.0 beta 6

I've hit a repeatable issue where I can put the juju agent into an infinite loop while allocating, and no combination of destroy-environment (1.25.5), destroy-model, destroy-controller, destroy-model (2.0 beta6) will complete.

Scenario:

Deploy charm with storage requirement that exceeds the default machine constraints.

Steps to recreate:

juju bootstrap
juju deploy cs:~3-bruno/trusty/quobyte-registry-4

The quobyte-registry charm has a minimum storage requirement of 10G. This exceeds the default machine constraints. The juju agent attempts to install but can't because the storage loopback file creation has run the disk out of space. The unit agent remains in a pending state forever.

I've attempted every command to destroy the environment, model, and controller to no avail. `destroy-controller` hangs while "Waiting for hosted model resources to be reclaimed".

[Cheryl Jennings (cherylj)]

You can force kill-controller to go through the provider by first manually taking down the controller through your provider and running juju kill-controller.

We still need to address the situation which caused the machine to get stuck in pending.

[Adam Israel (aisrael)]

FWIW, the running machine was killed through the provider, but kill-controller still hangs with:

Destroying controller "local.reviewqueue"
Waiting for resources to be reclaimed
Waiting on 2 models, 2 machines, 2 services
[...]
Waiting on 2 models, 2 machines, 2 services

[Adam Israel (aisrael)]

Negate that. The controller machine didn't get reaped. Killing that unblocked the kill-controller.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 5 years, so we're marking it Expired. If you believe this is incorrect, please update the status.