Juju agent stuck in pending if charm storage's minimum-size exceeds available disk space on machine

Bug #1580717 reported by Adam Israel on 2016-05-11
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
juju
High
Unassigned
juju-core
High
Unassigned
1.25
High
Unassigned

Bug Description

Version(s) affected: Juju 1.25.5, 2.0 beta 6

I've hit a repeatable issue where I can put the juju agent into an infinite loop while allocating, and no combination of destroy-environment (1.25.5), destroy-model, destroy-controller, destroy-model (2.0 beta6) will complete.

Scenario:

Deploy charm with storage requirement that exceeds the default machine constraints.

Steps to recreate:

juju bootstrap
juju deploy cs:~3-bruno/trusty/quobyte-registry-4

The quobyte-registry charm has a minimum storage requirement of 10G. This exceeds the default machine constraints. The juju agent attempts to install but can't because the storage loopback file creation has run the disk out of space. The unit agent remains in a pending state forever.

I've attempted every command to destroy the environment, model, and controller to no avail. `destroy-controller` hangs while "Waiting for hosted model resources to be reclaimed".

Adam Israel (aisrael) wrote :
Adam Israel (aisrael) wrote :
description: updated
Cheryl Jennings (cherylj) wrote :

You can force kill-controller to go through the provider by first manually taking down the controller through your provider and running juju kill-controller.

We still need to address the situation which caused the machine to get stuck in pending.

Changed in juju-core:
status: New → Triaged
importance: Undecided → High
tags: added: deploy storage
Adam Israel (aisrael) wrote :

FWIW, the running machine was killed through the provider, but kill-controller still hangs with:

Destroying controller "local.reviewqueue"
Waiting for resources to be reclaimed
Waiting on 2 models, 2 machines, 2 services
[...]
Waiting on 2 models, 2 machines, 2 services

Adam Israel (aisrael) wrote :

Negate that. The controller machine didn't get reaped. Killing that unblocked the kill-controller.

Curtis Hovey (sinzui) on 2016-05-19
Changed in juju-core:
milestone: none → 2.0-beta8
Changed in juju-core:
milestone: 2.0-beta8 → 2.0-beta9
Curtis Hovey (sinzui) on 2016-06-16
Changed in juju-core:
milestone: 2.0-beta9 → 2.0-beta10
Curtis Hovey (sinzui) on 2016-06-24
Changed in juju-core:
milestone: 2.0-beta10 → 2.0-beta11
Curtis Hovey (sinzui) on 2016-07-01
Changed in juju-core:
milestone: 2.0-beta11 → 2.0-beta12
Changed in juju-core:
milestone: 2.0-beta12 → 2.0-beta13
tags: added: 2.0
Curtis Hovey (sinzui) on 2016-07-22
Changed in juju-core:
milestone: 2.0-beta13 → 2.0-beta14
Curtis Hovey (sinzui) on 2016-08-04
Changed in juju-core:
milestone: 2.0-beta14 → 2.0-beta15
Changed in juju-core:
milestone: 2.0-beta15 → 2.0.0
affects: juju-core → juju
Changed in juju:
milestone: 2.0.0 → none
milestone: none → 2.0.0
Changed in juju-core:
importance: Undecided → High
status: New → Won't Fix
Changed in juju:
milestone: 2.0.0 → 2.0.1
Curtis Hovey (sinzui) on 2016-10-28
Changed in juju:
milestone: 2.0.1 → none
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers