Juju 1.25.6's unused charm garbage collection fails to remove all references to the old charm, causing unit agents to infinitely loop and exhaust all disk space in certain circumstances when a new version of the charm is deployed under a reused revision.
Specifically, a revision may be reused if the greatest revision of a charm becomes unreferenced and is GCed, letting PrepareLocalCharmUpload allocate the same revision for a second time. The state server's /var/lib/juju/charm-get-cache is indexed by revision, so deploying a charm by the same name but with different content will reuse the revision number, erroneously hit the cache, and return the old archive.
The unit agent notices the hash mismatch and retries the download attempt, but only after leaving a copy of bad file in /var/lib/juju/agents/unit-*/state/bundles/downloads/inprogress-*. After looping for a while, all disk space is exhausted by thousands of copies of the old charm archive.
On a fresh deployment, affected units show up as "failed" and "Waiting for agent initialization to finish", and at normal levels the logs will be silent except for "ERROR juju.worker.uniter.filter filter.go:137 tomb: dying" every few seconds. This can probably affect upgrade-charm too.
Steps to reproduce
------------------
juju deploy local:trusty/some-charm
# Wait for charm to install.
juju destroy-service some-charm
# Wait for 'running "charm" cleanup: "local:trusty/some-charm-1"' log message
touch trusty/some-charm/anything-that-will-cause-the-zip-to-change
juju deploy local:trusty/some-charm
# Wait for the unit agent to drop into failed, and note the hash mismatches in its log.
Workaround
----------
Delete /var/lib/juju/charm-get-cache/* on the state server and /var/lib/juju/agents/unit-*/state/bundles/downloads/inprogress-* on any affected units. The units should immediately recover without further intervention.
We've managed to resolve the situation (see below for the fix). There are several small failures that all contributed to make this bug hard to find and hard to fix.
The root cause is that the copy of the charm on machine 0 in /var/lib/
There are several issues here:
1) The downloaded charms were never cleared up on disk, resulting in a full disk.
2) The hash comparison log message was at the INFO level, which made it very hard to see amidst the clutter of other log messages. The relevant log messages were:
```
2016-09-22 03:33:15 INFO juju.worker.uniter modes.go:567 ModeContinue starting
2016-09-22 03:33:15 INFO juju.worker.uniter modes.go:59 resuming charm install
2016-09-22 03:33:15 INFO juju.worker.uniter modes.go:569 ModeContinue exiting
2016-09-22 03:33:15 INFO juju.worker.uniter modes.go:567 ModeInstalling local:trusty/
2016-09-22 03:33:15 INFO juju.worker.
2016-09-22 03:33:15 INFO juju.worker.
2016-09-22 03:33:15 INFO juju.worker.
2016-09-22 03:33:16 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: install local:trusty/
2016-09-22 03:33:16 INFO juju.worker.uniter modes.go:569 ModeInstalling local:trusty/
2016-09-22 03:33:16 INFO juju.worker.uniter uniter.go:203 unit "sds-worker-
usty/snapdeltas
ttps://
53a737783602d1e
2016-09-22 03:33:16 DEBUG juju.worker.uniter runlistener.go:109 juju-run listener stopping
2016-09-22 03:33:16 DEBUG juju.worker.uniter runlistener.go:129 juju-run listener stopped
```
The fix was to run:
sudo mv /var/lib/
on machine 0.