[LXC caching] images do not return to cache once deleted (due to host caching)

[Peter Matulis]

With 1.24.4, attempt to delete LXC cached images fails:

$ juju status
environment: lxc
machines:
  "0":
    agent-state: started
    agent-version: 1.24.4.1
    dns-name: localhost
    instance-id: localhost
    series: trusty
    state-server-member-status: has-vote
services: {}

$ juju cached-images list
no matching images found

$ juju add-machine
created machine 1

$ juju cached-images list
Cached images:
- kind: lxc
  series: trusty
  arch: amd64
  source-url: https://cloud-images.ubuntu.com/server/releases/trusty/release-20150810/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
  created: Wed, 12 Aug 2015 06:01:38 UTC

# ls -lh /var/cache/lxc/cloud-trusty
total 179M
-rw-r--r-- 1 root root 179M Aug 12 06:01 ubuntu-14.04-server-cloudimg-amd64-root.tar.gz

$ juju cached-images delete --kind lxc --series trusty --arch amd64

$ juju cached-images list
no matching images found

# ls -lh /var/cache/lxc/cloud-trusty
total 179M
-rw-r--r-- 1 root root 179M Aug 12 06:01 ubuntu-14.04-server-cloudimg-amd64-root.tar.gz

So Juju does not report an error when deleting the image and lists the image as non-existent yet it continues to reside on the filesystem and Juju continues to use it (based on the short time required for a new machine). Not only is this confusing but it also means that the environment is stuck with the initial image.

Manual removal of the image file and a subsequent add-machine brings the image back on the filesystem and the list command reports the image.

I disabled cloning because it was interfering with my tests.

$ cat ~/.juju/environments.yaml
default: lxc
environments:
    lxc:
        type: local
        default-series: trusty
        network-bridge: lxcbr0
        lxc-clone: false

[Curtis Hovey (sinzui)]

The delete command deletes from the state server cache, not the host's cache.

[Peter Matulis (petermatulis)]

Ok, I didn't know about that.

1. Where is theJuju environment cache?
2. Why doesn't Juju just use the host cache?
3. Does an image get downloaded twice for every new release & architecture?
4. How does all this work in an HA context?

[Ian Booth (wallyworld)]

The feature was added for MAAS deployments where different machines are involved.
Consider the case where machine 1 starts and then a container is added. The image is downloaded from cloud-images. Then machine 2 is added. Again for a second time, the image has to be downloaded all over again from cloud-images. This was causing MAAS demos at ODS and elsewhere to take 10s of minutes to start up. By having each machine go via the Juju state server, and the state server caching the image, it means that for a given MAAS environment, the hit in going to the internet to get the image only occurs once, not for every new MAAS node added to the environment.

The environment cache set up by Juju is a blobstore in the mongo instance used by the state server.

Juju actually does use the host cache. Well, not Juju directly but the lxc scripts. The lxc scripts will first look at the host cache and if there's an image there, use that. Otherwise the lxc scripts will pull the image from a URL using wget. In this case, the default URL is overridden by Juju to point to the state server to allow a transparent cache to be used.

Juju's use of a state server cache ensures that an image is only downloaded once. Without this feature, every single new node that is acquired would result in a new internet download.

It works as expected in HA. The mongo database is replicated.

There's a little extra detail in the 1.22 release notes https://launchpad.net/juju-core/+milestone/1.22.0

This feature is a stop gap measure until LXD comes along with effectively baked in images out of the box.

[Peter Matulis (petermatulis)]

@Ian

So if host caching is enabled, then i) that cache needs to be flushed in order to make a fresh image available to Juju and ii) the images listed by Juju are never used (for containers)?

[Ian Booth (wallyworld)]

The first time a given host needs to start an LXC container, the image is retrieved via the Juju server, and this will cause the image to be cached by Juju. And if LXC host caching is used, the image is also cached locally on the host. So subsequent images *on that same host* will come from the host image cache, not Juju's cache.

The images listed by Juju are used for containers on *new* hosts. This prevents new hosts from having to go to the internet to get the images. This approach allowed a demo using MAAS to start up in a few minutes rather than 20 - 30 minutes due to each new acquired machine having to fetch the image over a slow link.

So listing images in Juju shows images that will be used when new hosts are created and an lxc container is first started.

The answer to the questions are therefore i) yes and ii) no (because the images are used on new hosts)

[Peter Matulis (petermatulis)]

Thanks.

This bug can therefore be reduced to an image not being returned to the Juju cache if the image is ever deleted from it. I presume this is due to the image already residing in the host cache. If I flush the host cache (rm -rf /var/cache/lxc/cloud-trusty) and then 'juju add-machine' both caches get populated.