Unused kernels and ramdisks not removed by libvirt image cache manager

[Mark McLoughlin]

When the image cache manager runs, it only cleans up unused base disk images ... not kernels and ramdisks

This is clearly the understood behaviour of the code:

    def _list_base_images(self, base_dir):
        """Return a list of the images present in _base.

        Determine what images we have on disk. There will be other files in
        this directory (for example kernels) so we only grab the ones which
        are the right length to be disk images.
        ...
        """
        digest_size = hashlib.sha1().digestsize * 2
        for ent in os.listdir(base_dir):
            if len(ent) == digest_size:
                self._store_image(base_dir, ent, original=True)

and it's only the base disk images which are stored using their sha1 sum as their filename:

        if disk_images['kernel_id']:
            fname = disk_images['kernel_id']
            raw('kernel').cache(fetch_func=libvirt_utils.fetch_image,
                                context=context,
                                filename=fname,
                                ...)
            if disk_images['ramdisk_id']:
                fname = disk_images['ramdisk_id']
                raw('ramdisk').cache(fetch_func=libvirt_utils.fetch_image,
                                     context=context,
                                     filename=fname,
                                     ...)

        root_fname = hashlib.sha1(str(disk_images['image_id'])).hexdigest()
        ...
        if 'disk' in disk_mapping:
            image('disk').cache(fetch_func=libvirt_utils.fetch_image,
                                context=context,
                                filename=root_fname,
                                ...)

Assuming I'm not missing something and it does make sense to clean these up, I guess we can:

  1) start storing kernels and ramdisks using their sha1 sum as their filename or

  2) have the image cache manager somehow recognize the kernel/ramdisk filenames as images to examine

We also need the image cache manager to track kernels and ramdisks as used images when listing instances

i.e. need to include kernel_id and ramdisk_id here:

            image_ref_str = str(instance['image_ref'])
            local, remote, insts = self.used_images.get(image_ref_str,
                                                        (0, 0, []))

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/22777

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/22777
Committed: http://github.com/openstack/nova/commit/38997fc916904e47dcb847e1737eb26cbd38f6e1
Submitter: Jenkins
Branch: master

commit 38997fc916904e47dcb847e1737eb26cbd38f6e1
Author: Mark McLoughlin <email address hidden>
Date: Sat Feb 23 15:24:20 2013 +0000

    Clean unused kernels and ramdisks from image cache

    Fixes bug #1132138

    Only unused disk images are currently cleaned up by the image cache
    manager but it seems logical to clean up unused kernels and ramdisks
    too.

    Achieve that by writing kernels and ramdisks to disk using the sha1
    sum of their ID as the filename. This is the same scheme as used for
    disk image filenames and causes the image cache manager to consider
    them for cleanup. We also make the cache manager take note of in use
    kernels and ramdisks when iterating over the list of instances.

    A nasty upgrade concern is that if we immediately switch to writing
    kernels to disk using this scheme then, where shared storage is used,
    we can have older image cache managers on remote compute nodes cleaning
    up kernels because they appear unused. To mitigate that, turn off this
    behaviour by default and allow it to be enabled using a new config
    option. This option will be removed in future and the behaviour enabled
    by default.

    DocImpact: new remove_unused_kernels option

    Change-Id: I56bba9fa6596601104498e262c2e657f0eae2fa0