Lockfiles for fetching images remain forever

[Arata Notsu]

When fetches an image from Glance, LibvirtDriver creates a lockfile that has a name corresponding to the image. It is not deleted automatically even if the image is delete from Glance. So lockfiles may accumulate in long term. Although this may be not so harmful, a little bit ugly.

The following is a list of files under /var/lib/nova in my compute node. Many old lockfiles are left in /var/lib/nova/instances/locks.

$ ls -lR /var/lib/nova/instances/
/var/lib/nova/instances/:
total 16
drwxr-xr-x 2 nova nova 4096 Nov 2 18:38 _base
-rw-r--r-- 1 nova nova 37 Nov 29 21:03 compute_nodes
drwxr-xr-x 2 nova nova 4096 Nov 1 17:59 locks
drwxr-xr-x 2 nova nova 4096 Sep 11 17:41 snapshots

/var/lib/nova/instances/_base:
total 24840
-rw-r--r-- 1 nova nova 20068399 Oct 4 20:30 03d8e206-6500-4d91-b47d-ee74897f9b4e
-rw-r--r-- 1 nova nova 5362320 Oct 4 20:30 cc60167d-4d4a-45af-afbb-8757aa6b7946

/var/lib/nova/instances/locks:
total 0
-rw-r--r-- 1 nova nova 0 Oct 4 20:40 nova-03d8e206-6500-4d91-b47d-ee74897f9b4e
-rw-r--r-- 1 nova nova 0 Oct 28 20:22 nova-46da6595b4561f079dc41f96efa7ab6881c5c1e9
-rw-r--r-- 1 nova nova 0 Oct 4 20:30 nova-5ca6db469255e9ab1ffa96cce880b1fc96fe0a4d
-rw-r--r-- 1 nova nova 0 Sep 11 17:26 nova-9e881789030568a317fad9daae82c5b1c65e0d4a
-rw-r--r-- 1 nova nova 0 Sep 16 18:57 nova-ab3ee132785116ec5192292c364e56a750f37f5b
-rw-r--r-- 1 nova nova 0 Sep 27 14:31 nova-abccd35ab48d01849de73d759104bb09c210bef7
-rw-r--r-- 1 nova nova 0 Nov 1 17:59 nova-b779965eca6fd7bfb1c4e34f8c3e9eb10aa97d6a
-rw-r--r-- 1 nova nova 0 Oct 4 20:40 nova-cc60167d-4d4a-45af-afbb-8757aa6b7946
-rw-r--r-- 1 nova nova 0 Oct 7 10:49 nova-f70c681bc242f3a6f6a64a5c28ce05e3be846b6a
-rw-r--r-- 1 nova nova 0 Oct 4 20:33 nova-fa750326e354854db61fe2d30d0d7e1049f4ea1b
-rw-r--r-- 1 nova nova 0 Nov 29 21:03 nova-storage-registry-lock
-rw-r--r-- 1 nova nova 0 Oct 4 20:40 nova-_var_lib_nova_instances__base_03d8e206-6500-4d91-b47d-ee74897f9b4e
-rw-r--r-- 1 nova nova 0 Oct 28 20:22 nova-_var_lib_nova_instances__base_46da6595b4561f079dc41f96efa7ab6881c5c1e9
-rw-r--r-- 1 nova nova 0 Oct 4 20:40 nova-_var_lib_nova_instances__base_5ca6db469255e9ab1ffa96cce880b1fc96fe0a4d
-rw-r--r-- 1 nova nova 0 Sep 11 17:26 nova-_var_lib_nova_instances__base_9e881789030568a317fad9daae82c5b1c65e0d4a
-rw-r--r-- 1 nova nova 0 Sep 16 18:57 nova-_var_lib_nova_instances__base_ab3ee132785116ec5192292c364e56a750f37f5b
-rw-r--r-- 1 nova nova 0 Sep 27 14:31 nova-_var_lib_nova_instances__base_abccd35ab48d01849de73d759104bb09c210bef7
-rw-r--r-- 1 nova nova 0 Nov 1 17:59 nova-_var_lib_nova_instances__base_b779965eca6fd7bfb1c4e34f8c3e9eb10aa97d6a
-rw-r--r-- 1 nova nova 0 Oct 4 20:40 nova-_var_lib_nova_instances__base_cc60167d-4d4a-45af-afbb-8757aa6b7946
-rw-r--r-- 1 nova nova 0 Oct 24 16:39 nova-_var_lib_nova_instances__base_f70c681bc242f3a6f6a64a5c28ce05e3be846b6a
-rw-r--r-- 1 nova nova 0 Oct 4 20:33 nova-_var_lib_nova_instances__base_fa750326e354854db61fe2d30d0d7e1049f4ea1b

/var/lib/nova/instances/snapshots:
total 0

[Joe Gordon (jogo)]

This sounds like an actual bug, but a cosmetic one which should be triaged as low (https://wiki.openstack.org/wiki/BugTriage#Task_2:_Prioritize_confirmed_bugs_.28bug_supervisors.29)

[jichenjc (jichenjc)]

I think if we can keep the lock file for the whole life cycle of instance will be helpful to avoid create and delete lock file many times
so remove the file at the end of lifecycle of the instance (e.g delete the instance) will be efficient

[jichenjc (jichenjc)]

Also, there are a lot of files which may exist , guess we may only need to consider

more lock files might be considered for read_stored_info//write_stored_info functions in libvirt imagecache when the checksum features is enabled .

        lock_name = 'info-%s' % os.path.split(target)[-1]
        lock_path = os.path.join(CONF.instances_path, 'locks')

        @utils.synchronized(lock_name, external=True, lock_path=lock_path)

[Michael Still (mikal)]

This is actually an oslo locking bug, we're just using their lockutils implementation.

[jichenjc (jichenjc)]

Sorry, should we ask them to specify a method to discard the lock file?
looks to me their lock mechanism works fine, I think it might be not easy to automatically delete the lock file ,
so I prefer to do it in nova side (e.g every image file delete will cause the lock file to be deleted)
if we can get a common method from oslo lock module, that will be great, otherwise, will implement in nova to delete the file

do anyone know how other openstack component handle this?

[jichenjc (jichenjc)]

I think some function in oslo should be provided such as 'delete_lock_file', will learn how to work on it .....

[jichenjc (jichenjc)]

when checksum is enabled, some locks such as 'info-xxx' and 'hash-xxx' while xxx represents the image name will also be created and remained forever, they should also be removed when the image is gone

this will also be fixed in the patch ( I opened another bug but looks like we need to track them in same place)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/69299

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/69299
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=b0d0c33521d143298ac13dbf1f478a96781b4f72
Submitter: Jenkins
Branch: master

commit b0d0c33521d143298ac13dbf1f478a96781b4f72
Author: jichenjc <email address hidden>
Date: Sat Jan 25 06:35:31 2014 +0800

    Add remove external lock files API in lockutils

    when external lock are used, there will be a lock files which
    is used for external file lock, it will not be removed and remain
    forever. it will be a problem when tens of thousands of this kind
    of actions. Add a API to allow other component to remove lock file
    internal lock will be acquired before further action

    Change-Id: I4d996b91bcc55becb0ebf4ec77ac80546f433ae9
    Closes-Bug: 1256306

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97424
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=01b208a06c6841b7e03e2a8657941cad1947b453
Submitter: Jenkins
Branch: master

commit 01b208a06c6841b7e03e2a8657941cad1947b453
Author: jichenjc <email address hidden>
Date: Fri May 30 09:52:09 2014 +0800

    Sync oslo lockutils to nova

    There is some update for lockutils in oslo, sync from oslo

    The commit to be merged:
    edd73c9 Merge "Improve help strings"
    e446fe5 Merge "Fixed several typos"
    fe3389e Improve help strings
    f3f14c9 Fixed several typos
    0f495ee Emit a log statement when releasing internal lock
    f0dd798 Remove rendundant parentheses of cfg help strings
    006d9a2 Allow external locks to work with threads
    9b73c19 Re-enable file-based locking behavior
    edce46c Use Posix IPC in lockutils
    ac84a40 Update log translation domains
    ee4df7f Merge "Add remove external lock files API in lockutils"
    fcf517d Update oslo log messages with translation domains
    37a1de8 Move the released file lock to the successful path
    3c125e6 Merge "Use threading.ThreadError instead of reraising IOError"
    de826f2 Merge "Have the interprocess lock follow lock conventions"
    b0d0c33 Add remove external lock files API in lockutils
    e71cd1a Merge "Trivial: Make vertical white space after license header consistent"
    4f6190a Use threading.ThreadError instead of reraising IOError
    195f0b1 Have the interprocess lock follow lock conventions
    15cca4b lockutils: move directory creation in lock class
    81fe39e lockutils: remove lock_path parameter
    14d3669 lockutils: expand add_prefix
    dc06d55 lockutils: do not grab the lock in creators
    a0948cb lockutils: remove local usage
    df8e051 lockutils: split code handling internal/external lock
    8b2b0b7 Use hacking import_exceptions for gettextutils._
    6d0a6c3 Correct invalid docstrings
    6fa29ae Trivial: Make vertical white space after license header consistent
    12bcdb7 Remove vim header
    79e6bc6 fix lockutils.lock() to make it thread-safe
    ace5120 Add main() to lockutils that creates temp dir for locks

    Related-bug: #1256306

    Change-Id: Iced751f059bd9a893ccefb5f98d63df29f40c622

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/103732

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/136986
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5d5ef1cc536ed32e4ea46ddfdeca6cad7fe59c9e
Submitter: Jenkins
Branch: master

commit 5d5ef1cc536ed32e4ea46ddfdeca6cad7fe59c9e
Author: jichenjc <email address hidden>
Date: Thu Nov 20 10:47:06 2014 +0800

    Update LVM lockfile name identical to RAW and Qcow

    Currently Raw and Qcow2 use filename as lock file identity
    while LVM use whole path. This patch change the lockfile
    name of LVM to identical to the others.
    It will be helpful to allow driver layer code to remove
    those files when the life cycle of an instance ends.

    Related-Bug: #1256306

    Change-Id: I07aa54032026ca479fe8de08308dae51527a6b13

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/144891

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144891
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0f90b2720dda0b5b656a10be0f879dfe3130a4c4
Submitter: Jenkins
Branch: master

commit 0f90b2720dda0b5b656a10be0f879dfe3130a4c4
Author: jichenjc <email address hidden>
Date: Fri Jan 2 00:06:35 2015 +0800

    Remove lock files when remove libvirt images

    When fetches an image from Glance, LibvirtDriver creates a
    lockfile that has a name corresponding to the image.
    It is not deleted automatically even if the image is delete
    from nova. So lockfiles may accumulate in long term.
    Although this may be not so harmful, a little bit ugly.

    So this patch removes the external lock files when the instance
    image file is deleted.

    Change-Id: Ia00388cc5a9dca84df8eea038d693f37940d494e
    Partial-Bug: #1256306