Disk usage leak on image import

Bug #1771928 reported by Alexander Kharkov on 2018-05-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
nova-lxd
High
Unassigned
nova-lxd (Ubuntu)
High
Unassigned

Bug Description

Nova LXD driver contains a disk usage resource leak.
Each disk image import from Glance leads to O(image_size) disk space lost in /tmp.
Leak exists until nova-compute service is restarted as during restart open descriptors will be freed and disk space will reclaimed by OS.

The reason is wrong usage of mkstemp in nova/virt/lxd/driver.py _sync_glance_image_to_lxd.
The code use mkstemp to generate "random" filename to pass to glance API as destination path for downloaded image. But mkstemp also open file description to this file which is lost and never closed. As result even when this temporary file (downloaded image) is deleted file is still locked by nova-compute process.

def _sync_glance_image_to_lxd(client, context, image_ref):
...
    try:
        image_file = tempfile.mkstemp()[1]
        # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        # in code above we lost msktemp[0] of tuple which is open file handle
        # so file never be 'freed' until process finished
...
        IMAGE_API.download(context, image_ref, dest_path=image_file)
....

Alexander Kharkov (shurka.xaa) wrote :

Just a simple test which tries to create instant from 'bad' image uploaded to glance
show that disk space is lost. In my case it was ~= 800Mb image

root@shura-edge1:~# lsof -P 2>/dev/null | grep nova-com | grep /tmp
nova-comp 25306 nova 10u unix 0xffff88007795c000 0t0 2128952 /tmp/tmpybLoe9/privsep.sock type=STREAM
nova-comp 25306 nova 17u REG 253,1 884226169 10489 /tmp/tmpDcl9E7 (deleted)
nova-comp 25306 nova 18u REG 253,1 0 26813 /tmp/tmpGX3y7K (deleted)
nova-comp 25306 nova 26u REG 253,1 884226169 56992 /tmp/tmpWMDoKu (deleted)
nova-comp 25306 nova 27u REG 253,1 0 57308 /tmp/tmp4eR46t (deleted)
nova-comp 25306 nova 30u REG 253,1 884226169 57334 /tmp/tmpFTMcjK (deleted)
nova-comp 25306 nova 31u REG 253,1 0 57394 /tmp/tmpfZOfB1 (deleted)

Changed in nova-lxd:
status: New → Triaged
Changed in nova-lxd (Ubuntu):
status: New → In Progress
Changed in nova-lxd:
status: Triaged → In Progress
importance: Undecided → High

Reviewed: https://review.openstack.org/569325
Committed: https://git.openstack.org/cgit/openstack/nova-lxd/commit/?id=9c411c88280e71a0643f66d8566c73c36fca1647
Submitter: Zuul
Branch: master

commit 9c411c88280e71a0643f66d8566c73c36fca1647
Author: Alexander Kharkov <email address hidden>
Date: Fri May 18 04:52:06 2018 +0000

    Fix file descriptors leak which as result disk usage leak
    produced by wrong use of python tempfile.mkstemp

    Change-Id: Ia5e015c4d7a838f7e7701e078204e7e9d0d363bb
    Closes-Bug: 1771928

Changed in nova-lxd:
status: In Progress → Fix Committed
Changed in nova-lxd (Ubuntu):
status: In Progress → Fix Committed
Corey Bryant (corey.bryant) wrote :

Moving this back to Triaged for the nova-lxd package until we get a new package version uploaded to Cosmic (Rocky).

Changed in nova-lxd (Ubuntu):
importance: Undecided → High
status: Fix Committed → Triaged
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers