OpenStack Image Registry and Delivery Service (Glance)

glance cache rollback causes 'OSError: [Errno 2] No such file or directory '

Reported by Dan Prince on 2011-11-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Low
Dan Prince

Bug Description

I'm getting the following stack trace from the Glance sqlite cache driver:

2011-11-09 19:18:18 DEBUG [routes.middleware] Match dict: {'action': u'show', 'controller': <glance.common.wsgi.Resource object at 0x1e2a650>, 'id': u'36d231ec-26d8-4ff1-99c2-48d3ef53feac'}
2011-11-09 19:18:18 DEBUG [glance.store.filesystem] Found image at /var/lib/glance/images/36d231ec-26d8-4ff1-99c2-48d3ef53feac. Returning in ChunkedFile.
2011-11-09 19:18:18 DEBUG [glance.image_cache] Tee'ing image '36d231ec-26d8-4ff1-99c2-48d3ef53feac' into cache
2011-11-09 19:18:18 DEBUG [glance.image_cache.drivers.sqlite] Fetch of cache file failed, rolling back by moving '/var/lib/glance/image-cache/incomplete/36d231ec-26d8-4ff1-99c2-48d3ef53feac' to '/var/lib/glance/image-cache/invalid/36d231ec-26d8-4ff1-99c2-48d3ef53feac'
2011-11-09 19:18:18 DEBUG [eventlet.wsgi.server] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 347, in handle_one_response
    for data in result:
  File "/usr/lib/python2.7/dist-packages/glance/image_cache/__init__.py", line 200, in tee_iter
    with self.driver.open_for_write(image_id) as cache_file:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/glance/image_cache/drivers/sqlite.py", line 322, in open_for_write
    rollback(e)
  File "/usr/lib/python2.7/dist-packages/glance/image_cache/drivers/sqlite.py", line 312, in rollback
    os.rename(incomplete_path, invalid_path)
OSError: [Errno 2] No such file or directory

---

This exception masks the root cause of the error (in my case a file system permission issue).

During the rollback we should have a simple exists check to make sure files exist before attempting to rename them.

Dan Prince (dan-prince) on 2011-11-09
Changed in glance:
assignee: nobody → Dan Prince (dan-prince)
importance: Undecided → Low
status: New → In Progress
Jay Pipes (jaypipes) wrote :

Thanks, Dan. Nice catch.

Changed in glance:
milestone: none → essex-2

Reviewed: https://review.openstack.org/1471
Committed: http://github.com/openstack/glance/commit/a7db2db948626e6727a88f50319306bf48dcc1d9
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit a7db2db948626e6727a88f50319306bf48dcc1d9
Author: Dan Prince <email address hidden>
Date: Wed Nov 9 14:26:11 2011 -0500

    Check to make sure the incomplete file exists before moving it during
    rollback. Fixes bug #888241.

    Change-Id: I98bca574a26f9ab7173ac593d3b32b68d78d2360

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2011-12-14
Changed in glance:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in glance:
milestone: essex-2 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers