Test on the xattr image cache fails because of noatime

[Victor Stinner]

The check job gate-glance-python27 failed on my patch https://review.openstack.org/#/c/197916/

I added debug traces and it looks like the OpenStack infra uses a ext4 partition mounted with noatime, whereas the xattr image cache driver explicitly requires atime.

Debug trace: http://logs.openstack.org/16/197916/6/check/gate-glance-python27/16760bf/testr_results.html.gz

glance.tests.unit.test_image_cache.TestImageCacheXattr.test_caching_iterator_falloffend
Traceback (most recent call last):
  File "glance/tests/unit/test_image_cache.py", line 411, in test_caching_iterator_falloffend
    self.assertFalse(os.path.exists(incomplete_file_path), debug_info)
  File "/home/jenkins/workspace/gate-glance-python27/.tox/py27/local/lib/python2.7/site-packages/unittest2/case.py", line 696, in assertFalse
    raise self.failureException(msg)
AssertionError: True is not false : /tmp/tmp.NOhmw2VXFN/tmpjnooYj

Content of the image cache:

/tmp/tmp.NOhmw2VXFN/tmpjnooYj/queue
/tmp/tmp.NOhmw2VXFN/tmpjnooYj/incomplete
/tmp/tmp.NOhmw2VXFN/tmpjnooYj/incomplete/1
/tmp/tmp.NOhmw2VXFN/tmpjnooYj/invalid

Filesystem of the image cache (/tmp):

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 41147472 11923388 27320644 31% /

Filesystem mounted with:

/dev/xvda1 on / type ext4 (rw,noatime,errors=remount-ro,barrier=0)

Notice "noatime" in the mount options of /dev/xvda1.

The test glance.tests.unit.test_image_cache.TestImageCacheXattr.test_caching_iterator_falloffend pass on my Fedora 22. On Fedora 22, the image cache is created on /tmp which mounted as tmpfs.

[Victor Stinner (vstinner)]

> whereas the xattr image cache driver explicitly requires atime.

See the first assumption of the driver:
https://github.com/openstack/glance/blob/c41216d3fcc74a978246ff7680155077d04998c5/glance/image_cache/drivers/xattr.py#L23

Assumptions
===========

1. Cache data directory exists on a filesytem that updates atime on reads ('noatime' should NOT be set)

[Victor Stinner (vstinner)]

I discussed on #openstack-infra. Rackspace recently changed kernel boot options on base images recently. So maybe the noatime option was added during this change.

There are two options to fix this issue:

* Convert the Glance unit test to a Glance functional test: for a functional test, we have the full control on the system (root access), you can remount the root fs (/) with the "right" mount option (atime, relatime, etc.)

* Mock the filesystem, at least the atime timestamp, in the unit test

IHMO the latter is simpler, but I didn't have time yet to check if it's feasible.

[Jeremy Stanley (fungi)]

I do find it relatively silly that one of our providers has decided to set noatime on their base images (that's not been much help to anyone since linux 2.6.30 switched the default to relatime, so atime's not especially I/O-intensive any longer).

That aside, I agree that having your unit tests depend on options presented by the local machine's filesystem is not great from an isolation standpoint. Telling your developers what filesystem options they'll need to set or unset on their machines before they can run unit tests on patches they're writing/reviewing is sort of an awkward situation to be in as a project.

[Ian Cordasco (icordasc)]

So, I did some investigation into our use of atime in glance. First, I noticed that we don't really rely on the last_accessed timestamp that we get back. We never actually use it. The only two places it's "used" is when listing cached images with `glance cache-manage` and in the `/v1/cached_images` endpoint. Other than that, we don't rely on it, and it isn't a critical piece of information for the operation of glance or its cache.

That said, in the module level docstring for the driver, Jay Pipes (going by git blame) specifically wrote that to use it you needed to ensure that noatime was not set. This is reasonable. The thing is that in glance_store, we will disable a driver if we're missing a prerequisite (e.g., a dependency, configuration, etc.). It's surprising to me that we aren't doing this with the cache drivers as well. It would seem perfectly logical to disable the xattr driver if the filesystem has noatime set.

Does this sound fair to everyone else?