glance add can fail for images larger than 2 GB on some Python installations (RHEL 6.2)

[Chris Hoge]

Some Python installations, in particular for RHEL 6.2 Python 2.6, are not 64 bit clean, and "glance add" will fail. For example, where testimage.img is larger than 2GB

glance --debug add name="testimage" disk_format=raw is_public=true < testimage.img
python: Objects/fileobject.c:2117: readahead_get_line_skip: Assertion `skip+len < 2147483647' failed.

The underlying problem is an INT_MAX macro in the fileobject.c method 'readahead_get_line_skip' uses a 32 bit value instead of a 64 bit value. The Python file iterator eventually calls down to this method, which causes a crash for large file sizes.

A workaround for RHEL users is to comment out the "elif self._iterable(body):" section for the image_iterator method in glance/common/client.py and fall back to the ImageBodyIterator.

While not directly a bug with Glance, it does impact a subset of users not running Ubuntu systems.

[Chris Hoge (hoge)]

Correction: we ran into this problem on RHEL 6.2.

[linuxdatacenter (linuxdatacenter)]

Hi,
I've run into this issue on ubuntu oneiric with python-glance version:

2012.1~e3-0ubuntu0~ppa1~oneiric1

[Eoghan Glynn (eglynn)]

Chris,

What exact version of the python-2.6 package are you using on RHEL?

Can you copy'n'paste the output of the following:

$ uname -a
$ rpm -qa | grep python-2.6

Thanks,
Eoghan

[Eoghan Glynn (eglynn)]

Linuxdatacenter,

Similar question for you, can you paste in the output of:

$ uname -a
$ dpkg -s python2.6

Cheers,
Eoghan

[Chris Hoge (hoge)]

Results of
$ uname -a
$ rpm -qa | grep python-2.6

Linux cn3 2.6.32-220.4.1.el6.x86_64 #1 SMP Thu Jan 19 14:50:54 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
python-2.6.6-29.el6.x86_64

[Eoghan Glynn (eglynn)]

Thanks for the quick response Chris.

Now I'm having difficulty re-producing the issue against glance latest, and in fact code inspection suggests that if falling through to the ImageBodyIterator clause in glance/common/client.py acts as a work-around in your case, then the problem should have been resolved by this commit:

  https://github.com/openstack/glance/commit/40b5ed73

as the codepaths followed in the presence of the read attr would be equivalent.

Is the glance version you're testing against slightly behind the curve? You can check very quickly with:

$ grep -A 1 "elif self._iterable" glance/common/client.py

Output like:

        elif self._iterable(body):
            return utils.chunkreadable(body)

indicates that you have the relevant commit, whereas:

        elif self._iterable(body):
            return body

indicates that you need to update to latest.

Can you try the above grep and get back to me?

Thanks,
Eoghan

[Chris Hoge (hoge)]

Yes, my branch is about 7 days old now. I'll update and test (and be sure to be on latest when I run into problems :-).

[Eoghan Glynn (eglynn)]

Thanks Chris - let me know who you fare after updating, so that I can close off the bug if indeed it's not reproducible against latest.

[Chris Hoge (hoge)]

That fixed it here. Thanks.

I also submitted a bug against the RHEL 6 build of Python, fwiw.

[Eoghan Glynn (eglynn)]

Thanks Chris, closing off this bug as invalid as the fix is already in.

[Eoghan Glynn (eglynn)]

Actually Fix Committed is a more accurate status, as the bug was orignally kosher but had been fixed as a side-effect of:

 https://github.com/openstack/glance/commit/40b5ed73