Horizon Glance calling get_image() instead of get_image_meta()

[Jay Pipes]

From glance-api.log on the Alpha free cloud server, using filesystem store driver, we're seeing lots of these:

2011-09-28 17:46:12 DEBUG [glance.store.filesystem] Found image at /opt/stack/glance/images/3. Returning in ChunkedFile.
2011-09-28 17:46:12 DEBUG [eventlet.wsgi.server] Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 351, in handle_one_response
    write(''.join(towrite))
  File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 301, in write
    _writelines(towrite)
  File "/usr/lib/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 283, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 269, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer

I think the connection isn't being properly closed in the ChunkedFileIterator...

It doesn't seem to be affecting the functionality of the GET /images/<ID> but it's annoying to see it all over the log files...

[Jay Pipes (jaypipes)]

After further investigation, I actually don't think this is due to the ChunkedFile iterator, but rather to the WSGI server code not catching connection reset errors and ignoring them...

[livemoon (mwjpiero)]

When I click on "images" webpage in dashboard, if there is images, the errors above occurs immediately.
And glance-api may down sometime.
I use git install dialbo on ubuntu 11.04

[Jay Pipes (jaypipes)]

When you say "glance-api may down sometime", does that mean the server restarts? If you use the glance CLI tool, do you see the same error show up in the glance-api log?

We're trying to determine if this is isolated to Django using Glance, or whether it is a Glance issue...

-jay

[livemoon (mwjpiero)]

No other errlog in glance-api log, only errors pasted above.

Only glance-api service down,not restart,just down and have no glance-api process in `ps aux`. Then I start glance-api, it continue to work well.

[livemoon (mwjpiero)]

Today when I click on "instances" page in dashboard, errors occurs, not "images" pages

[Boris Deschenes (boris-michel-deschenes)]

I think Jay Pipes might be right,

I get this:

  File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 283, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 269, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer

And I'm not using horizon at all, I'm trying to spawn an image on Xen, Xen is probably not configured properly so it crashes/closes connection while downloading the image and glance reports the error listed above... it could be that it does not properly handles a closed connection... just wanted to let you know.

[Jay Pipes (jaypipes)]

I think I have figured out what's going on... still investigating but I can reproduce this easily in Horizon. It seems that the Horizon code is calling the wrong method of the glance client -- calling glance.client.Client.get_image() instead of get_image_meta(). get_image() returns the metadata and a file iterator object, and the Horizon code is simply ignoring the file iterator object which, upon garbage collection, tries to reset the socket connection in eventlet...

I think the solution is to patch horizon/horizon/api/glance.py:70-71 from this:

def image_get(request, image_id):
    return Image(glance_api(request).get_image(image_id)[0])

to this:

def image_get(request, image_id):
    return Image(glance_api(request).get_image_meta(image_id))

I will propose a patch to Horizon that fixes the issue.

Cheers,
-jay

[Jay Pipes (jaypipes)]

https://review.openstack.org/2104

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/2104
Committed: http://github.com/openstack/horizon/commit/068c782d03fe5fff793505f56623fcd70f465927
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 068c782d03fe5fff793505f56623fcd70f465927
Author: Jay Pipes <email address hidden>
Date: Tue Dec 6 12:27:19 2011 -0500

    Fixes LP Bug#862664 - Improper calls to get_image

    The glance.client.Client.get_image() call returns a
    tuple of (metadata, image_iterator). Unfortunately,
    Horizon's glance API calls get_image() when it means
    to call get_image_meta(). Because the call to get_image()
    simply ignores the image iterator returned from get_image(),
    when the image iterator is garbage-collected, this causes
    the connection to Glance to be closed, however by that time
    the socket bound to the iterator has been switched out by
    eventlet. The result is lots of these in the Glance API log:

    2011-09-28 17:46:12 DEBUG [glance.store.filesystem] Found image at /opt/stack/glance/images/3. Returning in ChunkedFile.
    2011-09-28 17:46:12 DEBUG [eventlet.wsgi.server] Traceback (most recent call last):
        File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 351, in handle_one_response
            write(''.join(towrite))
          File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 301, in write
            _writelines(towrite)
          File "/usr/lib/python2.7/socket.py", line 334, in writelines
            self.flush()
          File "/usr/lib/python2.7/socket.py", line 303, in flush
            self._sock.sendall(view[write_offset:write_offset+buffer_size])
          File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 283, in sendall
            tail = self.send(data, flags)
          File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 269, in send
            total_sent += fd.send(data[total_sent:], flags)
        error: [Errno 104] Connection reset by peer

    This patch fixes the improper calls to get_image() by replacing them
    with appropriate calls to get_image_meta().

    Change-Id: I741a207ba0e222820492aeb48bab9464d17539ab

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/diablo
Review: https://review.openstack.org/4486