Horizon Glance calling get_image() instead of get_image_meta()

Bug #862664 reported by Jay Pipes on 2011-09-29
46
This bug affects 9 people
Affects Status Importance Assigned to Milestone
OpenStack Dashboard (Horizon)
Medium
Jay Pipes

Bug Description

From glance-api.log on the Alpha free cloud server, using filesystem store driver, we're seeing lots of these:

2011-09-28 17:46:12 DEBUG [glance.store.filesystem] Found image at /opt/stack/glance/images/3. Returning in ChunkedFile.
2011-09-28 17:46:12 DEBUG [eventlet.wsgi.server] Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 351, in handle_one_response
    write(''.join(towrite))
  File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 301, in write
    _writelines(towrite)
  File "/usr/lib/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 283, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 269, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer

I think the connection isn't being properly closed in the ChunkedFileIterator...

It doesn't seem to be affecting the functionality of the GET /images/<ID> but it's annoying to see it all over the log files...

Jay Pipes (jaypipes) on 2011-09-30
Changed in glance:
assignee: nobody → Jay Pipes (jaypipes)
status: Confirmed → In Progress
Jay Pipes (jaypipes) wrote :

After further investigation, I actually don't think this is due to the ChunkedFile iterator, but rather to the WSGI server code not catching connection reset errors and ignoring them...

Changed in glance:
assignee: Jay Pipes (jaypipes) → nobody
milestone: essex-1 → none
status: In Progress → Confirmed
livemoon (mwjpiero) wrote :

When I click on "images" webpage in dashboard, if there is images, the errors above occurs immediately.
And glance-api may down sometime.
I use git install dialbo on ubuntu 11.04

Changed in glance:
status: Confirmed → Incomplete
Jay Pipes (jaypipes) wrote :

When you say "glance-api may down sometime", does that mean the server restarts? If you use the glance CLI tool, do you see the same error show up in the glance-api log?

We're trying to determine if this is isolated to Django using Glance, or whether it is a Glance issue...

-jay

livemoon (mwjpiero) wrote :

No other errlog in glance-api log, only errors pasted above.

Only glance-api service down,not restart,just down and have no glance-api process in `ps aux`. Then I start glance-api, it continue to work well.

livemoon (mwjpiero) wrote :

Today when I click on "instances" page in dashboard, errors occurs, not "images" pages

Jay Pipes (jaypipes) on 2011-11-16
summary: - Repeated connection reset errors when returning ChunkedFile
+ Repeated connection reset errors when calling from Horizon

I think Jay Pipes might be right,

I get this:

  File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 283, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 269, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer

And I'm not using horizon at all, I'm trying to spawn an image on Xen, Xen is probably not configured properly so it crashes/closes connection while downloading the image and glance reports the error listed above... it could be that it does not properly handles a closed connection... just wanted to let you know.

Jay Pipes (jaypipes) wrote :

I think I have figured out what's going on... still investigating but I can reproduce this easily in Horizon. It seems that the Horizon code is calling the wrong method of the glance client -- calling glance.client.Client.get_image() instead of get_image_meta(). get_image() returns the metadata and a file iterator object, and the Horizon code is simply ignoring the file iterator object which, upon garbage collection, tries to reset the socket connection in eventlet...

I think the solution is to patch horizon/horizon/api/glance.py:70-71 from this:

def image_get(request, image_id):
    return Image(glance_api(request).get_image(image_id)[0])

to this:

def image_get(request, image_id):
    return Image(glance_api(request).get_image_meta(image_id))

I will propose a patch to Horizon that fixes the issue.

Cheers,
-jay

Changed in glance:
status: Incomplete → Triaged
affects: glance → horizon
Changed in horizon:
assignee: nobody → Jay Pipes (jaypipes)
milestone: none → essex-2
Jay Pipes (jaypipes) on 2011-12-06
summary: - Repeated connection reset errors when calling from Horizon
+ Horizon Glance calling get_image() instead of get_image_meta()
Devin Carlen (devcamcar) on 2011-12-06
Changed in horizon:
status: Triaged → Fix Committed

Reviewed: https://review.openstack.org/2104
Committed: http://github.com/openstack/horizon/commit/068c782d03fe5fff793505f56623fcd70f465927
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 068c782d03fe5fff793505f56623fcd70f465927
Author: Jay Pipes <email address hidden>
Date: Tue Dec 6 12:27:19 2011 -0500

    Fixes LP Bug#862664 - Improper calls to get_image

    The glance.client.Client.get_image() call returns a
    tuple of (metadata, image_iterator). Unfortunately,
    Horizon's glance API calls get_image() when it means
    to call get_image_meta(). Because the call to get_image()
    simply ignores the image iterator returned from get_image(),
    when the image iterator is garbage-collected, this causes
    the connection to Glance to be closed, however by that time
    the socket bound to the iterator has been switched out by
    eventlet. The result is lots of these in the Glance API log:

    2011-09-28 17:46:12 DEBUG [glance.store.filesystem] Found image at /opt/stack/glance/images/3. Returning in ChunkedFile.
    2011-09-28 17:46:12 DEBUG [eventlet.wsgi.server] Traceback (most recent call last):
        File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 351, in handle_one_response
            write(''.join(towrite))
          File "/usr/lib/pymodules/python2.7/eventlet/wsgi.py", line 301, in write
            _writelines(towrite)
          File "/usr/lib/python2.7/socket.py", line 334, in writelines
            self.flush()
          File "/usr/lib/python2.7/socket.py", line 303, in flush
            self._sock.sendall(view[write_offset:write_offset+buffer_size])
          File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 283, in sendall
            tail = self.send(data, flags)
          File "/usr/lib/pymodules/python2.7/eventlet/greenio.py", line 269, in send
            total_sent += fd.send(data[total_sent:], flags)
        error: [Errno 104] Connection reset by peer

    This patch fixes the improper calls to get_image() by replacing them
    with appropriate calls to get_image_meta().

    Change-Id: I741a207ba0e222820492aeb48bab9464d17539ab

Thierry Carrez (ttx) on 2011-12-14
Changed in horizon:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in horizon:
milestone: essex-2 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers