urlib.request fails to open Eucalyptus metadata service

Bug #813295 reported by Scott Moser
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Eucalyptus
New
Undecided
chris grzegorczyk
Python
Fix Released
Unknown
python2.7 (Ubuntu)
High
Unassigned

Bug Description

Oneiric alpha-2 images do not run on eucalyptus.

I've tried running an oneiric alpha2 image on 2 different eucalyptus clouds (internal canonical, and public eucalyptus cloud). Both instances have timed out on metadata service.

I really do not think this is a cloud-init bug at the moment, but am opening it here until i find more information.

The images *did* run in alpha1.

One bit of possibly relevant information, both clouds are running the oneiric kernel after kexec from a lucid era kernel using https://code.launchpad.net/~smoser/+junk/kexec-loader .

Related branches

Revision history for this message
Scott Moser (smoser) wrote :
description: updated
Revision history for this message
Scott Moser (smoser) wrote : Re: oneiric images do not run in eucalyptus
summary: - oneiric alpha2 images do not run in eucalyptus
+ oneiric images do not run in eucalyptus
description: updated
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :

the above log in comment 3 is from 20110719 image. So this is still broken.

Revision history for this message
Scott Moser (smoser) wrote :

I have tried booting alpha1 image, and then upgraded each of the following (1 by 1) to archive level:
 - cloud-init
 - udev
 - linux-virtual
then 'rm -Rf /var/lib/cloud && reboot'

The system has come back up in all cases.
I'm now checking a full 'apt-get dist-upgrade' from alpha1 -> current archive and seeing if that breaks.

Revision history for this message
Scott Moser (smoser) wrote :

Well, partially good news.
launching an alpha1 instance (i386) and then
 - apt-et update && apt-get dist-upgrade
 - rm -Rf /var/lib/cloud
 - reboot
Then failed to come back up all the way to reachable ssh.

Due to a bug in eucalpytus in lucid, i cna't give a useful console log. However, I did copy off the output of 'dpkg-query --show' before reboot. That serves to limit the list of packages to try 1x1.

Revision history for this message
Dave Walker (davewalker) wrote :

Have you been able to reproduce this on AWS or Nova?

Thanks.

tags: added: server-o-rs
Dave Walker (davewalker)
Changed in cloud-init (Ubuntu):
importance: Undecided → High
Revision history for this message
Scott Moser (smoser) wrote : Re: [Bug 813295] Re: oneiric images do not run in eucalyptus

On Wed, 20 Jul 2011, Dave Walker wrote:

> Have you been able to reproduce this on AWS or Nova?

Not on AWS. I have not tried on Nova.

Revision history for this message
Scott Moser (smoser) wrote : Re: oneiric images do not run in eucalyptus

ok. some progress. After guessing a *slew* of other packages that affect early boot, and upgrading them in small groups, i finally got to python*. upgrading one of the following packages causes the issue:

libpython2.7 python python-minimal python2.7 python2.7-minimal

tags: added: python-2.7
Revision history for this message
Scott Moser (smoser) wrote :

looks like it was a change in python2.7 between 2.7.2~rc1-2 and 2.7.2-2.

Revision history for this message
Scott Moser (smoser) wrote :

I'm attaching a simple reproducing test case.
My guess is that the issue is that the [lack of] headers that the Eucalyptus meta data service returns causes issues with python-urllib2.

This testcase works fine on alpha-1 and fails after upgrade.

Here are the differences in headers sent by EC2 (where it works) and Eucalyptus (where it fails):

=== EC2 Metadata Service ===
$ wget -S -q -O /dev/null http://169.254.169.254/latest/meta-data/instance-id
  HTTP/1.0 200 OK
  Content-Type: text/plain
  Accept-Ranges: bytes
  ETag: "3417059440"
  Last-Modified: Wed, 20 Jul 2011 19:41:04 GMT
  Content-Length: 10
  Connection: keep-alive
  Date: Wed, 20 Jul 2011 19:46:38 GMT
  Server: EC2ws

=== Eucalyptus Metadata Service ===
$ wget -S -q -O /dev/null http://169.254.169.254/latest/meta-data/instance-id
  HTTP/1.0 200 OK
  Content-Length: 10
  Content-Type: text/html

affects: cloud-init (Ubuntu) → python2.7 (Ubuntu)
Changed in python2.7 (Ubuntu):
status: New → Confirmed
Revision history for this message
Scott Moser (smoser) wrote :

looking at changes between alpha1 and alpha2 (tag:2.7.2~rc1-2..tag:2.7.2-2) one change that looks suspicious is

diff -r 8527427914a2 Lib/urllib2.py
--- a/Lib/urllib2.py
+++ b/Lib/urllib2.py
@@ -1172,6 +1172,8 @@
                 r = h.getresponse()
         except socket.error, err: # XXX what error?
             raise URLError(err)
+ finally:
+ h.close()

         # Pick apart the HTTPResponse object to get the addinfourl
         # object initialized properly.

that came from upstream changeset 70837:18e6ccc332d5 [1] that reports to fix issue 12133 [2]

--
[1] http://code.activestate.com/lists/python-checkins/97757/
[2] http://bugs.python.org/issue12133

Revision history for this message
Barry Warsaw (barry) wrote :

Looks like Python issue 12576 is tracking the regression in this case. It's primarily focused on Python 3, but I think it also affects Python 2.7 in this case. I'll update that bug report.

Revision history for this message
Barry Warsaw (barry) wrote :
Scott Moser (smoser)
description: updated
summary: - oneiric images do not run in eucalyptus
+ urlib.request fails to open some sites
summary: - urlib.request fails to open some sites
+ urlib.request fails to open Eucalyptus metadata service
Changed in python:
status: Unknown → New
Daniel Nurmi (nurmi)
Changed in eucalyptus:
assignee: nobody → chris grzegorczyk (chris-grze)
Changed in python:
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python2.7 - 2.7.2-3ubuntu1

---------------
python2.7 (2.7.2-3ubuntu1) oneiric; urgency=low

  [ Scott Moser ]
  * Apply upstream Python 2.7 patch for Python issue 12576.
    (LP: #813295)
 -- Barry Warsaw <email address hidden> Thu, 28 Jul 2011 17:42:58 -0400

Changed in python2.7 (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

Verified fixed in 20110822.2 images.

Revision history for this message
Andy Grimm (agrimm) wrote :

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2777

Please watch that issue for further updates.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.