urlib.request fails to open Eucalyptus metadata service

[Scott Moser]

Oneiric alpha-2 images do not run on eucalyptus.

I've tried running an oneiric alpha2 image on 2 different eucalyptus clouds (internal canonical, and public eucalyptus cloud). Both instances have timed out on metadata service.

I really do not think this is a cloud-init bug at the moment, but am opening it here until i find more information.

The images *did* run in alpha1.

One bit of possibly relevant information, both clouds are running the oneiric kernel after kexec from a lucid era kernel using https://code.launchpad.net/~smoser/+junk/kexec-loader .

[Scott Moser (smoser)]

the above log in comment 3 is from 20110719 image. So this is still broken.

[Scott Moser (smoser)]

I have tried booting alpha1 image, and then upgraded each of the following (1 by 1) to archive level:
 - cloud-init
 - udev
 - linux-virtual
then 'rm -Rf /var/lib/cloud && reboot'

The system has come back up in all cases.
I'm now checking a full 'apt-get dist-upgrade' from alpha1 -> current archive and seeing if that breaks.

[Scott Moser (smoser)]

Well, partially good news.
launching an alpha1 instance (i386) and then
 - apt-et update && apt-get dist-upgrade
 - rm -Rf /var/lib/cloud
 - reboot
Then failed to come back up all the way to reachable ssh.

Due to a bug in eucalpytus in lucid, i cna't give a useful console log. However, I did copy off the output of 'dpkg-query --show' before reboot. That serves to limit the list of packages to try 1x1.

[Dave Walker (davewalker)]

Have you been able to reproduce this on AWS or Nova?

Thanks.

[Scott Moser (smoser)]

On Wed, 20 Jul 2011, Dave Walker wrote:

> Have you been able to reproduce this on AWS or Nova?

Not on AWS. I have not tried on Nova.

[Scott Moser (smoser)]

ok. some progress. After guessing a *slew* of other packages that affect early boot, and upgrading them in small groups, i finally got to python*. upgrading one of the following packages causes the issue:

libpython2.7 python python-minimal python2.7 python2.7-minimal

[Scott Moser (smoser)]

looks like it was a change in python2.7 between 2.7.2~rc1-2 and 2.7.2-2.

[Scott Moser (smoser)]

I'm attaching a simple reproducing test case.
My guess is that the issue is that the [lack of] headers that the Eucalyptus meta data service returns causes issues with python-urllib2.

This testcase works fine on alpha-1 and fails after upgrade.

Here are the differences in headers sent by EC2 (where it works) and Eucalyptus (where it fails):

=== EC2 Metadata Service ===
$ wget -S -q -O /dev/null http://169.254.169.254/latest/meta-data/instance-id
  HTTP/1.0 200 OK
  Content-Type: text/plain
  Accept-Ranges: bytes
  ETag: "3417059440"
  Last-Modified: Wed, 20 Jul 2011 19:41:04 GMT
  Content-Length: 10
  Connection: keep-alive
  Date: Wed, 20 Jul 2011 19:46:38 GMT
  Server: EC2ws

=== Eucalyptus Metadata Service ===
$ wget -S -q -O /dev/null http://169.254.169.254/latest/meta-data/instance-id
  HTTP/1.0 200 OK
  Content-Length: 10
  Content-Type: text/html

[Scott Moser (smoser)]

looking at changes between alpha1 and alpha2 (tag:2.7.2~rc1-2..tag:2.7.2-2) one change that looks suspicious is

diff -r 8527427914a2 Lib/urllib2.py
--- a/Lib/urllib2.py
+++ b/Lib/urllib2.py
@@ -1172,6 +1172,8 @@
                 r = h.getresponse()
         except socket.error, err: # XXX what error?
             raise URLError(err)
+ finally:
+ h.close()

         # Pick apart the HTTPResponse object to get the addinfourl
         # object initialized properly.

that came from upstream changeset 70837:18e6ccc332d5 [1] that reports to fix issue 12133 [2]

--
[1] http://code.activestate.com/lists/python-checkins/97757/
[2] http://bugs.python.org/issue12133

[Barry Warsaw (barry)]

Looks like Python issue 12576 is tracking the regression in this case. It's primarily focused on Python 3, but I think it also affects Python 2.7 in this case. I'll update that bug report.

[Barry Warsaw (barry)]

http://bugs.python.org/issue12576

[Launchpad Janitor (janitor)]

This bug was fixed in the package python2.7 - 2.7.2-3ubuntu1

---------------
python2.7 (2.7.2-3ubuntu1) oneiric; urgency=low

  [ Scott Moser ]
  * Apply upstream Python 2.7 patch for Python issue 12576.
    (LP: #813295)
 -- Barry Warsaw <email address hidden> Thu, 28 Jul 2011 17:42:58 -0400

[Scott Moser (smoser)]

Verified fixed in 20110822.2 images.

[Andy Grimm (agrimm)]

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2777

Please watch that issue for further updates.