cloud-init hangs with url error 'Network is unreachable'

[Scott Moser]

Binary package hint: cloud-init

during lucid testing, on reboot of an instance, the instance did not come back up. the interesting part of the log is:

^M[ 0.868623] Write protecting the kernel text: 4332k
^M[ 0.868949] Write protecting the kernel read-only data: 1336k
^Minit: console-setup main process (63) terminated with status 1^M^M
%Ginit: plymouth-splash main process (229) terminated with status 2^M^M
init: plymouth main process (45) killed by SEGV signal^M^M
cloud-init running: Thu, 21 Oct 2010 02:32:28 +0000. up 67.27 seconds^M
waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id^M
  02:32:28 [ 1/100]: url error [[Errno 101] Network is unreachable]^M
  02:32:29 [ 2/100]: url error [[Errno 101] Network is unreachable]^M
  02:32:30 [ 3/100]: url error [[Errno 101] Network is unreachable]^M
  02:32:31 [ 4/100]: url error [[Errno 101] Network is unreachable]^M
  02:32:32 [ 5/100]: url error [[Errno 101] Network is unreachable]^M

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: cloud-init 0.5.10-0ubuntu1.3
ProcVersionSignature: User Name 2.6.32-309.18-ec2 2.6.32.21+drm33.7
Uname: Linux 2.6.32-309-ec2 i686
Architecture: i386
Date: Thu Oct 21 14:24:07 2010
Ec2AMI: ami-880b5bcd
Ec2AMIManifest: ubuntu-images-testing-us-west-1/ubuntu-lucid-daily-i386-server-20101020.manifest.xml
Ec2AvailabilityZone: us-west-1a
Ec2InstanceType: m1.small
Ec2Kernel: aki-f20b5bb7
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-init

[Scott Moser (smoser)]

I just saw this on a maverick instance of t1.micro:
us-east-1 ami-548c783d canonical ebs/ubuntu-maverick-10.10-amd64-server-20101007.1

[Scott Moser (smoser)]

in my maverick instance, a reboot didn't fix the issue. This was surprising to me, as I thought it must be race , some issue with networking not being all the way up. But a reboot should have given us another chance. Also, happening on a t1.micro seems to remove a lot of instance-based race possibility.

[Scott Moser (smoser)]

Just saw this again in lucid refresh testing for 20110611.
Note, this seems related, but is different error output from cloud init than bug 745930.

[Scott Moser (smoser)]

i'd not commented on this since maverick, and i'm not fixing this very rare occurance there.
I think its most likely a race.