AMI created from latest i386 Lucid image fails to boot

[Chetan Sarva]

I bundled an image based on ami-809a48e9 (latest i386 lucid image) and it will not boot. It fails with the following error in the console:

http://pastebin.com/raw.php?i=2m16AfFs

I got the same error with the AMI 2-3 releases before this one as well. AMIs from the middle of last year did not have the same problem.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: udisks (not installed)
ProcVersionSignature: User Name 2.6.32-342.43-ec2 2.6.32.52+drm33.21
Uname: Linux 2.6.32-342-ec2 i686
Architecture: i386
Date: Wed Feb 22 21:54:59 2012
Ec2AMI: ami-809a48e9
Ec2AMIManifest: ubuntu-us-east-1/images/ubuntu-lucid-10.04-i386-server-20120221.manifest.xml
Ec2AvailabilityZone: us-east-1c
Ec2InstanceType: c1.medium
Ec2Kernel: aki-407d9529
Ec2Ramdisk: unavailable
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: udisks
Symptom: storage
Title: Internal hard disk partition cannot be mounted manually

[Eric Hammond (esh)]

This is the key error:

    ALERT! /dev/disk/by-label/cloudimg-rootfs does not exist.

[Eric Hammond (esh)]

Chetan: Can you provide the commands you are using to bundle the AMI? ec2-bundle-vol or ec2-bundle-image?

[Eric Hammond (esh)]

Here are a couple posts that reference the issue. It looks like smoser provided a fixed ec2-bundle-vol:

  https://forums.aws.amazon.com/thread.jspa?threadID=58232

  https://lists.ubuntu.com/archives/ubuntu-cloud/2011-January/000477.html

It isn't clear if the patch has made it into the official Amazon command line tools.

[Chetan Sarva (csarva)]

Eric, it looks like you're right -- the version of the EC2 tools contained in multiverse is over 4 years old (1.3-45758 20071010). I manually downloaded the latest version from Amazon and re-bundled and that did the trick.

I see that the newer versions are in Precise -- any reason these can't also be provided for Lucid? It's pretty badly broken as is.

[Scott Moser (smoser)]

Chetan,
  This is bug 667793 , which should be fixed in lucid updates. You should be able to get a good version from multiverse in lucid-updates, or in backports. You'll need to make sure you apt-get update.

  Please collect:
apt-cache policy ec2-ami-tools

 Additionally, there is a ppa at https://launchpad.net/~awstools-dev/+archive/awstools that has up to date versions of those tools that should also work.

[Scott Moser (smoser)]

I'm moving this back to ec2-ami-tools.
If this bug is still present, its fix is in that package. I do not think it is still present.

The one thing i'm not sure of is if the fix was ever applied to backports. If not, we should probably just put a newer version (like one from precise) back to backports.

[Chetan Sarva (csarva)]

Scott, I just tested the version in lucid-backports and it doesn't work either. The version in the PPA works as expected.

[Scott Moser (smoser)]

Chetan,
 I suspect you have not done an apt-get update before installing the ec2-ami-tools that you think are broken. I've done the following and verified that an instance will boot correctly after being re-bundled with lucid-updates versions of ec2-ami-tools:

## run an instance of ami-809a48e9
## us-east-1 ami-809a48e9 canonical ubuntu-lucid-10.04-i386-server-20120221
$ ec2-run-instances --region us-east-1 --instance-type m1.small ami-809a48e9
$ rhost=ec2-107-22-87-33.compute-1.amazonaws.com
$ ssh rhost

## now, assuming all variables are set up for use of ec2-api and ec2-ami
## tools

## add multiverse
$ sudo sed -i 's,main universe$,main universe multiverse,g' \
   /etc/apt/sources.list
$ sudo apt-get update -y
$ sudo apt-get install ec2-api-tools ec2-ami-tools -y
$ dpkg-query --show ec2-api-tools ec2-ami-tools
ec2-ami-tools 1.3-45758-0ubuntu1.1
ec2-api-tools 1.3.46266-0ubuntu1

## assuming all variables set up for use of this
$ BUNDLE_D=/mnt/my.bundle
$ sudo rm -Rf $BUNDLE_D
$ mkdir -p $BUNDLE_D
$ sudo sh -c modprobe loop || :
## may take quite a long time (10s of minutes)
$ sudo -E ec2-bundle-vol --arch i386 --destination $BUNDLE_D --prefix testbundle.img --user 950047163771 --size 10240 --exclude /mnt,/root/.ssh --cert=$EC2_CERT --privatekey=$EC2_PRIVATE_KEY
$ sudo chown -R 1000:1000 /mnt/wd
# get current availability-zone, security-group, instance-type, keypair
# from metadata service using ec2metadata
$ ec2-upload-bundle --access-key=$EC2_ACCESS_KEY --secret-key=$EC2_SECRET_KEY --bucket smoser-amitools-us-east-1 --manifest $BUNDLE_D/testbundle.img.manifest.xml
$ ec2-register --region us-east-1 --cert=$EC2_CERT --private-key=$EC2_PRIVATE_KEY smoser-amitools-us-east-1/testbundle.img.manifest.xml

AMI=ami-abcdefg
## run instance of $AMI
$ ec2-run-instances --region us-east-1 --cert=$EC2_CERT --private-key=$EC2_PRIVATE_KEY --key default $AMI --group default --instance-type m1.small

The instance came up fine, and I verified ssh to it.

[Chetan Sarva (csarva)]

Scott,

I just cleaned everything up and tried again using your steps above (for installing the multiverse package) and everything seems to be working now.

I'm quite certain I did previously first remove the installed ec2 tools and then apt-get update and reinstall, so I'm not sure what went wrong. The only difference I can see is that I had added multiverse to only one of the repos (backports or updates, but not both) whereas you added it to all of them.

chetan