investigate "full disk" images on EC2 for non-hvm instance types

[Scott Moser]

Hey,
  Currently in EC2, we register 3 types of images:
 * hvm
 * ebs
 * instance-store

 The hvm boot volumes have a bootloader installed and are "full disk images".
 The instance-store and ebs are partitionless disks.

The thought of doing this came into my mind when someone asked me how they could take their images out of EC2 and to openstack. Doing that would have 2 trickier pieces:
 * ebs or instance volume types would have no MBR and openstack images more commonly would have an MBR.
 * user woudl have to get the image out of openstack.

The benefits of doing it would be:
 * less differences between openstack and ec2
 * remove need to create (and publish) the "uec tarball" images.
 * potentially a happier grub-pc

some things that play in this area to make things work and might have to be changed/updated are:
 * grub-legacy-ec2
 * cloud-init (cloudinit/config/cc_grub_dpkg.py)
 * ec2-automated-builds and publishing scripts (we'd have to use krn/pv-grub-hd00_1.03-x86_64.gz rather than krn/pv-grub-hd0_1.03-x86_64.gz)
 * initramfs/kernel

I tried doing this with the following general steps:
 * start with a volume created from an hvm snapshot of 12.04
 * modify /boot/grub/menu.lst in the image to say '(hd0,0)' rather than (hd0) (otherwise it can't find the kernel)
 * snapshot it (SNAP-A)
 * register that snapshot and boot it

It seems there are some variables at play here, and i'm not sure i found the right values. when you register an instance, you can specify '--root-device-name' and also a block-device-mapping for the root device. I'd think that we want either 'sda' or 'xvda' for those (--block-device-mapping=/dev/sda=snap-7eb96d16 --root-device-name=sda).

[Scott Moser (smoser)]

OK. so i did successfully get a hvm snapshot to boot as a non-hvm ebs root.
 * get a volume of hvm snapshot
 * modify /boot/grub/menu.lst entries to:
    - root (hd0)
    + root (hd0,0)
 * snapshot volume
 * register instance
    euca-register --snapshot=snap-3f020d61 --name "smtest-hvm-to-ebs6" --kernel=aki-b4aa75dd --arch=x86_64 --root-device-name=/dev/xvda

$ xc2 describe-images aki-b4aa75dd ami-2fc5b446
IMAGE ami-2fc5b446 950047163771/smtest-hvm-to-ebs6 950047163771 available private x86_64 machine aki-b4aa75dd ebs paravirtual xen
BLOCKDEVICEMAPPING EBS /dev/xvda snap-3f020d61 8 true standard
IMAGE aki-b4aa75dd amazon/pv-grub-hd00_1.03-x86_64.gz amazon available public x86_64 kernel instance-store paravirtual xen

Inside the instance:
$ cat /proc/partitions
major minor #blocks name

 202 0 8388608 xvda
 202 1 8377897 xvda1
$ cat /proc/cmdline
root=LABEL=cloudimg-rootfs ro console=hvc0

$ grep "debconf" /var/log/cloud-init.log
2013-06-20 14:39:53,015 - cc_grub_dpkg.py[DEBUG]: setting grub debconf-set-selections with '/dev/sda','false'

I was then able to 'dpkg-reconfigure grub-pc' and select 'xvda' as the thing to install to, and aso dpkg-reconfigure grub-legacy-ec2 and accept my changes.

[Scott Moser (smoser)]

Another benefit of doing this is that we might at some point be able to even have the same snapshot used as hvm and ebs root, and the block device contents of the instance-store image to be bit for bit the same.

ie, "one image".

[Scott Moser (smoser)]

I'm attaching instructions that I just used to create a "full disk" image from disk1.img on cloud-images.ubuntu.com.
In the instructions 2 things to mention:
 a.) my 'dd' version failed to boot. I'm not sure why, if I had to guess I would have thought that cp would have failed. Its also possible I just mixed them up. I tried 'cp --sparse' just to ensure that zeros on were zeros.
 b.) When I created an 8GB root disk, causing growpart to take place, the resize took 10 seconds. bug 1179610 is relevant here, and that would be much shorter time in 12.10 and later. However, I'd suggest that if we're making 8G roots, we go ahead and resize the filesystem up before snapshotting. At least something to think about.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1192991/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Ben Howard (darkmuggle-deactivatedaccount)]

Scott,

With out having run your code, I think that there is an easier way....

instead of converting the disk to a raw first and then dd'ing it, why not use:
$ sudo modprobe nbd
$ sudo qemu-nbd -c /dev/nbd0 ${DISK_IMG}
$ sudo dd if=/dev/nbd0 of=/dev/xvdh

In my experience, this just works and it maintains the sparseness.

[Ben Howard (darkmuggle-deactivatedaccount)]

Also, regarding comment #5, I have created new images via the dd from a qemu-nbd backed device and have had it boot. The last time I did that was a few month ago.

[Scott Moser (smoser)]

interesting...
i just tried doing a instance-store disk this way.
$ DISK_IMG_URL="http://cloud-images.ubuntu.com/releases/precise/release/ubuntu-12.04-server-cloudimg-amd64-disk1.img"
$ DISK_IMG="precise-disk1.img"
$ DISK_IMG_RAW="${DISK_IMG%.img}.raw"

$ sudo apt-get update && sudo apt-get install --assume-yes qemu-utils bzr
$ bzr branch lp:~smoser/+junk/backdoor-image bdi
$ bzr branch lp:cloud-utils cloud-utils
$ export PATH=$PWD/cloud-utils/bin:$PATH
$ wget $DISK_IMG_URL -O $DISK_IMG
$ qemu-img convert -O raw $DISK_IMG $DISK_IMG_RAW
$ sudo ./bdi/mount-callback-umount $DISK_IMG_RAW -- \
   chroot _MOUNTPOINT_ sed -i.dist 's/(hd0)/(hd0,0)/g' /boot/grub/menu.lst

# grow it to be 10G for maxium instance store root size
$ truncate --size 10G $DISK_IMG_RAW
$ cloud-publish-image -vv --name="my-test-raw-07" \
   x86_64 "$DISK_IMG_RAW" smoser-us-east-1 \
   --kernel=aki-b4aa75dd --root-device-name=/dev/xvda

When I ran it as a m1.small I got the attached console output. Yesterday I'd run ebs root's with t1.micro only so this issue may be relevant to them also.

I had registered with '--root-device-name', but for m1.small's, the hypervisor puts together a partition table:
 /dev/xvda1 (root disk)
 /dev/xvda2 (instance-store)
 /dev/xvda3 (swap)

there *is* no /dev/xvda device.
so, in this case, because the hypervisor attached the full disk image at /dev/xvda1, so the root filesystem would then have been found at /dev/xvda1p1.

bah.

I also saw this ith m1.large.

So, it would seem that more investigation is needed. this might not be possible.