Too much image copying when using vmbuilder for EC2

[Eric Hammond]

I'm using the latest lp:vmbuilder source to build EBS boot AMIs for EC2. I noticed that there is an awful lot of image file copying being done which seems unnecessary when using vmbuilder for this purpose.

1. vmbuilder installs the image files into a temporary "root" subdirectory
2. The "root" subdirectory is copied with rsync into the "target" directory, which is a loopback filesystem on a huge "root.img" temporary file.
3. The big "root.img" file is copied with "cp" from the temporary directory to a destination directory.
4. I then have to mount this image file and copy the files a fourth time onto the EBS volume in order to create an EBS boot AMI.

Ideally, three of these steps could be eliminated so that the image files are installed directly onto the EBS volume, with the path provided through an option.

Though this does not affect the functionality in a major way, the current four copies take up a good chunk of time, IO, and disk space when building AMIs on EC2.

Parts of the vmbuilder log file for reference:

2010-01-03 00:34:01,285 DEBUG : ['/usr/bin/kvm-img', 'create', '-f', 'raw', '/mnt/vmbuilderw8mCwo/root.img', '10240M']
2010-01-03 00:34:09,061 DEBUG : ['mount', '-o', 'loop', '/mnt/vmbuilderw8mCwo/root.img', '/mnt/vmbuilderw8mCwo/target/']
[...]
2010-01-03 00:40:13,684 DEBUG : ['rsync', '-aHA', '/mnt/vmbuilderw8mCwo/root/', '/mnt/vmbuilderw8mCwo/target']
[...]
2010-01-03 00:40:32,288 DEBUG : ['cp', '--sparse=always', '/mnt/vmbuilderw8mCwo/root.img', '/mnt/dest-karmic-20100103-0033/root.img']

ProblemType: Bug
Architecture: i386
Date: Sun Jan 3 00:47:11 2010
DistroRelease: Ubuntu 9.10
Ec2AMI: ami-1515f67c
Ec2AMIManifest: ubuntu-images-us/ubuntu-karmic-9.10-i386-server-20091027.1.manifest.xml
Ec2AvailabilityZone: us-east-1a
Ec2InstanceType: c1.medium
Ec2Kernel: aki-5f15f636
Ec2Ramdisk: ari-0915f660
Package: ubuntu-vm-builder (not installed)
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: User Name 2.6.31-302.7-ec2
SourcePackage: vm-builder
Tags: ec2-images
Uname: Linux 2.6.31-302-ec2 i686

[Chuck Short (zulcss)]

Thanks for the bug report.

Regards
chuck

[Eric Hammond (esh)]

For reference, this article enumerates the steps I took to run vmbuilder:

  http://alestic.com/2010/01/vmbuilder-ebs-boot-ami

[Scott Moser (smoser)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

 subscribe
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAktCEPgACgkQDZzbdwr3wEdgbQCfUBBMS0H2mSMxxTstKjGDRk8x
6vYAn0Qc0OcO3oW5K3NUybKiypVE444N
=uvKY
-----END PGP SIGNATURE-----