vmbuilder doesn't recognize --ec2 flag

[Miguel Landaeta]

Description: Ubuntu 10.04 LTS
Release: 10.04

python-vm-builder-ec2:
  Installed: 0.12.3-0ubuntu1
  Candidate: 0.12.3-0ubuntu1
  Version table:
 *** 0.12.3-0ubuntu1 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ lucid/universe Packages
        100 /var/lib/dpkg/status

Hi,

I can't generate EC2 images with vmbuilder. I'm using a command like this:

sudo vmbuilder xen ubuntu --suite=lucid --ec2 \
--ec2-cert=.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem \
--ec2-key=.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem \
--ec2-access-key=WHATEVER \
--ec2-secret-key=WHATEVER \
--ec2-user=424242424242 --ec2-bucket=anyrandombucket --ec2-prefix=test \
--ec2-version='first' \
--firstboot=/usr/share/doc/python-vm-builder-ec2/examples/ec2-firstboot.sh \
--part=/usr/share/doc/python-vm-builder-ec2/examples/ec2-i386-part-file.txt \
--debug

Actually, I was following this howto: https://wiki.ubuntu.com/EC2Vmbuilder

When I run vmbuilder with those flags, I just get this message:

Usage: vmbuilder hypervisor distro [options]

vmbuilder: error: no such option: --ec2

So, I can not build images for Amazon EC2.

I inspected /usr/share/pyshared/VMBuilder/plugins/ec2/__init__.py and I found
at the last line of that file this:

#register_plugin(EC2)

I uncommented that line and it didn't work.
Also I tried using register_hypervisor_plugin(EC2), but it didn't work.

If I'm missing something obvious here, just let me know.

In fact, what I'm trying to do is just build an EC2 image like the ones described in
http://uec-images.ubuntu.com/releases/lucid/release/

It would be nice to know how were generated the Ubuntu official images available
in Amazon EC2.

Regards,

--
Miguel Landaeta
miguel at miguel.cc

[Chuck Short (zulcss)]

Thank you for taking the time to report this issue and helping to make Ubuntu better. Examining the information you have given us, this does not appear to be a bug report so we are closing it and converting it to a question in the support tracker. We appreciate the difficulties you are facing, but it would make more sense to raise problems you are having in the support tracker at https://answers.launchpad.net/ubuntu if you are uncertain if they are bugs. For help on reporting bugs, see https://help.ubuntu.com/community/ReportingBugs#When%20not%20to%20file%20a%20bug.

[Miguel Landaeta (nomadium)]

IMHO, this is a valid bug because the software doesn't work as expected. It just doesn't accept --ec2 flag.
In fact, I have to downgrade to 0.11 (version shipped with karmic) to get this software working.

If this is a duplicated report please point to the valid one, if this is not a duplicated report please reconsider your decision.

[Seren (r-launchpad-seren-net)]

We too are having this problem. This bug report was phrased like a question, but it does seems like a bug since the --ec2 flag doesn't work.

[Gerry Reno (greno-verizon)]

Same here. Encountering the exact same problem.

vmbuilder: error: no such option: --ec2

[C de-Avillez (hggdh2)]

I am converting this back to a bug -- vmbuilder (...) --ec2 is indeed failing.

[C de-Avillez (hggdh2)]

Marking Confirmed/High.

[David Konerding (dakoner)]

I took a look at the source code in question. This package appears to be out of date wrt the underlying package it plugs into (vmbuilder). There are several API differences (changed method names, changed attributes). It's not limited to missing --ec2 option; even with minor hacking to correct the API, it still does not work. It's likely there is an upstream version of this package that needs to be integrated.

[austin (bang-a-rang)]

Been up all night trying to figure this out. OP: Thanks for posting a bug!

vmbuilder not accepting --ec2 options in lucid 10.04

[Gustavo (gbaratto)]

Hey guys...

Any update on this bug? It's set to high, but it still unassigned.
I'm certain a lot of people out there trying to make EC2 images are stuck :)

Is there any workaround for this?

Thanks all
g.

[Neil Wilson (neil-aldur)]

Series 12 vmbuilder just doesn't support seem to support ec2 building yet. If you need that facility then you need to use the 0.11 branch.

I'm sure they are trying to tidy it up as it all seems a bit messy at the moment. Really it needs refactoring so that the EC2 stuff is a decorator pattern around the standard distro classes. Then it can be adapted easily for different clouds.

As usual what we really need is a clone of Soren...

[Serge Hallyn (serge-hallyn)]

The attached updated EC2 class works for me. Can you please test it? If it works for you, I'll update the vmbuilder package and pursue and SRU for all previous releases.

To test, please replace your VMBuilder/plugins/ec2/__init__.py file with the attached one. I.e.:

dest=`find /usr/lib -maxdepth 3 -name VMBuilder | tail -1`
cp $dest/plugins/ec2/__init__.py $dest/plugins/ec2/init.py.orig
cp newinit.py $dest/plugins/ec2/__init__.py
chmod ugo+x $dest/plugins/ec2/__init__.py

[Henrik Bach (bach-henrik)]

Hi Serge.

Sorry for the late reply. I'll do some testing asap. I'll return back
to you within a week.

-Henrik

On Sun, Apr 3, 2011 at 3:22 AM, Serge Hallyn <email address hidden> wrote:
> The attached updated EC2 class works for me.  Can you please test it?
> If it works for you, I'll update the vmbuilder package and pursue and
> SRU for all previous releases.
>
> To test, please replace your VMBuilder/plugins/ec2/__init__.py file with
> the attached one.  I.e.:
>
> dest=`find /usr/lib -maxdepth 3 -name VMBuilder | tail -1`
> cp $dest/plugins/ec2/__init__.py $dest/plugins/ec2/init.py.orig
> cp newinit.py $dest/plugins/ec2/__init__.py
> chmod ugo+x $dest/plugins/ec2/__init__.py
>
> ** Attachment added: "new ec2 init.py"
>   https://bugs.launchpad.net/ubuntu/+source/vm-builder/+bug/576501/+attachment/1971044/+files/__init__.py
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/576501
>
> Title:
>  vmbuilder doesn't recognize --ec2 flag
>
> Status in Virtual Machine Builder:
>  New
> Status in “vm-builder” package in Ubuntu:
>  In Progress
>
> Bug description:
>  Description:    Ubuntu 10.04 LTS
>  Release:        10.04
>
>  python-vm-builder-ec2:
>    Installed: 0.12.3-0ubuntu1
>    Candidate: 0.12.3-0ubuntu1
>    Version table:
>   *** 0.12.3-0ubuntu1 0
>          500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ lucid/universe Packages
>          100 /var/lib/dpkg/status
>
>  Hi,
>
>  I can't generate EC2 images with vmbuilder. I'm using a command like
>  this:
>
>  sudo vmbuilder xen ubuntu --suite=lucid --ec2 \
>  --ec2-cert=.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem \
>  --ec2-key=.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem \
>  --ec2-access-key=WHATEVER \
>  --ec2-secret-key=WHATEVER \
>  --ec2-user=424242424242 --ec2-bucket=anyrandombucket --ec2-prefix=test \
>  --ec2-version='first' \
>  --firstboot=/usr/share/doc/python-vm-builder-ec2/examples/ec2-firstboot.sh \
>  --part=/usr/share/doc/python-vm-builder-ec2/examples/ec2-i386-part-file.txt \
>  --debug
>
>  Actually, I was following this howto:
>  https://wiki.ubuntu.com/EC2Vmbuilder
>
>  When I run vmbuilder with those flags, I just get this message:
>
>  Usage: vmbuilder hypervisor distro [options]
>
>  vmbuilder: error: no such option: --ec2
>
>  So, I can not build images for Amazon EC2.
>
>  I inspected /usr/share/pyshared/VMBuilder/plugins/ec2/__init__.py and I found
>  at the last line of that file this:
>
>  #register_plugin(EC2)
>
>  I uncommented that line and it didn't work.
>  Also I tried using register_hypervisor_plugin(EC2), but it didn't work

[Henrik Bach (bach-henrik)]

Hi Serge!

I installed the attached file as per instructions given.

I issued the command:

~$ sudo vmbuilder xen ubuntu --suite=lucid --ec2 --ec2-cert=$cert
--ec2-key=$pk --ec2-access-key=$AWS_ACCESS_KEY_ID
--ec2-secret-key=$AWS_SECRET_ACCESS_KEY --ec2-user=$AWS_USER_ID
2011-04-16 04:19:35,713 INFO : Calling hook: preflight_check
2011-04-16 04:19:35,750 INFO : Calling hook: set_defaults
2011-04-16 04:19:35,751 INFO : Calling hook: bootstrap
2011-04-16 04:30:56,807 INFO : Calling hook: configure_os
2011-04-16 04:31:27,595 INFO :
2011-04-16 04:31:27,596 INFO : Current default time zone: 'Etc/UTC'
2011-04-16 04:31:27,782 INFO : Local time is now: Sat Apr 16
02:31:27 UTC 2011.
2011-04-16 04:31:27,783 INFO : Universal Time is now: Sat Apr 16
02:31:27 UTC 2011.
2011-04-16 04:31:27,783 INFO :
Extracting templates from packages: 100%
2011-04-16 04:32:46,506 INFO :
2011-04-16 04:32:46,506 INFO : Current default time zone: 'Etc/UTC'
2011-04-16 04:32:46,544 INFO : Local time is now: Sat Apr 16
02:32:46 UTC 2011.
2011-04-16 04:32:46,544 INFO : Universal Time is now: Sat Apr 16
02:32:46 UTC 2011.
2011-04-16 04:32:46,545 INFO : Run 'dpkg-reconfigure tzdata' if you
wish to change it.
2011-04-16 04:32:46,546 INFO :
2011-04-16 04:33:50,466 INFO : start: Unknown parameter: JOB
2011-04-16 04:33:57,543 INFO : Calling hook: post_install
2011-04-16 04:33:57,552 INFO : Calling hook: preflight_check
2011-04-16 04:33:57,565 INFO : Calling hook: configure_networking
2011-04-16 04:33:57,602 INFO : Calling hook: configure_mounting
2011-04-16 04:33:57,665 INFO : Calling hook: mount_partitions
2011-04-16 04:33:57,672 INFO : Mounting target filesystems
2011-04-16 04:33:57,674 INFO : Creating filesystem: /, size: 4096,
dummy: False
2011-04-16 04:33:57,675 INFO : Not preallocated, so we create it.
2011-04-16 04:33:58,175 INFO : mke2fs 1.41.12 (17-May-2010)
2011-04-16 04:33:58,176 INFO : warning: Unable to get device
geometry for /tmp/tmp1Mnqgd
2011-04-16 04:34:03,521 INFO : mke2fs 1.41.12 (17-May-2010)
2011-04-16 04:34:03,523 INFO : warning: Unable to get device
geometry for /tmp/tmp1Mnqgd
2011-04-16 04:34:08,443 INFO : Creating filesystem: None, size:
1024, dummy: False
2011-04-16 04:34:08,443 INFO : Not preallocated, so we create it.
2011-04-16 04:35:11,876 INFO : Calling hook: install_kernel
2011-04-16 04:35:44,922 INFO : Done.
2011-04-16 04:35:59,134 INFO : Running depmod.
2011-04-16 04:35:59,636 INFO : update-initramfs: Generating
/boot/initrd.img-2.6.32-30-generic-pae
2011-04-16 04:36:12,379 INFO :
2011-04-16 04:36:12,380 INFO : Creating config file
/etc/default/grub with new version
2011-04-16 04:36:12,970 INFO : /proc/devices: fopen failed: No such
file or directory
2011-04-16 04:36:12,971 INFO : Is device-mapper driver missing from kernel?
2011-04-16 04:36:12,972 INFO : Failure to communicate with kernel
device-mapper driver.
2011-04-16 04:36:15,029 INFO : Calling hook: unmount_partitions
2011-04-16 04:36:15,031 INFO : Unmounting target filesystem
2011-04-16 04:36:16,880 INFO : Calling hook: convert
2011-04-16 04:36:16,882 INFO ...

[Serge Hallyn (serge-hallyn)]

Thanks for testing, Henrik. But is there any way you could test
on a natty system? Your failure looks to be due to a different
bug, and my test just now in a maverick container also failed,
differently from yours. But on two different natty systems it
has succeeded (using your exact commandline, and also using
--ec2-bundle). I'd like to get the fix upstream first, and then
figure out what needs to be done to backport it.

In fact, a simple
vmbuilder kvm ubuntu --suite=lucid \
--mirror=http://10.42.43.65:3142/us.archive.ubuntu.com/ubuntu/ --debug

failed for me. So that is unrelated to --ec2

-serge

[Henrik Bach (bach-henrik)]

Hi Serge!

I acknowledge your point of view and more thorough insight into
vmbuilder than me. What can I do to help you?

-Henrik

On Mon, Apr 18, 2011 at 7:19 PM, Serge Hallyn <email address hidden> wrote:
> Thanks for testing, Henrik.  But is there any way you could test
> on a natty system?  Your failure looks to be due to a different
> bug, and my test just now in a maverick container also failed,
> differently from yours.  But on two different natty systems it
> has succeeded (using your exact commandline, and also using
> --ec2-bundle).  I'd like to get the fix upstream first, and then
> figure out what needs to be done to backport it.
>
> In fact, a simple
> vmbuilder kvm ubuntu --suite=lucid  \
> --mirror=http://10.42.43.65:3142/us.archive.ubuntu.com/ubuntu/ --debug
>
> failed for me.  So that is unrelated to --ec2
>
> -serge
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/576501
>
> Title:
>  vmbuilder doesn't recognize --ec2 flag
>
> Status in Virtual Machine Builder:
>  New
> Status in “vm-builder” package in Ubuntu:
>  In Progress
>
> Bug description:
>  Description:    Ubuntu 10.04 LTS
>  Release:        10.04
>
>  python-vm-builder-ec2:
>    Installed: 0.12.3-0ubuntu1
>    Candidate: 0.12.3-0ubuntu1
>    Version table:
>   *** 0.12.3-0ubuntu1 0
>          500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ lucid/universe Packages
>          100 /var/lib/dpkg/status
>
>  Hi,
>
>  I can't generate EC2 images with vmbuilder. I'm using a command like
>  this:
>
>  sudo vmbuilder xen ubuntu --suite=lucid --ec2 \
>  --ec2-cert=.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem \
>  --ec2-key=.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem \
>  --ec2-access-key=WHATEVER \
>  --ec2-secret-key=WHATEVER \
>  --ec2-user=424242424242 --ec2-bucket=anyrandombucket --ec2-prefix=test \
>  --ec2-version='first' \
>  --firstboot=/usr/share/doc/python-vm-builder-ec2/examples/ec2-firstboot.sh \
>  --part=/usr/share/doc/python-vm-builder-ec2/examples/ec2-i386-part-file.txt \
>  --debug
>
>  Actually, I was following this howto:
>  https://wiki.ubuntu.com/EC2Vmbuilder
>
>  When I run vmbuilder with those flags, I just get this message:
>
>  Usage: vmbuilder hypervisor distro [options]
>
>  vmbuilder: error: no such option: --ec2
>
>  So, I can not build images for Amazon EC2.
>
>  I inspected /usr/share/pyshared/VMBuilder/plugins/ec2/__init__.py and I found
>  at the last line of that file this:
>
>  #register_plugin(EC2)
>
>  I uncommented that line and it didn't work.
>  Also I tried using register_hypervisor_plugin(EC2), but it didn't work

[Serge Hallyn (serge-hallyn)]

Quoting Henrik Bach (<email address hidden>):
> 2011-04-16 04:36:12,380 INFO : Creating config file
> /etc/default/grub with new version
> 2011-04-16 04:36:12,970 INFO : /proc/devices: fopen failed: No such
> file or directory
> 2011-04-16 04:36:12,971 INFO : Is device-mapper driver missing from kernel?

Hm. Actually all the problems I've had I've traced to environments (bad
network setup for one, device permissions for another). I finally got a
maverick xen host in a working environment - and vmbuilder -ec2 worked
there.

What does /proc/devices on your host show? What sort of storage setup
do you have? The 'Is device-mapper driver missing from kernel?' is
odd, and presumably comes from grub in the chroot. It almost sounds
like your host is multipathed. What does 'df -h' and 'mount' on the
host show?

Does a regular 'vmbuilder xen ubuntu' work for you?

[Henrik Bach (bach-henrik)]

On Mon, Apr 18, 2011 at 10:32 PM, Serge Hallyn
<email address hidden> wrote:
> Quoting Henrik Bach (<email address hidden>):
>> 2011-04-16 04:36:12,380 INFO    : Creating config file
>> /etc/default/grub with new version
>> 2011-04-16 04:36:12,970 INFO    : /proc/devices: fopen failed: No such
>> file or directory
>> 2011-04-16 04:36:12,971 INFO    : Is device-mapper driver missing from kernel?
>
> Hm.  Actually all the problems I've had I've traced to environments (bad
> network setup for one, device permissions for another).  I finally got a
> maverick xen host in a working environment - and vmbuilder -ec2 worked
> there.

Ok.

>
> What does /proc/devices on your host show?

$ cat /proc/devices
Character devices:
  1 mem
  4 /dev/vc/0
  4 tty
  4 ttyS
  5 /dev/tty
  5 /dev/console
  5 /dev/ptmx
  6 lp
  7 vcs
 10 misc
 13 input
 21 sg
 29 fb
 99 ppdev
108 ppp
116 alsa
128 ptm
136 pts
180 usb
189 usb_device
226 drm
252 usbmon
253 bsg
254 rtc

Block devices:
  1 ramdisk
  2 fd
259 blkext
  7 loop
  8 sd
  9 md
 11 sr
 65 sd
 66 sd
 67 sd
 68 sd
 69 sd
 70 sd
 71 sd
128 sd
129 sd
130 sd
131 sd
132 sd
133 sd
134 sd
135 sd
252 device-mapper
253 pktcdvd
254 mdp

> What sort of storage setup
> do you have?

I'm building the image on an ordinary harddisk with my home folder encrypted.

> The 'Is device-mapper driver missing from kernel?' is
> odd, and presumably comes from grub in the chroot.  It almost sounds
> like your host is multipathed.
> What does 'df -h' and 'mount' on the host show?

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 36G 7,5G 26G 23% /
none 491M 228K 491M 1% /dev
none 497M 1,4M 496M 1% /dev/shm
none 497M 140K 497M 1% /var/run
none 497M 0 497M 0% /var/lock
none 36G 7,5G 26G 23% /var/lib/ureadahead/debugfs
$ mount
/dev/sda1 on / type ext4 (rw,errors=remount-ro,commit=0)
proc on /proc type proc (rw,noexec,nosuid,nodev)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
none on /dev type devtmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /var/lib/ureadahead/debugfs type debugfs (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc
(rw,noexec,nosuid,nodev)
gvfs-fuse-daemon on /home/sad/.gvfs type fuse.gvfs-fuse-daemon
(rw,nosuid,nodev,user=X)

>
> Does a regular 'vmbuilder xen ubuntu' work for you?
>

First try was:

$ sudo vmbuilder xen ubuntu
[sudo] password for X:
2011-05-03 19:50:53,587 INFO : Calling hook: preflight_check
2011-05-03 19:50:53,641 INFO : Calling hook: set_defaults
2011-05-03 19:50:53,647 INFO : Calling hook: bootstrap
2011-05-03 20:02:47,612 INFO : Calling hook: configure_os
2011-05-03 20:03:23,304 INFO :
2011-05-03 20:03:23,305 INFO : Current default time zone: 'Etc/UTC'
2011-05-03 20:...