vmbuilder randomly fails to install grub

[David Clarke]

Command line and options are identical between runs, except for the hostname and LV (as passed to --raw)

# vmbuilder kvm ubuntu --hostname=$1 --raw=/dev/vg0/$1 -c /etc/vmbuilder.cfg --debug -o --rootsize=8192 --swapsize=2048 --mem=512

# egrep -v "^$|^#" /etc/vmbuilder.cfg
[DEFAULT]
[ubuntu]
mirror = http://ubuntu.catalyst.net.nz/ubuntu
security-mirror = http://ubuntu.catalyst.net.nz/ubuntu
suite = lucid
flavour = virtual
[kvm]
libvirt = qemu:///system
bridge = br0

Successful run excerpt:

2010-07-28 11:25:00,179 DEBUG : ['rsync', '-a', '/tmp/tmpXcZ1l2/usr/lib/grub/x86_64-pc/', '/tmp/tmpXcZ1l2/boot/grub/']
2010-07-28 11:25:00,233 DEBUG : ['chroot', '/tmp/tmpXcZ1l2', 'grub', '--device-map=/tmp/vmbuilder-grub/device.map', '--batch']
2010-07-28 11:25:00,234 DEBUG : stdin was set and it was a string: root (hd0,0)
setup (hd0)
EOT
2010-07-28 11:25:01,239 DEBUG :
2010-07-28 11:25:01,239 DEBUG : [ Minimal BASH-like line editing is supported. For
2010-07-28 11:25:01,240 DEBUG : the first word, TAB lists possible command
2010-07-28 11:25:01,240 DEBUG : completions. Anywhere else TAB lists the possible
2010-07-28 11:25:01,240 DEBUG : completions of a device/filename. ]
2010-07-28 11:25:01,241 DEBUG : grub> root (hd0,0)
2010-07-28 11:25:01,241 DEBUG : grub> setup (hd0)
2010-07-28 11:25:01,241 DEBUG : Checking if "/boot/grub/stage1" exists... yes
2010-07-28 11:25:01,241 DEBUG : Checking if "/boot/grub/stage2" exists... yes
2010-07-28 11:25:01,242 DEBUG : Checking if "/boot/grub/e2fs_stage1_5" exists... yes
2010-07-28 11:25:01,242 DEBUG : Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... failed (this is not fatal)
2010-07-28 11:25:01,242 DEBUG : Running "embed /boot/grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal)
2010-07-28 11:25:01,242 DEBUG : Running "install /boot/grub/stage1 (hd0) /boot/grub/stage2 p /boot/grub/menu.lst "... succeeded
2010-07-28 11:25:01,243 DEBUG : Done.
2010-07-28 11:25:01,243 DEBUG : grub> EOT
2010-07-28 11:25:01,244 DEBUG : ['chroot', '/tmp/tmpXcZ1l2', '/usr/sbin/update-grub', '-y']
2010-07-28 11:25:01,381 INFO : Searching for GRUB installation directory ... found: /boot/grub
2010-07-28 11:25:01,447 INFO : Searching for default file ... Generating /boot/grub/default file and setting the default boot entry to 0
2010-07-28 11:25:01,449 INFO : Searching for GRUB installation directory ... found: /boot/grub
2010-07-28 11:25:01,460 INFO : Testing for an existing GRUB menu.lst file ...
2010-07-28 11:25:01,460 INFO :
2010-07-28 11:25:01,460 INFO : Could not find /boot/grub/menu.lst file.
2010-07-28 11:25:01,460 INFO : Generating /boot/grub/menu.lst
2010-07-28 11:25:01,582 INFO : Searching for splash image ... none found, skipping ...
2010-07-28 11:25:01,825 INFO : grep: /boot/config*: No such file or directory
2010-07-28 11:25:01,966 INFO : Updating /boot/grub/menu.lst ... done
2010-07-28 11:25:01,966 INFO :

Failed run:

2010-07-28 11:28:16,816 DEBUG : ['rsync', '-a', '/tmp/tmpa5EOR4/usr/lib/grub/x86_64-pc/', '/tmp/tmpa5EOR4/boot/grub/']
2010-07-28 11:28:16,870 DEBUG : ['chroot', '/tmp/tmpa5EOR4', 'grub', '--device-map=/tmp/vmbuilder-grub/device.map', '--batch']
2010-07-28 11:28:16,870 DEBUG : stdin was set and it was a string: root (hd0,0)
setup (hd0)
EOT
2010-07-28 11:28:17,526 DEBUG :
2010-07-28 11:28:17,527 DEBUG : [ Minimal BASH-like line editing is supported. For
2010-07-28 11:28:17,527 DEBUG : the first word, TAB lists possible command
2010-07-28 11:28:17,527 DEBUG : completions. Anywhere else TAB lists the possible
2010-07-28 11:28:17,528 DEBUG : completions of a device/filename. ]
2010-07-28 11:28:17,528 DEBUG : grub> root (hd0,0)
2010-07-28 11:28:17,528 DEBUG : grub> setup (hd0)
2010-07-28 11:28:17,528 DEBUG : Checking if "/boot/grub/stage1" exists... no
2010-07-28 11:28:17,529 DEBUG : Checking if "/grub/stage1" exists... no
2010-07-28 11:28:17,529 DEBUG :
2010-07-28 11:28:17,529 DEBUG : Error 15: File not found
2010-07-28 11:28:17,530 DEBUG : grub> EOT
2010-07-28 11:28:17,530 DEBUG : ['chroot', '/tmp/tmpa5EOR4', '/usr/sbin/update-grub', '-y']
2010-07-28 11:28:17,664 INFO : Searching for GRUB installation directory ... found: /boot/grub
2010-07-28 11:28:17,728 INFO : Searching for default file ... Generating /boot/grub/default file and setting the default boot entry to 0
2010-07-28 11:28:17,731 INFO : Searching for GRUB installation directory ... found: /boot/grub
2010-07-28 11:28:17,741 INFO : Testing for an existing GRUB menu.lst file ...
2010-07-28 11:28:17,742 INFO :
2010-07-28 11:28:17,742 INFO : Could not find /boot/grub/menu.lst file.
2010-07-28 11:28:17,742 INFO : Generating /boot/grub/menu.lst
2010-07-28 11:28:17,863 INFO : Searching for splash image ... none found, skipping ...
2010-07-28 11:28:18,112 INFO : grep: /boot/config*: No such file or directory
2010-07-28 11:28:18,255 INFO : Updating /boot/grub/menu.lst ... done
2010-07-28 11:28:18,255 INFO :

Have tried different mirrors, {python,ubuntu}-vm-builder from the daily ppa, not using tmpfs.

It works approximately half the time:

http://paste.ubuntu.com/469994/

[David Clarke (a-launchpad-uber-geek-nz)]

Seems to be related to the size and/or position of the LV on disk or partitioning setup inside the LV. 1TB drives (software RAID 1) with LBA enabled.

4 GB LV, 2 GB root, 2GB swap:

test01.txt:2010-07-28 13:39:00,848 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test02.txt:2010-07-28 13:40:32,217 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test03.txt:2010-07-28 13:42:08,167 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test04.txt:2010-07-28 13:43:46,199 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test05.txt:2010-07-28 13:45:20,067 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test06.txt:2010-07-28 13:46:53,004 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test07.txt:2010-07-28 13:48:25,589 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test08.txt:2010-07-28 13:49:56,128 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test09.txt:2010-07-28 13:51:27,426 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test10.txt:2010-07-28 13:53:00,229 DEBUG : Checking if "/boot/grub/stage1" exists... yes

40 GB LV, 32 GB root, 8 GB swap:

test11.txt:2010-07-28 13:55:45,188 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test12.txt:2010-07-28 13:57:23,889 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test13.txt:2010-07-28 13:59:04,599 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test14.txt:2010-07-28 14:00:43,285 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test15.txt:2010-07-28 14:02:25,818 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test16.txt:2010-07-28 14:04:04,428 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test17.txt:2010-07-28 14:05:46,409 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test18.txt:2010-07-28 14:07:26,049 DEBUG : Checking if "/boot/grub/stage1" exists... no
test19.txt:2010-07-28 14:09:08,828 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test20.txt:2010-07-28 14:10:48,468 DEBUG : Checking if "/boot/grub/stage1" exists... yes

10 GB LV, 8 GB root, 2 GB swap:

test21.txt:2010-07-28 14:14:40,718 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test22.txt:2010-07-28 14:16:20,228 DEBUG : Checking if "/boot/grub/stage1" exists... no
test23.txt:2010-07-28 14:17:59,287 DEBUG : Checking if "/boot/grub/stage1" exists... no
test24.txt:2010-07-28 14:19:37,439 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test25.txt:2010-07-28 14:21:12,338 DEBUG : Checking if "/boot/grub/stage1" exists... no
test26.txt:2010-07-28 14:22:51,709 DEBUG : Checking if "/boot/grub/stage1" exists... no
test27.txt:2010-07-28 14:24:26,248 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test28.txt:2010-07-28 14:26:02,670 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test29.txt:2010-07-28 14:27:36,028 DEBUG : Checking if "/boot/grub/stage1" exists... yes
test30.txt:2010-07-28 14:29:12,604 DEBUG : Checking if "/boot/grub/stage1" exists... yes

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug and helping to make Ubuntu better. I'm a little
confused about your last comment: you first said that you were able to
reproduce roughly 50% of the time with identical conditions. But your last
comment says it *does* seem related to size of the LV? You don't show any
results which failed though. Can you elaborate?

[David Clarke (a-launchpad-uber-geek-nz)]

The original 'it fails ~50% of the time' was when I was testing with my desired setup of a 10 GB LV.

The test01 -> test30 lines are from a different, and later, run to the original more verbose excerpts.

[Aaron Brady (bradya)]

I also see this. Have deployed 50 production VMs, it happened approximately 10% of the time.

I'm using KVM with LVM and vm-builder; these options:

vmbuilder kvm ubuntu --hostname=vmbuilt --domain=manc.iws-hosting.co.uk --raw=/dev/svs4/vmbuilt-disk --mem=512 --user=iweb --pass=xxx --rootpass=xxx --cpus=1 --templates=/templates --dest=vmbuilt --ip=192.168.1.201 --mask=255.255.255.0 --gw=192.168.1.182 --dns=193.228.155.86 --rootsize=8192 --swapsize=1024

dd'ing off the first sector of the LV shows that GRUB just isn't present in the MBR and launching a VM just stops with no GRUB messages.

[Chuck Short (zulcss)]

Which version is this with?

chuck

[Launchpad Janitor (janitor)]

[Expired for vm-builder (Ubuntu) because there has been no activity for 60 days.]

[Sergey Svishchev (svs)]

0.12.4-0ubuntu0.2 on 10.04

[Sergey Svishchev (svs)]

bug 654723 looks like a duplicate of this one.

[lhotari (lartsa)]

vmbuilder should generate a proper /etc/mtab file in the disk image / chroot before running grub.
For some reason grub requires that /etc/mtab contains the root partition in the mtab list.

embedded mode for grub fails since vmbuilder makes partitions starting from sector 1. virt-manager starts from sector 63.

[Will Bryant (willbryant)]

Just found this one after posting on #946022. I have the same problem. For me it occurs around half the time. Fails in the same way in grub:

 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,044 DEBUG : ['chroot', '/tmp/tmpgvQfyy', 'grub', '--device-map=/tmp/vmbuilder-grub/device.map', '--batch']
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,044 DEBUG : stdin was set and it was a string: root (hd0,0)
 ** [out :: host-nwtn1.nwtn] setup (hd0)
 ** [out :: host-nwtn1.nwtn] EOT
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,197 DEBUG :
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,197 DEBUG : [ Minimal BASH-like line editing is supported. For
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,198 DEBUG : the first word, TAB lists possible command
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,198 DEBUG : completions. Anywhere else TAB lists the possible
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,198 DEBUG : completions of a device/filename. ]
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,198 DEBUG : grub> root (hd0,0)
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,198 DEBUG : grub> setup (hd0)
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,198 DEBUG : Checking if "/boot/grub/stage1" exists... no
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,199 DEBUG : Checking if "/grub/stage1" exists... no
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,199 DEBUG :
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,199 DEBUG : Error 15: File not found
 ** [out :: host-nwtn1.nwtn] 2013-03-17 10:52:33,199 DEBUG : grub> EOT

My full failed & successful logs are attached to #946022.