Bug #607355 “Problems launching instances” : Bugs : OpenStack Compute (nova)

Revision history for this message

justinsb (justin-fathomdb) wrote on 2010-07-19:

#1

Just to clarify, those steps to get the machine to boot the original image are cumulative... so the final working command was this:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin /usr/bin/qemu-system-x86_64 -M pc-0.12 -m 1024 -smp 1 -name i-lrpnh3ta -uuid b43ba850-2170-6a96-18ec-0f4f67a472dc -boot c -drive file=/home/justinsb/openstack/images/kvm/ubuntu-kvm/tmpK440Qi.qcow2,if=virtio,index=0,boot=on -net nic,macaddr=00:16:3e:76:88:f5,vlan=0,name=nic.0 -chardev file,id=serial0,path=/home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/console.log -serial chardev:serial0 -parallel none -usb

And then if I swap that image back to the nova one it still fails to boot as it did before the other modifications.

Revision history for this message

Soren Hansen (soren) wrote on 2010-07-20: Re: [Bug 607355] [NEW] Problems launching instances

#2

On Mon, Jul 19, 2010 at 05:35:37PM -0000, justinsb wrote:
> qemu: linux kernel too old to load a ram disk

This usually means that one or both of kernel and ramdisk has been
transferred incorrectly. You can go and see in
/var/lib/instances/<instance-id>/ and inspect the kernel and ramdisk

> * If I remove the -kernel, -initrd, and -append arguments I can get
> KVM to stop complaining about "kernel too old to load a ram disk"
>
> * If I leave the disk image as-is, it won't boot - it just hangs at the
> BIOS "Booting from Hard Disk" message (maybe nova is creating a
> corrupted disk image, c.f. "Input partition size not evenly divisible by
> sector size: 169 / 512" message?)

Make sure you've got parted installed. Nova does not currently check
properly if parted is installed, and does not handle the error if it
isn't.

> * If I swap the disk image to the original disk image created by
> vmbuilder, it hangs at "GRUB loading stage2"

> There's a whole bunch of questions here:
>
> Why are we using a kernel and ramdisk anyway - why not just rely on hardware virtualization?

Heritage from EC2/Eucalyptus. You provide a filesystem image, a kernel
and a ramdisk, and it gets stitched together to make up a virtual
machine.

> We presumably also wouldn't need to mess with the disk image...

Again, heritage from EC2/Eucalyptus. The idea was to look as much like
EC2 as possible. On EC2, the filesystem you provide is reachable on
/dev/sda2, and to make that happen with KVM, you need to fiddle with
disk images.

This definitely needs to be revisisted.

> How is that -no-kvm argument sneaking in there?

Can you attach the libvirt.xml from /var/lib/instances/<instance id>/ ?

> I'm guessing it's something I've misconfigured in the libvirt
> template?

That's my hypothesis, too. Did you run the sed command from the
NovaInstallFestInstructions page per chance?

> What is the correct way to build disk images, and where should the
> ramdisk & kernel come from?

That's a good question. Ḯ'll post info on how I built the one I used
tomorrow.

> Is there a way to boot a raw image without ramdisk/kernel?

Not at the moment. Patches to fix this are very welcome. You seem to
have identified that things that need to change. If not, don't hesitate
to ask.

--
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/

On Mon, Jul 19, 2010 at 05:35:37PM -0000, justinsb wrote:
> qemu: linux kernel too old to load a ram disk

This usually means that one or both of kernel and ramdisk has been
transferred incorrectly. You can go and see in
/var/lib/instances/<instance-id>/ and inspect the kernel and ramdisk

> * If I remove the -kernel, -initrd, and -append arguments I can get
> KVM to stop complaining about "kernel too old to load a ram disk"
> 
> * If I leave the disk image as-is, it won't boot - it just hangs at the
> BIOS "Booting from Hard Disk" message  (maybe nova is creating a
> corrupted disk image, c.f. "Input partition size not evenly divisible by
> sector size: 169 / 512" message?)

Make sure you've got parted installed. Nova does not currently check
properly if parted is installed, and does not handle the error if it
isn't.

> * If I swap the disk image to the original disk image created by
> vmbuilder, it hangs at "GRUB loading stage2"

> There's a whole bunch of questions here:
> 
> Why are we using a kernel and ramdisk anyway - why not just rely on hardware virtualization?

Heritage from EC2/Eucalyptus. You provide a filesystem image, a kernel
and a ramdisk, and it gets stitched together to make up a virtual
machine.

> We presumably also wouldn't need to mess with the disk image...

Again, heritage from EC2/Eucalyptus. The idea was to look as much like
EC2 as possible. On EC2, the filesystem you provide is reachable on
/dev/sda2, and to make that happen with KVM, you need to fiddle with
disk images.

This definitely needs to be revisisted.

> How is that -no-kvm argument sneaking in there?

Can you attach the libvirt.xml from /var/lib/instances/<instance id>/ ?

> I'm guessing it's something I've misconfigured in the libvirt
> template?

That's my hypothesis, too. Did you run the sed command from the
NovaInstallFestInstructions page per chance?

> What is the correct way to build disk images, and where should the
> ramdisk & kernel come from?

That's a good question. Ḯ'll post info on how I built the one I used
tomorrow.

> Is there a way to boot a raw image without ramdisk/kernel?

Not at the moment. Patches to fix this are very welcome. You seem to
have identified that things that need to change. If not, don't hesitate
to ask.

-- 
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/

Revision history for this message

Joshua McKenty (joshua-mckenty) wrote on 2010-07-20:

#3

Download full text (7.1 KiB)

I'm also working on what's been loosely called "bootable volume" support,
which involves treating all disk images the same way - as network-attached
storage, exported from one of the volume nodes. This solves several things
at once: VM migration, Xen support, snapshotting, etc.

We'll need to finalize the 'best' way to manage kernel, ramdisk, etc. NASA
has a requirement to keep these separate from the disk images, so it's not
JUST legacy. But it doesn't mean it should be the only way nova works.

Joshua

On Mon, Jul 19, 2010 at 7:47 PM, Soren Hansen <email address hidden> wrote:

> On Mon, Jul 19, 2010 at 05:35:37PM -0000, justinsb wrote:
> > qemu: linux kernel too old to load a ram disk
>
> This usually means that one or both of kernel and ramdisk has been
> transferred incorrectly. You can go and see in
> /var/lib/instances/<instance-id>/ and inspect the kernel and ramdisk
>
> > * If I remove the -kernel, -initrd, and -append arguments I can get
> > KVM to stop complaining about "kernel too old to load a ram disk"
> >
> > * If I leave the disk image as-is, it won't boot - it just hangs at the
> > BIOS "Booting from Hard Disk" message (maybe nova is creating a
> > corrupted disk image, c.f. "Input partition size not evenly divisible by
> > sector size: 169 / 512" message?)
>
> Make sure you've got parted installed. Nova does not currently check
> properly if parted is installed, and does not handle the error if it
> isn't.
>
> > * If I swap the disk image to the original disk image created by
> > vmbuilder, it hangs at "GRUB loading stage2"
>
> > There's a whole bunch of questions here:
> >
> > Why are we using a kernel and ramdisk anyway - why not just rely on
> hardware virtualization?
>
> Heritage from EC2/Eucalyptus. You provide a filesystem image, a kernel
> and a ramdisk, and it gets stitched together to make up a virtual
> machine.
>
> > We presumably also wouldn't need to mess with the disk image...
>
> Again, heritage from EC2/Eucalyptus. The idea was to look as much like
> EC2 as possible. On EC2, the filesystem you provide is reachable on
> /dev/sda2, and to make that happen with KVM, you need to fiddle with
> disk images.
>
> This definitely needs to be revisisted.
>
> > How is that -no-kvm argument sneaking in there?
>
> Can you attach the libvirt.xml from /var/lib/instances/<instance id>/ ?
>
> > I'm guessing it's something I've misconfigured in the libvirt
> > template?
>
> That's my hypothesis, too. Did you run the sed command from the
> NovaInstallFestInstructions page per chance?
>
> > What is the correct way to build disk images, and where should the
> > ramdisk & kernel come from?
>
> That's a good question. Ḯ'll post info on how I built the one I used
> tomorrow.
>
> > Is there a way to boot a raw image without ramdisk/kernel?
>
> Not at the moment. Patches to fix this are very welcome. You seem to
> have identified that things that need to change. If not, don't hesitate
> to ask.
>
> --
> Soren Hansen
> Ubuntu Developer
> http://www.ubuntu.com/
>
> --
> Problems launching instances
> https://bugs.launchpad.net/bugs/607355
> You received this bug notification because you are a member of Nova
> Bugs, which is subscribed to O...

I'm also working on what's been loosely called "bootable volume" support,
which involves treating all disk images the same way - as network-attached
storage, exported from one of the volume nodes. This solves several things
at once: VM migration, Xen support, snapshotting, etc.

We'll need to finalize the 'best' way to manage kernel, ramdisk, etc. NASA
has a requirement to keep these separate from the disk images, so it's not
JUST legacy. But it doesn't mean it should be the only way nova works.

Joshua

On Mon, Jul 19, 2010 at 7:47 PM, Soren Hansen <soren@ubuntu.com> wrote:

> On Mon, Jul 19, 2010 at 05:35:37PM -0000, justinsb wrote:
> > qemu: linux kernel too old to load a ram disk
>
> This usually means that one or both of kernel and ramdisk has been
> transferred incorrectly. You can go and see in
> /var/lib/instances/<instance-id>/ and inspect the kernel and ramdisk
>
> > * If I remove the -kernel, -initrd, and -append arguments I can get
> > KVM to stop complaining about "kernel too old to load a ram disk"
> >
> > * If I leave the disk image as-is, it won't boot - it just hangs at the
> > BIOS "Booting from Hard Disk" message  (maybe nova is creating a
> > corrupted disk image, c.f. "Input partition size not evenly divisible by
> > sector size: 169 / 512" message?)
>
> Make sure you've got parted installed. Nova does not currently check
> properly if parted is installed, and does not handle the error if it
> isn't.
>
> > * If I swap the disk image to the original disk image created by
> > vmbuilder, it hangs at "GRUB loading stage2"
>
> > There's a whole bunch of questions here:
> >
> > Why are we using a kernel and ramdisk anyway - why not just rely on
> hardware virtualization?
>
> Heritage from EC2/Eucalyptus. You provide a filesystem image, a kernel
> and a ramdisk, and it gets stitched together to make up a virtual
> machine.
>
> > We presumably also wouldn't need to mess with the disk image...
>
> Again, heritage from EC2/Eucalyptus. The idea was to look as much like
> EC2 as possible. On EC2, the filesystem you provide is reachable on
> /dev/sda2, and to make that happen with KVM, you need to fiddle with
> disk images.
>
> This definitely needs to be revisisted.
>
> > How is that -no-kvm argument sneaking in there?
>
> Can you attach the libvirt.xml from /var/lib/instances/<instance id>/ ?
>
> > I'm guessing it's something I've misconfigured in the libvirt
> > template?
>
> That's my hypothesis, too. Did you run the sed command from the
> NovaInstallFestInstructions page per chance?
>
> > What is the correct way to build disk images, and where should the
> > ramdisk & kernel come from?
>
> That's a good question. Ḯ'll post info on how I built the one I used
> tomorrow.
>
> > Is there a way to boot a raw image without ramdisk/kernel?
>
> Not at the moment. Patches to fix this are very welcome. You seem to
> have identified that things that need to change. If not, don't hesitate
> to ask.
>
> --
> Soren Hansen
> Ubuntu Developer
> http://www.ubuntu.com/
>
> --
> Problems launching instances
> https://bugs.launchpad.net/bugs/607355
> You received this bug notification because you are a member of Nova
> Bugs, which is subscribed to OpenStack Compute (nova).
>
> Status in OpenStack Compute (Nova): New
>
> Bug description:
> I'm struggling to launch an image (on physical hardware, using KVM)
>
> I've tried building my own image, using ubuntu's vmbuilder, and got that to
> build an image.  However, I can't launch this through nova.
>
> The compute service logs this error:
>
> DEBUG:root:Starting instance i-lrpnh3ta...
> DEBUG:root:Result was 1
> DEBUG:root:Starting VLAN inteface for 100 network
> DEBUG:root:Result was 1
> DEBUG:root:Result was 1
> DEBUG:root:Result was 255
> DEBUG:root:Finished init of Instance with id of i-lrpnh3ta
> INFO:root:Instances current state is 0
> DEBUG:root:Instance state is: 0
> DEBUG:root:Starting spawn in Instance
> DEBUG:root:Starting the toXML method
> DEBUG:root:Finished the toXML method
> INFO:root:self <nova.compute.node.Instance object at 0x2e56e50>
> INFO:root:Creating image for: i-lrpnh3ta
> WARNING:root:Input partition size not evenly divisible by sector size: 169
> / 512
> libvir: QEMU error : monitor socket did not show up.: Connection refused
> DEBUG:root:monitor socket did not show up.: Connection refused
>
>
> Peeking behind the libvirt obfuscation:
>
> sudo more /var/log/libvirt/qemu/i-lrpnh3ta.log
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> /usr/bin/qemu-system-x86_64 -S -M pc-0.12 -no-kvm -m 1024 -smp 1 -name
> i-lrpnh3ta -uuid b43ba850-2170-6a96-18ec-0f4f67a472dc -nographic -chardev
> socket,id=monitor,path=/var/lib/libvirt/qemu/i-lrpnh3ta.monitor,server,nowait
> -monitor chardev:monitor -boot c -kernel
> /home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/kernel
> -initrd
> /home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/ramdisk
> -append root=/dev/vda1 console=ttyS0 -drive
> file=/home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/disk,if=virtio,index=0,boot=on
> -net nic,macaddr=00:16:3e:76:88:f5,vlan=0,name=nic.0 -net
> tap,fd=44,vlan=0,name=tap.0 -chardev
> file,id=serial0,path=/home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/console.log
> -serial chardev:serial0 -parallel none -usb
> qemu: linux kernel too old to load a ram disk
>
>
>
> I'm using the kernel from /boot/vmlinuz-$(uname -r) on the host (Lucid
> Lynx); this might not be right.
>
> However, taking that KVM command, I can change it so that I can boot the
> original VM:
>
> * I remove the -S (stop) and monitor stuff so that it won't wait for
> libvirt, and the -nographic arg so that I can see what's going on.
>
> * If I remove the -kernel, -initrd, and -append arguments I can get KVM to
> stop complaining about "kernel too old to load a ram disk"
>
> * If I leave the disk image as-is, it won't boot - it just hangs at the
> BIOS "Booting from Hard Disk" message  (maybe nova is creating a corrupted
> disk image, c.f. "Input partition size not evenly divisible by sector size:
> 169 / 512" message?)
>
> * If I swap the disk image to the original disk image created by vmbuilder,
> it hangs at "GRUB loading stage2"
>
> * If I remove -no-kvm, I can use the grub menu, but it then hangs when
> starting the actual image (at "Starting up").  I now get these error
> messages from KVM:
> TUNGETIFF ioctl() failed: Bad file descriptor
> TUNSETOFFLOAD ioctl() failed: Bad file descriptor
>
> * If I remove the tap device (-net tap,fd=44,vlan=0,name=tap.0), then the
> machine boots and I can log in.
>
> * (If I put the disk image back to the disk image nova created, it hangs as
> before on "Booting from Hard Disk")
>
>
>
> There's a whole bunch of questions here:
>
> Why are we using a kernel and ramdisk anyway - why not just rely on
> hardware virtualization?  We presumably also wouldn't need to mess with the
> disk image...
> How is that -no-kvm argument sneaking in there?  I'm guessing it's
> something I've misconfigured in the libvirt template?
> What is the correct way to build disk images, and where should the ramdisk
> & kernel come from?  Is there a way to boot a raw image without
> ramdisk/kernel?
>
>
>

Revision history for this message

Soren Hansen (soren) wrote on 2010-09-08:

#4

Justin, I'm having trouble triaging this bug. Do you think you could go through the issues you highlight here, file bugs for the ones that still apply and close this bug once you're done? That way we can track the individual issues separately which makes working on them easier and it makes it more clear when the bug is fixed.

Changed in nova:
status:	New → Incomplete

Revision history for this message

Thierry Carrez (ttx) wrote on 2011-01-18:

#5

Works for me, closing as Invalid, please reopen if you still experience issues and can provide the requested info.

Changed in nova:
status:	Incomplete → Invalid

OpenStack Compute (nova)

Problems launching instances

Bug Description

Other bug subscribers

Remote bug watches