OpenStack Compute (nova)

Bug #607355
Comment #3

Comment 3 for bug 607355

Revision history for this message

Joshua McKenty (joshua-mckenty) wrote on 2010-07-20: Re: [Bug 607355] [NEW] Problems launching instances

I'm also working on what's been loosely called "bootable volume" support,
which involves treating all disk images the same way - as network-attached
storage, exported from one of the volume nodes. This solves several things
at once: VM migration, Xen support, snapshotting, etc.

We'll need to finalize the 'best' way to manage kernel, ramdisk, etc. NASA
has a requirement to keep these separate from the disk images, so it's not
JUST legacy. But it doesn't mean it should be the only way nova works.

Joshua

On Mon, Jul 19, 2010 at 7:47 PM, Soren Hansen <email address hidden> wrote:

> On Mon, Jul 19, 2010 at 05:35:37PM -0000, justinsb wrote:
> > qemu: linux kernel too old to load a ram disk
>
> This usually means that one or both of kernel and ramdisk has been
> transferred incorrectly. You can go and see in
> /var/lib/instances/<instance-id>/ and inspect the kernel and ramdisk
>
> > * If I remove the -kernel, -initrd, and -append arguments I can get
> > KVM to stop complaining about "kernel too old to load a ram disk"
> >
> > * If I leave the disk image as-is, it won't boot - it just hangs at the
> > BIOS "Booting from Hard Disk" message (maybe nova is creating a
> > corrupted disk image, c.f. "Input partition size not evenly divisible by
> > sector size: 169 / 512" message?)
>
> Make sure you've got parted installed. Nova does not currently check
> properly if parted is installed, and does not handle the error if it
> isn't.
>
> > * If I swap the disk image to the original disk image created by
> > vmbuilder, it hangs at "GRUB loading stage2"
>
> > There's a whole bunch of questions here:
> >
> > Why are we using a kernel and ramdisk anyway - why not just rely on
> hardware virtualization?
>
> Heritage from EC2/Eucalyptus. You provide a filesystem image, a kernel
> and a ramdisk, and it gets stitched together to make up a virtual
> machine.
>
> > We presumably also wouldn't need to mess with the disk image...
>
> Again, heritage from EC2/Eucalyptus. The idea was to look as much like
> EC2 as possible. On EC2, the filesystem you provide is reachable on
> /dev/sda2, and to make that happen with KVM, you need to fiddle with
> disk images.
>
> This definitely needs to be revisisted.
>
> > How is that -no-kvm argument sneaking in there?
>
> Can you attach the libvirt.xml from /var/lib/instances/<instance id>/ ?
>
> > I'm guessing it's something I've misconfigured in the libvirt
> > template?
>
> That's my hypothesis, too. Did you run the sed command from the
> NovaInstallFestInstructions page per chance?
>
> > What is the correct way to build disk images, and where should the
> > ramdisk & kernel come from?
>
> That's a good question. Ḯ'll post info on how I built the one I used
> tomorrow.
>
> > Is there a way to boot a raw image without ramdisk/kernel?
>
> Not at the moment. Patches to fix this are very welcome. You seem to
> have identified that things that need to change. If not, don't hesitate
> to ask.
>
> --
> Soren Hansen
> Ubuntu Developer
> http://www.ubuntu.com/
>
> --
> Problems launching instances
> https://bugs.launchpad.net/bugs/607355
> You received this bug notification because you are a member of Nova
> Bugs, which is subscribed to OpenStack Compute (nova).
>
> Status in OpenStack Compute (Nova): New
>
> Bug description:
> I'm struggling to launch an image (on physical hardware, using KVM)
>
> I've tried building my own image, using ubuntu's vmbuilder, and got that to
> build an image. However, I can't launch this through nova.
>
> The compute service logs this error:
>
> DEBUG:root:Starting instance i-lrpnh3ta...
> DEBUG:root:Result was 1
> DEBUG:root:Starting VLAN inteface for 100 network
> DEBUG:root:Result was 1
> DEBUG:root:Result was 1
> DEBUG:root:Result was 255
> DEBUG:root:Finished init of Instance with id of i-lrpnh3ta
> INFO:root:Instances current state is 0
> DEBUG:root:Instance state is: 0
> DEBUG:root:Starting spawn in Instance
> DEBUG:root:Starting the toXML method
> DEBUG:root:Finished the toXML method
> INFO:root:self <nova.compute.node.Instance object at 0x2e56e50>
> INFO:root:Creating image for: i-lrpnh3ta
> WARNING:root:Input partition size not evenly divisible by sector size: 169
> / 512
> libvir: QEMU error : monitor socket did not show up.: Connection refused
> DEBUG:root:monitor socket did not show up.: Connection refused
>
>
> Peeking behind the libvirt obfuscation:
>
> sudo more /var/log/libvirt/qemu/i-lrpnh3ta.log
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> /usr/bin/qemu-system-x86_64 -S -M pc-0.12 -no-kvm -m 1024 -smp 1 -name
> i-lrpnh3ta -uuid b43ba850-2170-6a96-18ec-0f4f67a472dc -nographic -chardev
> socket,id=monitor,path=/var/lib/libvirt/qemu/i-lrpnh3ta.monitor,server,nowait
> -monitor chardev:monitor -boot c -kernel
> /home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/kernel
> -initrd
> /home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/ramdisk
> -append root=/dev/vda1 console=ttyS0 -drive
> file=/home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/disk,if=virtio,index=0,boot=on
> -net nic,macaddr=00:16:3e:76:88:f5,vlan=0,name=nic.0 -net
> tap,fd=44,vlan=0,name=tap.0 -chardev
> file,id=serial0,path=/home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/console.log
> -serial chardev:serial0 -parallel none -usb
> qemu: linux kernel too old to load a ram disk
>
>
>
> I'm using the kernel from /boot/vmlinuz-$(uname -r) on the host (Lucid
> Lynx); this might not be right.
>
> However, taking that KVM command, I can change it so that I can boot the
> original VM:
>
> * I remove the -S (stop) and monitor stuff so that it won't wait for
> libvirt, and the -nographic arg so that I can see what's going on.
>
> * If I remove the -kernel, -initrd, and -append arguments I can get KVM to
> stop complaining about "kernel too old to load a ram disk"
>
> * If I leave the disk image as-is, it won't boot - it just hangs at the
> BIOS "Booting from Hard Disk" message (maybe nova is creating a corrupted
> disk image, c.f. "Input partition size not evenly divisible by sector size:
> 169 / 512" message?)
>
> * If I swap the disk image to the original disk image created by vmbuilder,
> it hangs at "GRUB loading stage2"
>
> * If I remove -no-kvm, I can use the grub menu, but it then hangs when
> starting the actual image (at "Starting up"). I now get these error
> messages from KVM:
> TUNGETIFF ioctl() failed: Bad file descriptor
> TUNSETOFFLOAD ioctl() failed: Bad file descriptor
>
> * If I remove the tap device (-net tap,fd=44,vlan=0,name=tap.0), then the
> machine boots and I can log in.
>
> * (If I put the disk image back to the disk image nova created, it hangs as
> before on "Booting from Hard Disk")
>
>
>
> There's a whole bunch of questions here:
>
> Why are we using a kernel and ramdisk anyway - why not just rely on
> hardware virtualization? We presumably also wouldn't need to mess with the
> disk image...
> How is that -no-kvm argument sneaking in there? I'm guessing it's
> something I've misconfigured in the libvirt template?
> What is the correct way to build disk images, and where should the ramdisk
> & kernel come from? Is there a way to boot a raw image without
> ramdisk/kernel?
>
>
>

Joshua

On Mon, Jul 19, 2010 at 7:47 PM, Soren Hansen <soren@ubuntu.com> wrote:

> On Mon, Jul 19, 2010 at 05:35:37PM -0000, justinsb wrote:
> > qemu: linux kernel too old to load a ram disk
>
> This usually means that one or both of kernel and ramdisk has been
> transferred incorrectly. You can go and see in
> /var/lib/instances/<instance-id>/ and inspect the kernel and ramdisk
>
> > * If I remove the -kernel, -initrd, and -append arguments I can get
> > KVM to stop complaining about "kernel too old to load a ram disk"
> >
> > * If I leave the disk image as-is, it won't boot - it just hangs at the
> > BIOS "Booting from Hard Disk" message  (maybe nova is creating a
> > corrupted disk image, c.f. "Input partition size not evenly divisible by
> > sector size: 169 / 512" message?)
>
> Make sure you've got parted installed. Nova does not currently check
> properly if parted is installed, and does not handle the error if it
> isn't.
>
> > * If I swap the disk image to the original disk image created by
> > vmbuilder, it hangs at "GRUB loading stage2"
>
> > There's a whole bunch of questions here:
> >
> > Why are we using a kernel and ramdisk anyway - why not just rely on
> hardware virtualization?
>
> Heritage from EC2/Eucalyptus. You provide a filesystem image, a kernel
> and a ramdisk, and it gets stitched together to make up a virtual
> machine.
>
> > We presumably also wouldn't need to mess with the disk image...
>
> Again, heritage from EC2/Eucalyptus. The idea was to look as much like
> EC2 as possible. On EC2, the filesystem you provide is reachable on
> /dev/sda2, and to make that happen with KVM, you need to fiddle with
> disk images.
>
> This definitely needs to be revisisted.
>
> > How is that -no-kvm argument sneaking in there?
>
> Can you attach the libvirt.xml from /var/lib/instances/<instance id>/ ?
>
> > I'm guessing it's something I've misconfigured in the libvirt
> > template?
>
> That's my hypothesis, too. Did you run the sed command from the
> NovaInstallFestInstructions page per chance?
>
> > What is the correct way to build disk images, and where should the
> > ramdisk & kernel come from?
>
> That's a good question. Ḯ'll post info on how I built the one I used
> tomorrow.
>
> > Is there a way to boot a raw image without ramdisk/kernel?
>
> Not at the moment. Patches to fix this are very welcome. You seem to
> have identified that things that need to change. If not, don't hesitate
> to ask.
>
> --
> Soren Hansen
> Ubuntu Developer
> http://www.ubuntu.com/
>
> --
> Problems launching instances
> https://bugs.launchpad.net/bugs/607355
> You received this bug notification because you are a member of Nova
> Bugs, which is subscribed to OpenStack Compute (nova).
>
> Status in OpenStack Compute (Nova): New
>
> Bug description:
> I'm struggling to launch an image (on physical hardware, using KVM)
>
> I've tried building my own image, using ubuntu's vmbuilder, and got that to
> build an image.  However, I can't launch this through nova.
>
> The compute service logs this error:
>
> DEBUG:root:Starting instance i-lrpnh3ta...
> DEBUG:root:Result was 1
> DEBUG:root:Starting VLAN inteface for 100 network
> DEBUG:root:Result was 1
> DEBUG:root:Result was 1
> DEBUG:root:Result was 255
> DEBUG:root:Finished init of Instance with id of i-lrpnh3ta
> INFO:root:Instances current state is 0
> DEBUG:root:Instance state is: 0
> DEBUG:root:Starting spawn in Instance
> DEBUG:root:Starting the toXML method
> DEBUG:root:Finished the toXML method
> INFO:root:self <nova.compute.node.Instance object at 0x2e56e50>
> INFO:root:Creating image for: i-lrpnh3ta
> WARNING:root:Input partition size not evenly divisible by sector size: 169
> / 512
> libvir: QEMU error : monitor socket did not show up.: Connection refused
> DEBUG:root:monitor socket did not show up.: Connection refused
>
>
> Peeking behind the libvirt obfuscation:
>
> sudo more /var/log/libvirt/qemu/i-lrpnh3ta.log
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
> /usr/bin/qemu-system-x86_64 -S -M pc-0.12 -no-kvm -m 1024 -smp 1 -name
> i-lrpnh3ta -uuid b43ba850-2170-6a96-18ec-0f4f67a472dc -nographic -chardev
> socket,id=monitor,path=/var/lib/libvirt/qemu/i-lrpnh3ta.monitor,server,nowait
> -monitor chardev:monitor -boot c -kernel
> /home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/kernel
> -initrd
> /home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/ramdisk
> -append root=/dev/vda1 console=ttyS0 -drive
> file=/home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/disk,if=virtio,index=0,boot=on
> -net nic,macaddr=00:16:3e:76:88:f5,vlan=0,name=nic.0 -net
> tap,fd=44,vlan=0,name=tap.0 -chardev
> file,id=serial0,path=/home/justinsb/openstack/iscsi-volumes/data/instances/i-lrpnh3ta/console.log
> -serial chardev:serial0 -parallel none -usb
> qemu: linux kernel too old to load a ram disk
>
>
>
> I'm using the kernel from /boot/vmlinuz-$(uname -r) on the host (Lucid
> Lynx); this might not be right.
>
> However, taking that KVM command, I can change it so that I can boot the
> original VM:
>
> * I remove the -S (stop) and monitor stuff so that it won't wait for
> libvirt, and the -nographic arg so that I can see what's going on.
>
> * If I remove the -kernel, -initrd, and -append arguments I can get KVM to
> stop complaining about "kernel too old to load a ram disk"
>
> * If I leave the disk image as-is, it won't boot - it just hangs at the
> BIOS "Booting from Hard Disk" message  (maybe nova is creating a corrupted
> disk image, c.f. "Input partition size not evenly divisible by sector size:
> 169 / 512" message?)
>
> * If I swap the disk image to the original disk image created by vmbuilder,
> it hangs at "GRUB loading stage2"
>
> * If I remove -no-kvm, I can use the grub menu, but it then hangs when
> starting the actual image (at "Starting up").  I now get these error
> messages from KVM:
> TUNGETIFF ioctl() failed: Bad file descriptor
> TUNSETOFFLOAD ioctl() failed: Bad file descriptor
>
> * If I remove the tap device (-net tap,fd=44,vlan=0,name=tap.0), then the
> machine boots and I can log in.
>
> * (If I put the disk image back to the disk image nova created, it hangs as
> before on "Booting from Hard Disk")
>
>
>
> There's a whole bunch of questions here:
>
> Why are we using a kernel and ramdisk anyway - why not just rely on
> hardware virtualization?  We presumably also wouldn't need to mess with the
> disk image...
> How is that -no-kvm argument sneaking in there?  I'm guessing it's
> something I've misconfigured in the libvirt template?
> What is the correct way to build disk images, and where should the ramdisk
> & kernel come from?  Is there a way to boot a raw image without
> ramdisk/kernel?
>
>
>