Ubuntu 12.10 cloud images do not full provision on Azure

[Ben Howard]

Details are still forcoming, but Ubuntu Cloud Images for 12.10 do not finish the provisioning state. Per the dmesg output, the images complete boot, but do not respond to SSH nor do they report to the Azure fabric that the images are role-ready.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1055686/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Ben Howard (darkmuggle-deactivatedaccount)]

The issue is that Grub2.00 is not recognizing the target. The error message is:
"error: couldn't find suitable memory target."

grub-core/lib/relocator.c
1410 #ifdef GRUB_MACHINE_EFI
1411 grub_efi_mmap_iterate (hook, avoid_efi_boot_services);
1412 #elif defined (__powerpc__)
1413 (void) avoid_efi_boot_services;
1414 grub_machine_mmap_iterate (hook);
1415 #else
1416 (void) avoid_efi_boot_services;
1417 grub_mmap_iterate (hook);
1418 #endif
1419 if (!found)
1420 return grub_error (GRUB_ERR_BAD_OS, "couldn't find suitable memory target");
1421 }
1422 while (1)

[Ben Howard (darkmuggle-deactivatedaccount)]

Increasing the boot time out did not fix the problem.

[Ben Howard (darkmuggle-deactivatedaccount)]

The analysis is that grub is attempting to allocate memory but is not able to find a suitable memory location. This would explain why the initrd is not being loaded into ram.

[Ben Howard (darkmuggle-deactivatedaccount)]

last part of boot. xz compress avi.

[Ben Howard (darkmuggle-deactivatedaccount)]

Full boot log with grub2 options of "set debug=all; set pager=1" turned on.

[Ben Howard (darkmuggle-deactivatedaccount)]

On a wild hunch, I built an image with a minimial initrd -- just the bits that are needed to boot -- and the system booted. This means that the problem is with the size of the initramfs.

On a default install the initramfs is ~4.5MB. However, if you install the linux-image-extras-virtual (requirement for Azure due to the need for the UFS module), that bloats the initramfs to ~19MB on the lastest kernel.

This means that the issue is likely the platform.

Here is the configuration for the initramfs that I used:
$ cat /etc/initramfs-tools/modules
sd_mod
scsi_mod
hid
hid_hyperv
hv_utils
hv_storvsc
hv_vmbus
hv_netvsc
crc_itu_t

$ cat /etc/initramfs-tools/initramfs.conf
MODULES=list
BUSYBOX=n
COMPCACHE_SIZE=""
COMPRESS=xz
BOOT=local
DEVICE=
NFSROOT=auto

[Colin Watson (cjwatson)]

So, there is certainly a limit on the size of the compressed kernel + its init_size + the compressed initrd, and on this platform as you say it's a little under 64MB. However, I can't make this add up to a problem in this case, and I think there may be some double-counting going on in GRUB that exacerbates the problem.

I looked hard at how GRUB allocates memory for the initrd. The maximum address is capped to 0x37FFFFFF, thereby excluding the 0x40000000-0xABFFFFFF range that's available, so the effective available range per E820 data is 0x100000-0x3FEFFFF, minus what's already been allocated for the kernel. The minimum address is then taken as the target physical address of the kernel (0x1000000) plus its required init space (0x134F000) plus the initrd size. But the initrd is going to be allocated *above* this address. Why does it need to add the initrd size? The effect of this calculation is to limit the initrd size to about 15MB when it should be limited to about 30MB.

You might like to try this patch:

  http://paste.ubuntu.com/1275848/

Beware that since it is nearly midnight here I've only minimally tested this.

[Ben Howard (darkmuggle-deactivatedaccount)]

Patch is confirmed to work on Windows Azure with a 18M initrd.

[ 0.894608] registered taskstats version 1
[ 1.093496] Freeing initrd memory: 18432k freed
[ 1.105264] Key type trusted registered
[ 1.110574] Key type encrypted registered

[Ben Howard (darkmuggle-deactivatedaccount)]

Regarding patching Grub2 or zero-day SRU: given that the known affected system, I think that a zero-day SRU is fine, unless there is another Grub2 build that needs committing. We would love to have this patch hit -proposed so that we build against it.

[Colin Watson (cjwatson)]

Thanks. I'm unavailable for most of today, but I've forwarded the patch upstream to start with. I'll prepare an upload for quantal-proposed as soon as I can.

[Colin Watson (cjwatson)]

This is in quantal-proposed now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.00-7ubuntu11

---------------
grub2 (2.00-7ubuntu11) quantal-proposed; urgency=low

  * Fix incorrect initrd minimum address calculation (LP: #1055686).
  * Add keystatus and loadenv to signed image (LP: #1066399).
 -- Colin Watson <email address hidden> Sun, 14 Oct 2012 09:30:55 +0100