Kernel Panic while booting Natty installer kernel (2.6.37-10-generic) on amd64 ISO

[Jeff Lane ]

Attempting to run some weekly ISO testing using the AMD64 images against systems in the Boston Data center. I refreshed the ISOs on the satellite as of today , 21 December 2010. I then set about setting up the servers PXE configurations and then rebooting each server. I did this as a batch run last night only to find that none of the servers had installed and run checkbox and submitted results over night.

So this morning, I refreshed the ISOs again, with the latest as of the morning of the 21st. I picked one system (Abiu) and configured and restarted that one. I watched that one via the KVM and found that once the installer kernel starts to load, it kernel panics and then stops.

I've attached a screen shot, which is the best I can get for now as far as logs go. But I also tried this on Coconut and Muscadine and all three systems stopped in the same place with the same kernel trace.

[Jeff Lane  (bladernr)]

Here's a screen shot captured through the remote KVM attached to Abiu this morning.

[Jeff Lane  (bladernr)]

Realized a little while later that 2.6.37-9 is not the correct kernel. Refreshed the images and tried again and got the same issue on the systems but with 2.6.37-10-generic as shown in the new screen shot attached.

[Jeff Lane  (bladernr)]

Here's the full set of boot messages

[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.37-10-generic (buildd@allspice) (gcc version 4.5.2 (Ubuntu/Linaro 4.5.1-12ubuntu1) ) #24-Ubuntu SMP Thu Dec 16 17:54:02 UTC 2010 (Ubuntu 2.6.37-10.24-generic 2.6.37-rc6)
[ 0.000000] Command line: vga=normal initrd=initrd.gz DEBCONF_DEBUG=5 debconf/priority=critical url=http://10.189.84.1/cgi-bin/preseed.cgi locale=en_US console-setup/ask_detect=false console-setup/layoutcode=us hw-detect/start_pcmcia=false netcfg/choose_interface=eth0 console=ttyS0,115200n8 BOOT_IMAGE=linux
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009c800 (usable)
[ 0.000000] BIOS-e820: 000000000009c800 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000ce000 - 00000000000d4000 (reserved)
[ 0.000000] BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 00000000dfe70000 (usable)
[ 0.000000] BIOS-e820: 00000000dfe70000 - 00000000dfe80000 (ACPI data)
[ 0.000000] BIOS-e820: 00000000dfe80000 - 00000000dfe81000 (ACPI NVS)
[ 0.000000] BIOS-e820: 00000000dfe81000 - 00000000e0000000 (reserved)
[ 0.000000] BIOS-e820: 00000000f0000000 - 00000000f8000000 (reserved)
[ 0.000000] BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
[ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[ 0.000000] BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved)
[ 0.000000] BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] DMI present.
[ 0.000000] No AGP bridge found
[ 0.000000] last_pfn = 0x120000 max_arch_pfn = 0x400000000
[ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[ 0.000000] last_pfn = 0xdfe70 max_arch_pfn = 0x400000000
[ 0.000000] found SMP MP-table at [ffff8800000f6550] f6550
[ 0.000000] Scanning 0 areas for low memory corruption
[ 0.000000] init_memory_mapping: 0000000000000000-00000000dfe70000
[ 0.000000] init_memory_mapping: 0000000100000000-0000000120000000
[ 0.000000] RAMDISK: 7f73c000 - 7ffff000
[ 0.000000] ACPI: RSDP 00000000000f64a0 00024 (v02 PTLTD )
[ 0.000000] ACPI: XSDT 00000000dfe7756a 0009C (v01 IBM EXA01ZEU 06040000 LTP 00000000)
[ 0.000000] ACPI: FACP 00000000dfe7fc76 000F4 (v03 INTEL 06040000 PTL 00000002)
[ 0.000000] ACPI: DSDT 00000000dfe7a707 054FB (v01 INTEL BIGBY 06040000 MSFT 0100000E)
[ 0.000000] ACPI: FACS 00000000dfe80fc0 00040
[ 0.000000] ACPI: SLIC 00000000dfe7fd6a 00176 (v01 IBM EXA01ZEU 06040000 PTL 00000000)
[ 0.000000] ACPI: MCFG 00000000dfe7fee0 0003C (v01 PTLTD MCFG 06040000 LTP 00000000)
[ 0.000000] ACPI: HPET 00000000dfe7ff1c 00038 (v01 PTLTD HPETTBL 06040000 LTP 00000001)
[ 0.000000] ACPI: APIC 00000000dfe7ff54 00084 (v01 PTLTD ? APIC 06040000 LTP 00000000)
[ 0.000000] ACPI: BOOT 00000000dfe7ffd8 00028 (v01 PTLTD $SBFTBL$ 06040000 LTP 00000001)
[ 0.000000] ACPI: SSDT 00000000dfe78e79 0025F (v01 PmRe...

[Jeff Lane  (bladernr)]

After further investigation, this issue seems to be with the natty server ISO images only. I was able so successfully netboot and install three different servers using the Maverick server IS0, where those same three always resulted in the above mentioned kernel panic when using the Natty ISO images (both alternate and server).

[Andy Whitcroft (apw)]

@Jeff -- I see this is tested using the -10 kernel, any chance you could test the latest -11 kernel, I think this would be on the latest ISOs as of the 23rd or so. Also does this occur on the non-server images, perhaps you could test that also?

[Andy Whitcroft (apw)]

Actually I note that this panic is implying that init exited, which it is not allowed to do. I wonder if this is an initramfs or upstart issue.

[C de-Avillez (hggdh2)]

This may be a side effect of bug 694772.

[Andy Whitcroft (apw)]

It would probabally make sense to hold off any testing on this one until the bug 694772 is fixed and integrated as that would trigger such a failure.

[Jeff Lane  (bladernr)]

Andy: not sure if that bug is really affecting or not (we're not even getting close). I'm wondering if this is due to PXE.

Here's a screenshot from an attempt today though run on a system in the Boston DC using the -11 kernel, same panic. This is using the ISO for server amd64 as of 5 Jan.

[Andy Whitcroft (apw)]

On Wed, Jan 05, 2011 at 03:32:56PM -0000, Jeff Lane wrote:
> Andy: not sure if that bug is really affecting or not (we're not even
> getting close). I'm wondering if this is due to PXE.

Not even getting close to what? The last few messages are the kernel
freeing unused memory, and then a mknod error. These freeing messages are
the very last thing the kernel says before envoking the first userspace
program. This would typically be the /sbin/init in the initramfs.
The mknod error is not coming from the kernel, that is executing in
userspace very likely out of that /sbin/init busybox script.

I would conjecture that this indicates the kernel did indeed start
'init' and the script therein died leading to the kernel panic.

-apw

[Jeff Lane  (bladernr)]

So it turns out this is NOT a kernel issue after all... or more like it's one that we generated... this was the result of changes made to the datacenter setup to fix a different problem regarding how ethernet devices are addressed.

The fix for that issue caused this one. As soon as that fix was reverted, the kernel panic issue immediately disappeared from all systems in the DCs

So I've marked this as invalid. Yay, no work necessary!