Segfaults during boot (from mount)

[Robert Vollmert]

Binary package hint: initramfs-tools

After upgrading to gutsy, I see segmentation faults during boot-up.
From what I've been able to find out by scrolling back the console
when booting in recovery mode, these occurs when the script
/scripts/init-bottom is executed. That's why I'm guessing it's a bug
in initramfs-tools.

kern.log contains the following:
Aug 12 12:17:16 krikkit kernel: [ 22.413222] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Aug 12 12:17:16 krikkit kernel: [ 22.441388] Probing IDE interface ide1...
Aug 12 12:17:16 krikkit kernel: [ 22.461772] ReiserFS: dm-0: Using r5 hash to sort names
Aug 12 12:17:16 krikkit kernel: [ 22.482226] exe[2669]: segfault at 0000000000000000 rip 0000000000424b07 rsp 00007fffe667cd08 error 4
Aug 12 12:17:16 krikkit kernel: [ 22.483101] exe[2672]: segfault at 0000000000000000 rip 0000000000424b07 rsp 00007fff4b458ae8 error 4
Aug 12 12:17:16 krikkit kernel: [ 22.483710] exe[2673]: segfault at 0000000000000000 rip 0000000000424b07 rsp 00007fffac0dc768 error 4
Aug 12 12:17:16 krikkit kernel: [ 22.551481] ieee1394: Host added: ID:BUS[0-00:1023] GUID[0011d8000155a1e5]
Aug 12 12:17:16 krikkit kernel: [ 23.178714] hdc: HL-DT-ST RW/DVD GCC-4480B, ATAPI CD/DVD-ROM drive
Aug 12 12:17:16 krikkit kernel: [ 23.854061] ide1 at 0x170-0x177,0x376 on irq 15

on a non-recovery boot, the last last lines of the console are:
kinit: no resume image, doing normal boot ...
Segmentation fault
Segmentation fault
Segmentation fault

I have executed update-initramfs -u with no result.

[Matti Lindell (mlind)]

kernel: 2.6.22-9-generic
initramfs-tools: 0.85eubuntu16
arch: i386

I'm having similar/same segfaults when init-bottom script is executed during boot.
Boot messages contain:

"
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Segmentation fault
Done.
Segmentation fault
Segmentation fault
"

[Peter L Jones (peter-drealm)]

These are happening here:
/scripts/init-bottom/udev
> mount -n -o move /dev /root/dev

/init
> mount -n -o move /sys /root/sys
> Segmentation fault
> mount -n -o move /proc /root/proc
> Segmentation fault

[Mark Junker (mjscod)]

I have the same / a similar problem:

> [ 62.691534] kjournald starting. Commit interval 5 seconds
> [ 62.691590] EXT3-fs: sdb1: orphan cleanup on readonly fs
> [ 62.691632] ext3_orphan_cleanup: deleting unreferenced inode 16239956
> [ 66.954441] EXT3-fs: sdb1: 1 orphan inode deleted
> [ 66.954483] EXT3-fs: recovery complete.
> [ 67.271500] EXT3-fs: mounted filesystem with ordered data mode.
> [ 67.346686] exe[2876]: segfault at 0000000000000000 rip 0000000000424b07 rsp 00007fffd5e684c8 error 4
> [ 67.350129] exe[2888]: segfault at 0000000000000000 rip 0000000000424b07 rsp 00007fffd6a880f8 error 4
> [ 67.350506] exe[2889]: segfault at 0000000000000000 rip 0000000000424b07 rsp 00007fff33a05068 error 4
> [ 81.567662] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> [ 81.614512] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> [ 81.688144] iTCO_vendor_support: vendor-support=0
> [ 81.689568] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.01 (21-Jan-2007)
> [ 81.689679] iTCO_wdt: Found a ICH8 or ICH8R TCO device (Version=2, TCOBASE=0x0460)

It happens after it shows "running scripts init-bottom" (or something like that).

[David Gibson (dwg)]

I'm also observing this problem. Here, with a root-on-lvm-on-raid1 setup, these failures confuse the udev initialization sufficiently that although the system boots, it fails to mount any filesystems other than root, and things are generally confused later on.

This looks like the same problem as bug #132120

[Robert Vollmert (rvollmert)]

I should note that I have root on lvm also, but I don't have any further problems. I don't have any other filesystems mounted automatically, though.

Also, /var/crash doesn't seem to contain anything relevant (kvm, gnome-panel_sensors-applet, apport-gtk).

[David Gibson (dwg)]

As bug #132120 suggests, I've verified that just replacing the busybox binary (both as /bin/busybox and /bin/sh on the initramfs) with an older, working version is sufficient to fix the problem.

Replacing the failing calls to mount with explicit calls to /bin/mount - thereby invoking the klibc version of mount, rather than the busybox version of mount - also fixes the problem (well as long as the -n flag to mount is also removed, since it is not recognized by the klibc version).

Thus, it looks like there's definitely a bug in busybox - although I don't know if initramfs-tools was heading towards removing use of busybox in favour of the klibc binaries, which might make it moot.

[David Gibson (dwg)]

In reference to Robert Vollmert's comment: not having any other filesystems would explain why you don't get further symptoms.

For me, this is a serious bug. In fact, contrary to my previous comment, the system won't really boot after this - instead it will give the "give root password for maintenance" prompt when it fails to fsck the non-root filesystems (/home, etc..) because it can't find the device nodes. In fact, the /dev/dm-* devices do exist, but the links from /dev/vgname/lvname (and the various other variants) do not. It's possible Ctrl-D that prompt at which point the system will boot up (to X even) - but it's not very useful without /home.

[Robert Vollmert (rvollmert)]

I can confirm that this is busybox. I get segfaults when calling 'busybox mount -o move ...' for both /usr/lib/initramfs-tools/bin/busybox and /bin/busybox (from the plain busybox package as well as busybox-static).

[Evan (ev)]

As eluded to, this appears to be a regression. Busybox mount -o bind works fine in Feisty.

[Flávio Martins (flavioxmartins)]

Well, i gave a it a shot. Seems that both the scripts/init-bottom/udev script and the two subsequent mounts cause the segmentation faults. 3 in total.

Could be that the pkill call inside the udev script is also managing to segfault.

I'd set this to busybox.

[Matt Zimmerman (mdz)]

This is easily reproduced with, e.g.:

sudo /usr/lib/initramfs-tools/bin/busybox mount -o bind /tmp /mnt

I'm attaching an apport crash report, which includes the core file. I'm not sure whether the retracing service will pick it up automatically; if it doesn't, then hopefully someone who knows the right magic will trigger it

[Brian Murray (brian-murray)]

I submitted a new bug report [bug 132790], using the command provided by Matt, as I am not sure the retracing service will retrace a crash report attached to an existing bug.

[Martin Pool (mbp)]

fwiw I have lvm-on-luks, and multiple filesystems, and they seem to
all get mounted properly - but maybe they're getting mounted later on.

--
Martin

[David Gibson (dwg)]

Martin, but are you seeing the SEGVs during boot?

[Martin Pool (mbp)]

On 8/16/07, dwg <email address hidden> wrote:
> Martin, but are you seeing the SEGVs during boot?

Hi David,

Yes, I see the message during bootup, but it does not seem to have any
lasting ill effects. I think I filed one of the duplicate bugs.

--
Martin

[David Gibson (dwg)]

Ok. How are the devices for your secondary filesystems specified in /etc/fstab? For me at least, the problem seems to be that some of the convenience symlinks are not created, so depending on the names you use, it might work.

[Martin Pool (mbp)]

On 8/16/07, dwg <email address hidden> wrote:
> Ok. How are the devices for your secondary filesystems specified in
> /etc/fstab? For me at least, the problem seems to be that some of the
> convenience symlinks are not created, so depending on the names you use,
> it might work.

Like this.

--
Martin

[Martin Pool (mbp)]

Apparently Launchpad doesn't turn mail attachments into bug attachments? Let's try again.

[Martin Pool (mbp)]

And my /etc/crypttab has just

home /dev/disk/by-uuid/8ce2a7ce-13c6-4496-9302-2506bf89d7ba none luks,tries=5,loud

which is used as an LVM PV.

[David Gibson (dwg)]

Uh.. did you forget an attachment there?

[Allan MacKinnon (theforkofjustice)]

I have nothing new to add but I have the same bug.

[blazerw (randy-1702)]

I really enjoyed Allan's comment. So much so that I feel I need to say the same thing. 3 seg faults on boot that seem to cause me no harm whatsoever.

[Colin Watson (cjwatson)]

Sorry, this was my fault; I'd tested my external mount helper changes, but not in the particular scenario that reproduces this bug (use of the bind/move/remount options). Ben Collins sent me a patch, which I've uploaded.

[Colin Watson (cjwatson)]

busybox (1:1.1.3-5ubuntu4) gutsy; urgency=low

  * Fix mount segfault while doing bind/move mounts or remounts, introduced
    by my external mount helper change (thanks, Ben Collins; LP: #131961).

 -- Colin Watson <email address hidden> Sat, 18 Aug 2007 19:06:43 +0100

[Steve.K (stevekettell)]

Also occurs on 2.6.22-8

Unfortunately removed 22-7 prior to rebooting so cant say if that's affected too.

[Matt Zimmerman (mdz)]

This is not a kernel bug, and a fix has already been implemented. It should be available shortly, at which point you merely need to install available updates for Gutsy.

[David Gibson (dwg)]

The busybox fix works here.

[Juan Pablo Salazar Bertín (snifer)]

I've updated to busybox-initramfs package version 1:1.1.3-5ubuntu4, but I still see the 3 "Segmentation fault" messages in VT1.

[David Gibson (dwg)]

Have you used update-initramfs to rebuild the ramdisks once the new busybox was installed?

[Peter L Jones (peter-drealm)]

Confirmed fixed here. :)

[Robert Vollmert (rvollmert)]

Fixed here, too. Though automatically causing update-initramfs to run would have been nice.

[Juan Pablo Salazar Bertín (snifer)]

Fixed after using update-initramfs, but I think the update package procedure should have done that (or at least explicitly tell me what to do to complete the update), since I've never manually installed this package. Please consider this for future updates.

[Colin Watson (cjwatson)]

Unfortunately that's difficult; I tried to arrange for it in the past and it caused problems due to exactly where busybox-initramfs sits in the dependency graph. It shouldn't be a problem for upgrades from Feisty to Gutsy, and it should sort itself out naturally as other packages get uploaded.