BUG ON crash probably due to unionfs when installing sudo update with dpkg

[Loïc Minier]

Hi,

while upgrading to a newer sudo from hardy-proposed under UME, dpkg crashed with SEGV when installing the package, just after unpack.

dmesg had this to say:
...
[ 62.899427] b44: eth0: Flow control is off for TX and off for RX.
[ 62.902382] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 65.760507] NET: Registered protocol family 17
[ 125.486423] ------------[ cut here ]------------
[ 125.486437] kernel BUG at /build/buildd/linux-2.6.24/debian/build/custom-source-lpiacompat/fs/attr.c:138!
[ 125.486444] invalid opcode: 0000 [#1] SMP
[ 125.486449] Modules linked in: af_packet rfcomm l2cap i915 drm ipv6 sbs sbshc container dock joydev hci_usb bluetooth usbtouchscreen wlan_scan_sta ath_rate_sample snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_hwdep snd_seq_dummy video output ath_pci serio_raw wlan ath_hal(P) battery ac button snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device intel_agp agpgart pcspkr evdev shpchp snd pci_hotplug iTCO_wdt iTCO_vendor_support soundcore psmouse sg ata_piix pata_acpi ata_generic b44 ssb mii libata thermal thermal_sysfs processor fan fan_sysfs fbcon tileblit font bitblit softcursor ext3 ext2 jbd mbcache loop nls_iso8859_1 nls_cp437 vfat fat squashfs unionfs usb_storage libusual sd_mod scsi_mod ide_core uhci_hcd ehci_hcd usbhid hid usbcore
[ 125.486555]
[ 125.486561] Pid: 5518, comm: dpkg Tainted: P (2.6.24-12-lpiacompat #1)
[ 125.486567] EIP: 0060:[<c01a1dcb>] EFLAGS: 00010202 CPU: 0
[ 125.486578] EIP is at fnotify_change+0x29b/0x3a0
[ 125.486583] EAX: 481881a8 EBX: f071df10 ECX: 00001847 EDX: 00000000
[ 125.486589] ESI: e99d2650 EDI: 00001847 EBP: e99d2650 ESP: f071de58
[ 125.486595] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 125.486601] Process dpkg (pid: 5518, ti=f071c000 task=f074d540 task.ti=f071c000)
[ 125.486605] Stack: e99a2db0 c0196b3b 00000000 e996fc38 f071df5c f07789ed 00000001 00000000
[ 125.486618] e9968110 00000000 e99d2650 e996fc38 00000000 c01a1edf 00000000 f88cd3eb
[ 125.486629] f0772009 df88bf60 df88bf00 f79f67f8 df849180 f071df10 e996fee0 e99a2db0
[ 125.486641] Call Trace:
[ 125.486648] [<c0196b3b>] __link_path_walk+0xaab/0xe10
[ 125.486671] [<c01a1edf>] notify_change+0xf/0x20
[ 125.486680] [<f88cd3eb>] unionfs_setattr+0x27b/0x2b0 [unionfs]
[ 125.486713] [<c01a1ea7>] fnotify_change+0x377/0x3a0
[ 125.486733] [<c018b7c8>] chown_common+0xd8/0xf0
[ 125.486748] [<c0195e8a>] getname+0xaa/0xe0
[ 125.486764] [<c018b98b>] sys_chown+0x4b/0x70
[ 125.486789] [<c0197d3f>] sys_link+0x2f/0x40
[ 125.486801] [<c01053c2>] sysenter_past_esp+0x6b/0xa9
[ 125.486829] =======================
[ 125.486832] Code: 8b 54 24 0c 8b 42 0c 83 e8 80 e8 21 4d 17 00 e9 35 fe ff ff 8b 3b 83 cf 01 89 3b 0f b7 45 6a 80 e4 f7 66 89 43 04 e9 e4 fd ff ff <0f> 0b eb fe 90 81 23 ff bf ff ff 8b 44 24 0c e8 51 f8 03 00 85
[ 125.486887] EIP: [<c01a1dcb>] fnotify_change+0x29b/0x3a0 SS:ESP 0068:f071de58
[ 125.486903] ---[ end trace ebcf70505253c19f ]---

Let me know what additional info I can provide.

This is a serious bug if we can't install sudo (security updates) in the next release... UME uses only unionfs.

Bye,

[Oliver Grawert (ogra)]

i see the same failure on a classmate PC image

[Oliver Grawert (ogra)]

to reproduce use a system with unionfs underneath (liveCD might suffice) , enable the hardy-proposed repo and install the sudo package from there (it ships two hardlinks which i suspect to be the cause here)

[Loïc Minier (lool)]

On Wed, Apr 30, 2008, Oliver Grawert wrote:
> to reproduce use a system with unionfs underneath (liveCD might suffice)
> , enable the hardy-proposed repo and install the sudo package from there
> (it ships two hardlinks which i suspect to be the cause here)

 (Just reinstalling the current sudo version works too.)

--
Loïc Minier

[Colin Ian King (colin-king)]

stracing the installation of sudo package, trips the bug on a chown() call. From the strace'd code I've got a stripped down minimal C program below reliably breaks it the same way. Definitely due to the hard linked file and chown32 system call on the file on unionfs. Got some hard evidence now. C code below:

#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

void main(void)
{
        int fd;

        fd = open("/usr/bin/sudoedit.dpkg-new", O_WRONLY|O_CREAT|O_EXCL, 0);
        fchown(fd, 0, 0);
        fchmod(fd, 04755);
        close(fd);
        link("/usr/bin/sudoedit", "/usr/bin/sudoedit.dpkg-tmp");
        rename("/usr/bin/sudoedit.dpkg-new", "/usr/bin/sudoedit");
        link("/./usr/bin/sudoedit", "/usr/bin/sudo.dpkg-new");
        chown("/usr/bin/sudo.dpkg-new", 0, 0);
}

Incidentally, removing the fchown and fchmod from above and the bug goes away.

[Tim Gardner (timg-tpi)]

SRU Justification:

Impact: Unionfs cause kernel crash

Fix Description: This fixes a problem with chown'ing a file which has the suid bit set and has more than one hardlink on a read-only file system. The stacked nature of the unionfs means that the attribute changes pass through the lower level attr fs routines with the ATTR_MODE flag set when it's already been handled before in a stacked operation, so clear the ATTR_MODE flag. (This is the same fix as in the stacked ecrypt fs.)

Patch: http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-hardy-lum.git;a=commit;h=40b5f73849e523d7385d5e6272037eeaddef4ee2

TEST CASE: See https://bugs.edge.launchpad.net/ubuntu/+source/linux-ubuntu-modules-2.6.24/+bug/224754/comments/4

[Colin Ian King (colin-king)]

Fix with made, I've put the the module into a Hardy LiveCD initrd and given it a test in VMWare - I managed to install sudo from the hardy-proposed repo without the segfault or the kernel Oops. The bug was due to the stacked nature of the unionfs with 2 or more hardlinks when dealing with a suid file attribute being modified by a chown32() system call and the attribute handling code in unionfs not clearing the ATTR_MODE bit appropriately. Nice corner case.

[Colin Watson (cjwatson)]

Accepted into hardy-proposed.

[Steve Langasek (vorlon)]

This is listed as verification-needed since the beginning of May. Loïc, are you able to verify that the updated linux-ubuntu-modules in hardy-proposed (now also hardy-security) addresses this issue?

[Loïc Minier (lool)]

Yes, this is fixed since a while now, I forgot to confirm here. I think the packages moved forward already.

[Steve Langasek (vorlon)]

Confirmed, the relevant package is now in hardy-updates.

[Steve Langasek (vorlon)]

lum-2.6.24 is not present in intrepid. If this bug applies to intrepid, please open a separate task on the relevant package.