chown of SUID executable in docker container on overlayfs fails with kernel BUG at linux-4.4.0/fs/attr.c:280

[Vladimir Rutsky]

Steps to reproduce:

1. Take any fresh installation of Ubuntu 16.04. I used Vagrant to reproduce this bug, but it also reproduces on my VM with Ubuntu 16.04 in Azure.

2. Upgrade kernel to current latest release (linux-image-4.4.0-36-generic).

    # uname -r
    4.4.0-36-generic

3. Install Docker 1.11.2 from Ubuntu repositories (also can be reproduced with Docker 1.12.1 from official Docker repositories for Debian/Ubuntu):

    # apt install docker.io

4. Use overlayfs as storage driver in Docker. Edit /etc/default/docker and add DOCKER_OPTS="--storage-driver=overlay":

# echo 'DOCKER_OPTS="--storage-driver=overlay"' >> /etc/default/docker
# systemctl restart docker
# docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-36-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.5 MiB
Name: vagrant
ID: COJW:JDNB:4KBK:VJJN:PDW4:ECVU:6TCT:BAEY:5Z4T:WYGD:Q5BD:PZHH
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

5. Start container and run following commands in container:

# docker run --rm -ti busybox:latest /bin/sh
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
8ddc19f16526: Pull complete
Digest: sha256:a59906e33509d14c036c8678d687bd4eec81ed7c4b8ce907b888c607f6a1e0e6
Status: Downloaded newer image for busybox:latest
/ # touch a
/ # chmod 04744 a
/ # stat a
  File: a
  Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: fc00h/64512d Inode: 264640 Links: 1
Access: (4744/-rwsr--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2016-09-09 19:18:50.000000000
Modify: 2016-09-09 19:18:50.000000000
Change: 2016-09-09 19:18:56.000000000

/ # chown 0:12345 a
Segmentation fault
/ #

During chown fault following appears in dmesg:

[ 753.808988] ------------[ cut here ]------------
[ 753.809003] kernel BUG at /build/linux-a2WvEb/linux-4.4.0/fs/attr.c:280!
[ 753.809016] invalid opcode: 0000 [#1] SMP
[ 753.809026] Modules linked in: overlay veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_na
t nf_conntrack br_netfilter bridge stp llc aufs vboxsf ppdev crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd input_leds serio_raw vboxvideo 8250_fintek parpo
rt_pc parport ttm drm_kms_helper mac_hid drm fb_sys_fops i2c_piix4 syscopyarea vboxguest sysfillrect sysimgblt sunrpc autofs4 psmouse ahci libahci e1000 pata_acpi video fjes
[ 753.809172] CPU: 0 PID: 5971 Comm: chown Tainted: G W 4.4.0-36-generic #55-Ubuntu
[ 753.809188] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 753.809203] task: ffff88001f042c40 ti: ffff880010c74000 task.ti: ffff880010c74000
[ 753.809217] RIP: 0010:[<ffffffff8122a3f3>] [<ffffffff8122a3f3>] notify_change+0x303/0x360
[ 753.809258] RSP: 0018:ffff880010c77db0 EFLAGS: 00010202
[ 753.809270] RAX: 0000000057d30b2d RBX: 0000000000001847 RCX: 0000000000000017
[ 753.809297] RDX: 000000000771653f RSI: 000000000771653f RDI: 0000000057d30b2d
[ 753.809312] RBP: ffff880010c77de0 R08: 0000000000000000 R09: 0000000000000001
[ 753.809332] R10: 0000000000000000 R11: ffff880017582a0c R12: ffff880010c77e78
[ 753.809352] R13: ffff8800194f7cc0 R14: 00000000000089e4 R15: ffff880016a77b88
[ 753.809389] FS: 00000000011991f0(0063) GS:ffff88001fc00000(0000) knlGS:0000000000000000
[ 753.809420] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 753.809431] CR2: 000000000119abf8 CR3: 0000000017cbe000 CR4: 00000000000406f0
[ 753.809446] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 753.809461] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 753.809491] Stack:
[ 753.809496] 0000000000000000 0000000000000000 ffff880010c77e78 ffff880012299e40
[ 753.809517] ffff8800194f7cc0 ffff880019ed46a8 ffff880010c77e10 ffffffffc03573d1
[ 753.809552] 0000000000001847 ffff880010c77e78 ffff880012299e40 0000000000000000
[ 753.809585] Call Trace:
[ 753.809596] [<ffffffffc03573d1>] ovl_setattr+0x81/0xc0 [overlay]
[ 753.809612] [<ffffffff8122a325>] notify_change+0x235/0x360
[ 753.809626] [<ffffffff8120a83b>] chown_common+0x18b/0x1e0
[ 753.809660] [<ffffffff8120bc6d>] SyS_chown+0x9d/0xe0
[ 753.809674] [<ffffffff8182dfb2>] entry_SYSCALL_64_fastpath+0x16/0x71
[ 753.810211] Code: 4c 89 ef e8 60 87 17 00 31 c0 e9 00 fe ff ff 83 ca 01 41 89 14 24 89 d3 41 0f b7 07 e9 23 fe ff ff b8 ff ff ff ff e9 e4 fd ff ff <0f> 0b 48 3b 50 30 0f 85 50 fe ff ff e9 08 ff ff ff 4c
89 e6 4c
[ 753.811863] RIP [<ffffffff8122a3f3>] notify_change+0x303/0x360
[ 753.812355] RSP <ffff880010c77db0>
[ 753.812839] fbcon_switch: detected unhandled fb_set_par error, error code -16
[ 753.813741] fbcon_switch: detected unhandled fb_set_par error, error code -16
[ 753.814663] ---[ end trace 4d5ff9f2f68c4235 ]---

This bug is not reproduced in linux-image-4.4.0-34-generic.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-36-generic 4.4.0-36.55
ProcVersionSignature: Ubuntu 4.4.0-36.55-generic 4.4.16
Uname: Linux 4.4.0-36-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 9 19:06 seq
 crw-rw---- 1 root audio 116, 33 Sep 9 19:06 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Fri Sep 9 19:22:32 2016
HibernationDevice: RESUME=/dev/mapper/vagrant--vg-swap_1
InstallationDate: Installed on 2016-08-01 (39 days ago)
InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: innotek GmbH VirtualBox
PciMultimedia:

ProcFB: 0 vboxdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-36-generic root=/dev/mapper/username--vg-root ro quiet
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-36-generic N/A
 linux-backports-modules-4.4.0-36-generic N/A
 linux-firmware 1.157.3
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/01/2006
dmi.bios.vendor: innotek GmbH
dmi.bios.version: VirtualBox
dmi.board.name: VirtualBox
dmi.board.vendor: Oracle Corporation
dmi.board.version: 1.2
dmi.chassis.type: 1
dmi.chassis.vendor: Oracle Corporation
dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr:
dmi.product.name: VirtualBox
dmi.product.version: 1.2
dmi.sys.vendor: innotek GmbH

[Tim Gardner (timg-tpi)]

Vladimir - please try the kernel in proposed, Ubuntu-4.4.0-38.57. It has a patch that likely addresses your bug: "UBUNTU: SAUCE: overlayfs: fix regression in whiteout detection".

[Vladimir Rutsky (rutsky-vladimir)]

Same result with kernel from Yakkety (linux-headers-4.4.0-9136-generic):

[ 57.178253] ------------[ cut here ]------------
[ 57.178269] kernel BUG at /build/linux-rTsl6N/linux-4.4.0/fs/attr.c:280!
[ 57.178297] invalid opcode: 0000 [#1] SMP
[ 57.178321] Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc overlay vboxsf(OE) ppdev crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd input_leds serio_raw i2c_piix4 parport_pc parport vboxvideo(OE) 8250_fintek vboxguest(OE) ttm drm_kms_helper mac_hid drm fb_sys_fops syscopyarea sysfillrect sysimgblt sunrpc autofs4 psmouse ahci libahci e1000 pata_acpi fjes video
[ 57.178467] CPU: 0 PID: 2311 Comm: chown Tainted: G W OE 4.4.0-9136-generic #55-Ubuntu
[ 57.178483] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 57.178498] task: ffff88001a75ac40 ti: ffff88001be98000 task.ti: ffff88001be98000
[ 57.178512] RIP: 0010:[<ffffffff8122a3f3>] [<ffffffff8122a3f3>] notify_change+0x303/0x360
[ 57.178536] RSP: 0018:ffff88001be9bdb0 EFLAGS: 00010202
[ 57.178547] RAX: 0000000057d31402 RBX: 0000000000001847 RCX: 0000000000000017
[ 57.178559] RDX: 00000000072eb4fc RSI: 00000000072eb4fc RDI: 0000000057d31402
[ 57.178572] RBP: ffff88001be9bde0 R08: 0000000000000000 R09: 0000000000000001
[ 57.178585] R10: 0000000000000000 R11: ffff88001ec4320c R12: ffff88001be9be78
[ 57.178598] R13: ffff880019ce0e40 R14: 00000000000089e4 R15: ffff880019d14948
[ 57.178612] FS: 0000000000f241f0(0063) GS:ffff88001fc00000(0000) knlGS:0000000000000000
[ 57.178626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 57.178637] CR2: 0000000000f25bf8 CR3: 000000001745a000 CR4: 00000000000406f0
[ 57.178652] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 57.178665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 57.178677] Stack:
[ 57.178698] 0000000000000000 0000000000000000 ffff88001be9be78 ffff880019ce0f00
[ 57.178716] ffff880019ce0e40 ffff88001f1bd3f8 ffff88001be9be10 ffffffffc02953d1
[ 57.178747] 0000000000001847 ffff88001be9be78 ffff880019ce0f00 0000000000000000
[ 57.178764] Call Trace:
[ 57.178773] [<ffffffffc02953d1>] ovl_setattr+0x81/0xc0 [overlay]
[ 57.178786] [<ffffffff8122a325>] notify_change+0x235/0x360
[ 57.178810] [<ffffffff8120a83b>] chown_common+0x18b/0x1e0
[ 57.178851] [<ffffffff8120bc6d>] SyS_chown+0x9d/0xe0
[ 57.178862] [<ffffffff8182dff2>] entry_SYSCALL_64_fastpath+0x16/0x71
[ 57.179294] Code: 4c 89 ef e8 90 87 17 00 31 c0 e9 00 fe ff ff 83 ca 01 41 89 14 24 89 d3 41 0f b7 07 e9 23 fe ff ff b8 ff ff ff ff e9 e4 fd ff ff <0f> 0b 48 3b 50 30 0f 85 50 fe ff ff e9 08 ff ff ff 4c 89 e6 4c
[ 57.180459] RIP [<ffffffff8122a3f3>] notify_change+0x303/0x360
[ 57.180828] RSP <ffff88001be9bdb0>
[ 57.181247] ---[ end trace 4d5ff9f2f68c4235 ]---

[Brad Figg (brad-figg)]

This change was made by a bot.

[Vladimir Rutsky (rutsky)]

Tim, thank you for the quick response!

I tried linux-image-4.4.0-38-generic package (version 4.4.0-38.57) from xenial-proposed and can't reproduce this bug on it, looks like it is fixed in proposed version.

Thanks for the help, this issue should be resolved when linux-image-4.4.0-38-generic will reach xenial-updates channel.