Kernel Panic wrt btrfs while sbuild/schroot
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | Linux |
Fix Released
|
Medium
|
||
| | linux (Ubuntu) |
High
|
Colin Ian King | ||
Bug Description
I'm running ubuntu Wily amd64 on a MSI Ghost Pro laptop.
I'm running btrfs on top of a bcached dmraid setup(intel software raid0).
I can't use sbuild, cause it crashes.
I tried to use a schroot env, and when updating it I got it to crash also.
All the keyboard/mouse/X11 are stalled, but I could ssh to it.
Trying to restart lightdm service didn't help.
I've got a kernel trace in the dmesg:
[76610.550953] BUG: unable to handle kernel NULL pointer dereference at 0000000000000334
[76610.550983] IP: [<ffffffffc032f
[76610.551025] PGD 160aaf067 PUD 161531067 PMD 0
[76610.551039] Oops: 0002 [#1] SMP
[76610.551050] Modules linked in: dm_crypt algif_skcipher af_alg drbg ansi_cprng ctr ccm rfcomm ipt_REJECT nf_reject_ipv4 nvram msr xt_multiport ebtable_filter ebtables ip6table_filter ip6_tables overlay bnep uvcvideo btusb videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel v4l2_common bluetooth videodev media xt_addrtype xt_conntrack xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_
[76610.551253] snd_hda_intel iwlmvm glue_helper snd_hda_codec ablk_helper mac80211 cryptd snd_hda_core snd_hwdep snd_pcm snd_seq_midi joydev input_leds snd_seq_midi_event snd_rawmidi serio_raw iwlwifi lpc_ich mei_me snd_seq mei cfg80211 snd_seq_device snd_timer snd ie31200_edac edac_core soundcore shpchp mac_hid kvm_intel kvm parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq dm_mirror dm_region_hash dm_log uas usb_storage bcache hid_generic usbhid hid i915 i2c_algo_bit drm_kms_helper psmouse ahci drm alx libahci mdio wmi video
[76610.551395] CPU: 1 PID: 32090 Comm: dpkg Not tainted 4.2.0-23-generic #28-Ubuntu
[76610.551414] Hardware name: Micro-Star International Co., Ltd. GS60 2PE Ghost Pro/MS-16H2, BIOS E16H2IMS.112 05/05/2015
[76610.551438] task: ffff8803cf422940 ti: ffff8801a0be4000 task.ti: ffff8801a0be4000
[76610.551455] RIP: 0010:[<
[76610.551485] RSP: 0018:ffff8801a0
[76610.551497] RAX: ffff8803cf422940 RBX: ffff880273bff100 RCX: 0000000000000000
[76610.551513] RDX: 0000000080000000 RSI: 0000000000000000 RDI: ffff88040a2fb178
[76610.551530] RBP: ffff8801a0be7eb8 R08: 0000000000000001 R09: 0000000000000000
[76610.551546] R10: 00000000000001f6 R11: 0000000000000246 R12: ffff88040a2fb178
[76610.551563] R13: ffff88040a2fb0d0 R14: 0000000000000000 R15: 0000000000000000
[76610.551580] FS: 00007f3c27bca80
[76610.551598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[76610.551611] CR2: 0000000000000334 CR3: 000000016159a000 CR4: 00000000001406e0
[76610.551628] Stack:
[76610.551633] 800000033fffa865 000000000000000b 8000000000000000 ffff88008a657e40
[76610.551652] 7fffffffffffffff 0000000000000000 ffff8801a0be7ea8 ffff8804069544a0
[76610.551671] 0000800173bff000 0000000010b6e564 0000000000000000 0000000010b6e564
[76610.551691] Call Trace:
[76610.551703] [<ffffffff8122f
[76610.551718] [<ffffffff8122f
[76610.551732] [<ffffffff81230
[76610.551751] [<ffffffff817f0
[76610.551766] Code: 8b 0e 48 85 c9 75 e8 eb 9e 48 8b 45 90 4c 8b 75 98 4d 8d a5 a8 00 00 00 4c 89 e7 4c 29 f0 48 83 c0 01 48 89 45 80 e8 f4 e2 4b c1 <f0> 41 ff 87 34 03 00 00 49 8b 85 70 ff ff ff 48 c1 e8 07 83 e0
[76610.551845] RIP [<ffffffffc032f
[76610.551871] RSP <ffff8801a0be7e28>
[76610.551881] CR2: 0000000000000334
[76610.558028] ---[ end trace 33509d397c46cd18 ]---
ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: linux-image-
ProcVersionSign
Uname: Linux 4.2.0-23-generic x86_64
ApportVersion: 2.19.1-0ubuntu5
Architecture: amd64
Date: Fri Jan 8 11:02:06 2016
HibernationDevice: RESUME=
InstallationDate: Installed on 2015-11-24 (44 days ago)
InstallationMedia: Ubuntu 15.10 "Wily Werewolf" - Alpha amd64 (20151016)
MachineType: Micro-Star International Co., Ltd. GS60 2PE Ghost Pro
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
XDG_RUNTIME_
LANG=fr_FR.UTF-8
SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=
RelatedPackageV
linux-
linux-
linux-firmware 1.149.3
SourcePackage: linux
UdevLog: Error: [Errno 2] Aucun fichier ou dossier de ce type: '/var/log/udev'
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/05/2015
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: E16H2IMS.112
dmi.board.
dmi.board.name: MS-16H2
dmi.board.vendor: Micro-Star International Co., Ltd.
dmi.board.version: REV:0.B
dmi.chassis.
dmi.chassis.type: 10
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.
dmi.modalias: dmi:bvnAmerican
dmi.product.name: GS60 2PE Ghost Pro
dmi.product.
dmi.sys.vendor: Micro-Star International Co., Ltd.
I have the same problem although i am using overlay while being in a docker container. Host fs is btrfs on arch linux kernel 4.2.0, guest docker image ubuntu:wily.
kernel oops log can be found here
https:/
| Kick In (kick-d) wrote : | #1 |
This change was made by a bot.
| Changed in linux (Ubuntu): | |
| status: | New → Confirmed |
| Joseph Salisbury (jsalisbury) wrote : | #3 |
Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?
Would it be possible for you to test the latest upstream kernel? Refer to https:/
If this bug is fixed in the mainline kernel, please add the following tag 'kernel-
If the mainline kernel does not fix this bug, please add the tag: 'kernel-
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".
Thanks in advance.
[0] http://
| Changed in linux (Ubuntu): | |
| importance: | Undecided → High |
| status: | Confirmed → Incomplete |
| Kick In (kick-d) wrote : | #4 |
I tried with the upstream v4.4-wily kernel.
kick@kick-
Linux kick-gs60 4.4.0-040400-
I couldn't re-use my previous schroot (couldn't mount with overlayfs). I've created a new one with --type=
My latop didn't freeze, but I still got traces in the syslog:
[ 458.091294] BUG: unable to handle kernel NULL pointer dereference at 0000000000000334
[ 458.091320] IP: [<ffffffffc0346
[ 458.091363] PGD 3e8de9067 PUD 3e8de8067 PMD 0
[ 458.091374] Oops: 0002 [#1] SMP
[ 458.091383] Modules linked in: overlay drbg ansi_cprng ctr ccm rfcomm nvram msr xt_multiport ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables xt_addrtype xt_conntrack xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_
[ 458.091575] cryptd snd_hda_intel input_leds snd_hda_codec iwlwifi snd_hda_core snd_hwdep joydev snd_seq_midi snd_seq_midi_event serio_raw snd_pcm snd_rawmidi snd_seq snd_seq_device snd_timer lpc_ich cfg80211 mei_me snd mei soundcore ie31200_edac shpchp edac_core mac_hid kvm_intel kvm irqbypass parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq dm_mirror dm_region_hash dm_log bcache hid_generic usbhid hid i915 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops psmouse drm ahci alx libahci mdio wmi video fjes
[ 458.091711] CPU: 1 PID: 5615 Comm: debconf-set-sel Not tainted 4.4.0-040400-
[ 458.091731] Hardware name: Micro-Star International Co., Ltd. GS60 2PE Ghost Pro/MS-16H2, BIOS E16H2IMS.112 05/05/2015
[ 458.091755] task: ffff8803e1013700 ti: ffff8803e8d38000 task.ti: ffff8803e8d38000
[ 458.091771] RIP: 0010:[<
[ 458.091804] RSP: 0018:ffff8803e8
[ 458.091815] RAX: ffff8803e1013700 RBX: ffff8803af242700 RCX: 0000000000000000
[ 458.091831] RDX: 0000000080000000 RSI: 0000000000000000 RDI: ffff880409d9c988
[ 458.091846] RBP: ffff8803e8d3bec8 R08: 0000000000000001 R09: 00000000021f5338
[ 458.091861] R10: 000000000000009c R11: 0000000000000246 R12: ffff880409d9c988
[ 458.091876] R13: ffff880409d9c8e0 R14: 0000000000000000 R15: 0000000000000000
[ 458.091892] FS: 00007f31a044470
[ 458.091910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 458.091922] CR2: 0000000000000334 CR3: 00000003e8f05000 CR4: 00000000001406e0
[ 458.091937] Stack:
[ 458.091942] 00000000027bf270 0000000100000000 8000...
| Changed in linux (Ubuntu): | |
| status: | Incomplete → Confirmed |
| tags: | added: kernel-bug-exists-upstream |
| Kick In (kick-d) wrote : | #5 |
output of lsblk:
http://
| tags: | added: kernel-key |
| Joseph Salisbury (jsalisbury) wrote : | #6 |
Did this issue start happening after an update/upgrade? Was there a kernel version where you were not having this particular problem? This will help determine if the problem you are seeing is the result of a regression, and when this regression was introduced. If this is a regression, we can perform a kernel bisect to identify the commit that introduced the problem.
| Kick In (kick-d) wrote : | #7 |
Same bug that I already had with another laptop, it seems to be related to overlay and btrfs:
https:/
Happens also on coreos:
https:/
Kernel bug:
https:/
| Changed in linux (Ubuntu): | |
| status: | Confirmed → In Progress |
| assignee: | nobody → Colin Ian King (colin-king) |
| Colin Ian King (colin-king) wrote : | #8 |
The failure occurs on an atomic_inc on root, when root is NULL
atomic_
I've debugged this a bit, the failure occurs on an atomic_inc on root, when root is NULL, cf:
atomic_
| Colin Ian King (colin-king) wrote : | #9 |
OK, bisectable, 3.13 good, 3.19 bad. Test is simple, build the following and run in a chroot on a btrfs system and it will trip the same bug.
#define _GNU_SOURCE
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
int main(void)
{
char buffer[1024] = { 0 };
int fd;
fd = open("test", O_CREAT | O_RDWR, 0777);
if (fd < 0)
fsync(fd);
return 0;
}
| Colin Ian King (colin-king) wrote : | #10 |
3.17 + a forward port of ubuntu utopic overlay fs - works fine, 3.18 with overlayfs has the problem, so now I've got a bisectable range to work on.
| Colin Ian King (colin-king) wrote : | #11 |
bisected: 4bacc9c9234c7c8
commit 4bacc9c9234c7c8
Author: David Howells <email address hidden>
Date: Thu Jun 18 14:32:31 2015 +0100
overlayfs: Make f_path always point to the overlay and f_inode to the underlay
Make file->f_path always point to the overlay dentry so that the path in
/proc/pid/fd is correct and to ensure that label-based LSMs have access to the
overlay as well as the underlay (path-based LSMs probably don't need it).
Using my union testsuite to set things up, before the patch I see:
[root@
[root@
...
lr-x------. 1 root root 64 Jun 5 14:38 5 -> /a/foo107
[root@
...
Device: 23h/35d Inode: 13381 Links: 1
...
[root@
...
Device: 23h/35d Inode: 13381 Links: 1
...
After the patch:
[root@
[root@
...
lr-x------. 1 root root 64 Jun 5 14:22 5 -> /mnt/a/foo107
[root@
...
Device: 23h/35d Inode: 40346 Links: 1
...
[root@
...
Device: 23h/35d Inode: 40346 Links: 1
...
Note the change in where /proc/$$/fd/5 points to in the ls command. It was
pointing to /a/foo107 (which doesn't exist) and now points to /mnt/a/foo107
(which is correct).
The inode accessed, however, is the lower layer. The union layer is on device
25h/37d and the upper layer on 24h/36d.
Signed-off-by: David Howells <email address hidden>
Signed-off-by: Al Viro <email address hidden>
bisected: 4bacc9c9234c7c8
commit 4bacc9c9234c7c8
Author: David Howells <email address hidden>
Date: Thu Jun 18 14:32:31 2015 +0100
overlayfs: Make f_path always point to the overlay and f_inode to the underlay
| tags: |
added: kernel-da-key removed: kernel-key |
| Colin Ian King (colin-king) wrote : | #12 |
I believe the follow may be the fix we require. I'll discuss this with upstream
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index 098bb8f..5e5df8b 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -1884,7 +1884,7 @@ static int start_ordered_
int btrfs_sync_
{
struct dentry *dentry = file->f_
- struct inode *inode = d_inode(dentry);
+ struct inode *inode = file_inode(file);
struct btrfs_root *root = BTRFS_I(
struct btrfs_trans_handle *trans;
struct btrfs_log_ctx ctx;
The following stops the issue. I'm not sure if it is the correct fix though.
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index 098bb8f..5e5df8b 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -1884,7 +1884,7 @@ static int start_ordered_
int btrfs_sync_
{
struct dentry *dentry = file->f_
- struct inode *inode = d_inode(dentry);
+ struct inode *inode = file_inode(file);
struct btrfs_root *root = BTRFS_I(
struct btrfs_trans_handle *trans;
struct btrfs_log_ctx ctx;
Commit 4bacc9c9234c7c8
always point to the overlay and f_inode to the underlay") resulted in an
issue when using a combination of btrfs and overlayfs. This is
noticeable when doing a fsync() on a file in a chroot with overlayfs on
top of btrfs; we hit a kernel oops in btrfs_sync_file() on
atomic_
I've debugged this further and found that in btrfs_sync_file():
struct inode *inode = d_inode(dentry);
does not return the inode I expected when using the stacked overlay fs,
where as:
struct inode *inode = file_inode(file);
does.
However, I'm not well at all well versed in btrfs, so I am not confident
this is a actually correct. Any comments?
| Colin Ian King (colin-king) wrote : | #13 |
https://<email address hidden>
| Colin Ian King (colin-king) wrote : | #14 |
I suggest a workaround for the moment, using aufs as the overlay to see if this helps.
Modify the chroot config and set the union-type to aufs:
union-type=aufs
you may see an error like:
"aufs au_xino_
this is caused by the fact that btrfs cannot store aufs xino files. aufs tries to store these files alongside the rw layer. If that fails, it tries to store them in /tmp/.aufs.xino, however, if this is btrfs it will fail. So the workaround for that is to mount /tmp as something other than btrfs, e.g. tmpfs
Thanks Colin!
2016-02-16 19:20 GMT+01:00 Colin Ian King <email address hidden>:
> I suggest a workaround for the moment, using aufs as the overlay to see
> if this helps.
>
> Modify the chroot config and set the union-type to aufs:
>
> union-type=aufs
>
> you may see an error like:
>
> "aufs au_xino_
> /tmp/.aufs.
>
> this is caused by the fact that btrfs cannot store aufs xino files.
> aufs tries to store these files alongside the rw layer. If that fails,
> it tries to store them in /tmp/.aufs.xino, however, if this is btrfs it
> will fail. So the workaround for that is to mount /tmp as something
> other than btrfs, e.g. tmpfs
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> Kernel Panic wrt btrfs while sbuild/schroot
>
> Status in Linux:
> Unknown
> Status in linux package in Ubuntu:
> In Progress
>
> Bug description:
> I'm running ubuntu Wily amd64 on a MSI Ghost Pro laptop.
>
> I'm running btrfs on top of a bcached dmraid setup(intel software
> raid0).
>
> I can't use sbuild, cause it crashes.
>
> I tried to use a schroot env, and when updating it I got it to crash
> also.
>
> All the keyboard/mouse/X11 are stalled, but I could ssh to it.
> Trying to restart lightdm service didn't help.
>
> I've got a kernel trace in the dmesg:
>
> [76610.550953] BUG: unable to handle kernel NULL pointer dereference at
> 0000000000000334
> [76610.550983] IP: [<ffffffffc032f
> [btrfs]
> [76610.551025] PGD 160aaf067 PUD 161531067 PMD 0
> [76610.551039] Oops: 0002 [#1] SMP
> [76610.551050] Modules linked in: dm_crypt algif_skcipher af_alg drbg
> ansi_cprng ctr ccm rfcomm ipt_REJECT nf_reject_ipv4 nvram msr xt_multiport
> ebtable_filter ebtables ip6table_filter ip6_tables overlay bnep uvcvideo
> btusb videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel
> v4l2_common bluetooth videodev media xt_addrtype xt_conntrack xt_CHECKSUM
> iptable_mangle ipt_MASQUERADE nf_nat_
> nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp
> bridge aufs stp llc iptable_filter ip_tables x_tables binfmt_misc arc4
> nls_iso8859_1 msi_wmi sparse_keymap intel_rapl iosf_mbi
> x86_pkg_
> crct10dif_pclmul snd_hda_
> aesni_intel aes_x86_64 lrw gf128mul
> [76610.551253] snd_hda_intel iwlmvm glue_helper snd_hda_codec
> ablk_helper mac80211 cryptd snd_hda_core snd_hwdep snd_pcm snd_seq_midi
> joydev input_leds snd_seq_midi_event snd_rawmidi serio_raw iwlwifi lpc_ich
> mei_me snd_seq mei cfg80211 snd_seq_device snd_timer snd ie31200_edac
> edac_core soundcore shpchp mac_hid kvm_intel kvm parport_pc ppdev lp
> parport autofs4 btrfs xor raid6_pq dm_mirror dm_region_hash dm_log uas
> usb_storage bcache hid_generic usbhid hid i915 i2c_algo_bit drm_kms_helper
> psmouse ahci drm alx libahci mdio wmi video
> [76610.551395] CPU: 1 PID: 32090 Comm: dpkg Not tainted 4.2.0-23-generic
> #28-Ubuntu
> [7...
| Colin Ian King (colin-king) wrote : | #16 |
Given that this is stuck waiting for upstream to figure out a way forward, which looks unlikely at the moment, plus we have a suitable workaround, I'm going to mark this as Won't Fix.
| Changed in linux (Ubuntu): | |
| status: | In Progress → Won't Fix |
Fixed by de17e793b104d690e1d "btrfs: fix crash/invalid memory access on fsync when using overlayfs", in 4.6.
| Changed in linux: | |
| importance: | Unknown → Medium |
| status: | Unknown → Fix Released |


Created attachment 183641
kernel oops log
How to reproduce: upper,work, overlay} lower,upperdir= upper,workdir= work overlay
$ mkdir {lower,
$ uname > lower/uname
# mount overlay -t overlay -o lowerdir=
$ # edit overlay/uname
Results:
Kernel oops, screen freezes.