bcache causes a kernel bug when adding a new cache device

[Jacky Wang]

Ubuntu 16.04 on virtualbox 5.0.20. VirtualBoxAddition is installed.

The issue happens on both out-of-box 4.4.0 kernel and the later-installed 4.4.11 kernel. 100% re-producible. The same issue applies to both virtual hard drive (/dev/sdb) and file-base block device (/dev/loop2) I created within Ubuntu.

$ sudo make-bcache -C /dev/sdb

The error shows (through dmesg):
[ 47.616016] ------------[ cut here ]------------
[ 47.616035] kernel BUG at /home/kernel/COD/linux/drivers/md/bcache/super.c:1822!
[ 47.616051] invalid opcode: 0000 [#1] SMP
[ 47.616063] Modules linked in: vboxsf(OE) ppdev vboxvideo(OE) ttm drm_kms_helper input_leds joydev drm serio_raw lpc_ich fb_sys_fops syscopyarea i2c_piix4 sysfillrect sysimgblt 8250_fintek parport_pc vboxguest(OE) parport mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 multipath linear bcache raid0 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse fjes ahci libahci pata_acpi video
[ 47.616255] CPU: 2 PID: 1247 Comm: bcache-register Tainted: G OE 4.4.11-040411-generic #201605182255
[ 47.616277] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 47.616295] task: ffff880035140000 ti: ffff880076840000 task.ti: ffff880076840000
[ 47.616311] RIP: 0010:[<ffffffffc0134086>] [<ffffffffc0134086>] cache_alloc.isra.20+0x646/0x670 [bcache]
[ 47.616340] RSP: 0018:ffff880076843d10 EFLAGS: 00010246
[ 47.616352] RAX: ffff880079528c60 RBX: ffff8800796a0000 RCX: 00000000000005d2
[ 47.616367] RDX: 00000000000005d1 RSI: 0000000000000000 RDI: 0000000000019d40
[ 47.616382] RBP: ffff880076843d28 R08: ffff88007fd19d40 R09: ffff88007d001d00
[ 47.616398] R10: ffff880079528c60 R11: ffff88007b590958 R12: 0000000000000000
[ 47.616413] R13: 0000000000000010 R14: ffff88007c3c4000 R15: ffff8800796a0000
[ 47.616428] FS: 00007ff8ffa10700(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
[ 47.616445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 47.616458] CR2: 00007ff8ff4c1d30 CR3: 0000000077f70000 CR4: 00000000000406e0
[ 47.616476] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 47.616491] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 47.616506] Stack:
[ 47.616511] ffffea0001e880c0 ffff88007cc28680 0000000000000000 ffff880076843dd0
[ 47.616531] ffffffffc01373e3 ffff880076843d58 ffffffff811c8de5 000000007fd61025
[ 47.616551] 00007ff8ff4cf000 ffff88007b590958 ffff8800796a0008 0000000000000009
[ 47.616570] Call Trace:
[ 47.616585] [<ffffffffc01373e3>] register_bcache+0x2b3/0x1140 [bcache]
[ 47.616626] [<ffffffff811c8de5>] ? page_add_file_rmap+0x25/0x60
[ 47.617074] [<ffffffff8118b7b9>] ? unlock_page+0x69/0x70
[ 47.617516] [<ffffffff813d25ef>] kobj_attr_store+0xf/0x20
[ 47.617946] [<ffffffff81289147>] sysfs_kf_write+0x37/0x40
[ 47.618353] [<ffffffff8128855d>] kernfs_fop_write+0x11d/0x170
[ 47.618755] [<ffffffff81209a27>] __vfs_write+0x37/0x110
[ 47.619149] [<ffffffff8137a238>] ? apparmor_file_permission+0x18/0x20
[ 47.619543] [<ffffffff8133aead>] ? security_file_permission+0x3d/0xc0
[ 47.619922] [<ffffffff810c8f12>] ? percpu_down_read+0x12/0x50
[ 47.620290] [<ffffffff8120a139>] vfs_write+0xa9/0x1a0
[ 47.620648] [<ffffffff8120adf5>] SyS_write+0x55/0xc0
[ 47.621021] [<ffffffff81808332>] entry_SYSCALL_64_fastpath+0x16/0x71
[ 47.621362] Code: 6a fd ff ff e9 4d fd ff ff be c0 00 40 02 4c 89 e7 e8 cf 61 0b c1 48 85 c0 48 89 83 f0 0a 00 00 0f 85 96 fd ff ff e9 79 fd ff ff <0f> 0b 0f 0b 0f 0b bf 40 00 00 00 e8 9a 90 09 c1 48 85 c0 48 89
[ 47.622456] RIP [<ffffffffc0134086>] cache_alloc.isra.20+0x646/0x670 [bcache]
[ 47.622812] RSP <ffff880076843d10>
[ 47.623177] ---[ end trace 19bcd02731ac9be1 ]---
---
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
DistroRelease: Ubuntu 16.04
InstallationDate: Installed on 2016-05-29 (0 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
Package: linux (not installed)
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
Tags: xenial
Uname: Linux 4.4.11-040411-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True
---
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
DistroRelease: Ubuntu 16.04
InstallationDate: Installed on 2016-05-29 (0 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
Package: linux (not installed)
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
Tags: xenial
Uname: Linux 4.4.11-040411-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1586838

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jacky Wang (jacky-chao-wang)]

apport information

[Jacky Wang (jacky-chao-wang)]

apport information

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.7-rc1 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.7-rc1-yakkety/

[Jacky Wang (jacky-chao-wang)]

I tried to install the latest kernel as following:

$ sudo dpkg -i *.deb
(Reading database ... 130671 files and directories currently installed.)
Preparing to unpack linux-headers-4.7.0-040700rc1_4.7.0-040700rc1.201605291331_all.deb ...
Unpacking linux-headers-4.7.0-040700rc1 (4.7.0-040700rc1.201605291331) over (4.7.0-040700rc1.201605291331) ...
Preparing to unpack linux-headers-4.7.0-040700rc1-generic_4.7.0-040700rc1.201605291331_amd64.deb ...
Unpacking linux-headers-4.7.0-040700rc1-generic (4.7.0-040700rc1.201605291331) over (4.7.0-040700rc1.201605291331) ...
Preparing to unpack linux-image-4.7.0-040700rc1-generic_4.7.0-040700rc1.201605291331_amd64.deb ...
Done.
Unpacking linux-image-4.7.0-040700rc1-generic (4.7.0-040700rc1.201605291331) over (4.7.0-040700rc1.201605291331) ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.7.0-040700rc1-generic /boot/vmlinuz-4.7.0-040700rc1-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.7.0-040700rc1-generic /boot/vmlinuz-4.7.0-040700rc1-generic
Setting up linux-headers-4.7.0-040700rc1 (4.7.0-040700rc1.201605291331) ...
Setting up linux-headers-4.7.0-040700rc1-generic (4.7.0-040700rc1.201605291331) ...
Setting up linux-image-4.7.0-040700rc1-generic (4.7.0-040700rc1.201605291331) ...
Running depmod.
update-initramfs: deferring update (hook will be called later)
Not updating initrd symbolic links since we are being updated/reinstalled
(4.7.0-040700rc1.201605291331 was configured last, according to dpkg)
Not updating image symbolic links since we are being updated/reinstalled
(4.7.0-040700rc1.201605291331 was configured last, according to dpkg)
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.7.0-040700rc1-generic /boot/vmlinuz-4.7.0-040700rc1-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.7.0-040700rc1-generic /boot/vmlinuz-4.7.0-040700rc1-generic
update-initramfs: Generating /boot/initrd.img-4.7.0-040700rc1-generic
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 4.7.0-040700rc1-generic /boot/vmlinuz-4.7.0-040700rc1-generic
run-parts: executing /etc/kernel/postinst.d/update-notifier 4.7.0-040700rc1-generic /boot/vmlinuz-4.7.0-040700rc1-generic
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.7.0-040700rc1-generic /boot/vmlinuz-4.7.0-040700rc1-generic
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.7.0-040700rc1-generic
Found initrd image: /boot/initrd.img-4.7.0-040700rc1-generic
Found linux image: /boot/vmlinuz-4.4.11-040411-generic
Found initrd image: /boot/initrd.img-4.4.11-040411-generic
Found linux image: /boot/vmlinuz-4.4.0-21-generic
Found initrd image: /boot/initrd.img-4.4.0-21-generic
done

But my server couldn't boot up and it shows:

BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

I have to roll back to the 4.4.11 kernel. Please advice the next step.

[Jacky Wang (jacky-chao-wang)]

v4.7-rc2 kernel (http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.7-rc2-yakkety/) sees the same issue:

[ 70.064983] kernel BUG at /home/kernel/COD/linux/drivers/md/bcache/super.c:1822!
[ 70.065214] invalid opcode: 0000 [#1] SMP
[ 70.065355] Modules linked in: fuse ppdev input_leds lpc_ich intel_powerclamp led_class joydev serio_raw parport_pc sg i2c_piix4 ac battery parport mac_hid acpi_cpufreq ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 multipath linear bcache raid0 md_mod sr_mod cdrom sd_mod ata_generic pata_acpi hid_generic usbhid hid ohci_pci crct10dif_pclmul virtio_net crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel ahci aes_x86_64 lrw gf128mul libahci glue_helper ablk_helper cryptd ohci_hcd ehci_pci ata_piix ehci_hcd libata usbcore scsi_mod psmouse usb_common fjes virtio_pci virtio_ring virtio video button jitterentropy_rng drbg ansi_cprng

[ 70.071156] CPU: 3 PID: 1557 Comm: bcache-register Not tainted 4.7.0-040700rc2-generic #201606051831
[ 70.071417] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 70.071655] task: ffff8800775f0000 ti: ffff88007bf84000 task.ti: ffff88007bf84000
[ 70.072051] RIP: 0010:[<ffffffffc02e4ff6>] [<ffffffffc02e4ff6>] cache_alloc.isra.21+0x646/0x670 [bcache]
[ 70.072757] RSP: 0018:ffff88007bf87d18 EFLAGS: 00010246
[ 70.073129] RAX: ffff88007757b070 RBX: ffff88007c228000 RCX: 00000000000007f8
[ 70.073516] RDX: 00000000000007f7 RSI: 0000000000000000 RDI: ffff88007d003b80
[ 70.073902] RBP: ffff88007bf87d30 R08: ffff88007fd99f40 R09: ffff88007d003b80
[ 70.074288] R10: ffff88007757b070 R11: 0000000000000040 R12: 0000000000000000
[ 70.074674] R13: 0000000000000010 R14: ffff88007b2308f0 R15: ffff880078479000
[ 70.075061] FS: 00007f2bd2778700(0000) GS:ffff88007fd80000(0000) knlGS:0000000000000000
[ 70.075708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 70.076064] CR2: 00007f2bd21e1130 CR3: 0000000079a4b000 CR4: 00000000000406e0
[ 70.076452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 70.076854] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 70.077267] Stack:
[ 70.077565] ffff880035568340 ffffea0001efbcc0 ffff88007c228000 ffff88007bf87dd0
[ 70.078547] ffffffffc02e83f9 ffff88007952b500 ffff880035648f00 000000000000005b
[ 70.079455] 0000000000000080 ffff88007c228008 ffff88007757b080 0000000000000009
[ 70.081160] Call Trace:
[ 70.081755] [<ffffffffc02e83f9>] register_bcache+0x2d9/0x1190 [bcache]
[ 70.082165] [<ffffffff813cabcf>] kobj_attr_store+0xf/0x20
[ 70.082530] [<ffffffff813cabcf>] ? kobj_attr_store+0xf/0x20
[ 70.082902] [<ffffffff812a5ce7>] sysfs_kf_write+0x37/0x40
[ 70.083278] [<ffffffff812a5058>] kernfs_fop_write+0x118/0x1a0
[ 70.083635] [<ffffffff81222287>] __vfs_write+0x37/0x160
[ 70.083977] [<ffffffff81370188>] ? apparmor_file_permission+0x18/0x20
[ 70.084349] [<ffffffff8133084d>] ? security_file_permission+0x3d/0xc0
[ 70.084721] [<fffffff...