FS-Cache: Assertion failed

[DanielT]

I'm encountering many random kernel panic after enabling fsc with NFS. Then I see a number of following err msgs right before it crashed.

Nov 20 12:04:26 g2 kernel: [170689.549374] FS-Cache:
Nov 20 12:04:26 g2 kernel: [170689.549381] FS-Cache: Assertion failed
Nov 20 12:04:26 g2 kernel: [170689.549384] FS-Cache: 6 == 5 is false
Nov 20 12:04:26 g2 kernel: [170689.549439] ------------[ cut here ]------------
Nov 20 12:04:26 g2 kernel: [170689.549490] kernel BUG at /build/linux-3phnTq/linux-4.4.0/fs/fscache/operation.c:494!
Nov 20 12:04:26 g2 kernel: [170689.549557] invalid opcode: 0000 [#1] SMP
Nov 20 12:04:26 g2 kernel: [170689.549598] Modules linked in: veth xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs cachefiles ipmi_devintf rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache snd_hda_codec_hdmi intel_rapl x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm ipmi_ssif irqbypass serio_raw joydev input_leds snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm nvidia_uvm(POE) sb_edac snd_seq_midi snd_seq_midi_event edac_core snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore mei_me lpc_ich mei shpchp wmi ipmi_si 8250_fintek ipmi_msghandler acpi_pad acpi_power_meter mac_hid sunrpc ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nct6775 hwmon_vid coretemp bonding parport_pc ppdev lp parport autofs4 xfs btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear nvidia_drm(POE) nvidia_modeset(POE) hid_generic crct10dif_pclmul ast crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul nvidia(POE) glue_helper ttm ablk_helper igb cryptd drm_kms_helper psmouse syscopyarea dca sysfillrect ptp sysimgblt fb_sys_fops pps_core ahci i2c_algo_bit drm libahci fjes usbhid hid
Nov 20 12:04:26 g2 kernel: [170689.550993] CPU: 22 PID: 6373 Comm: kworker/u98:1 Tainted: P OE 4.4.0-98-generic #121-Ubuntu
Nov 20 12:04:26 g2 kernel: [170689.551071] Hardware name: ASUSTeK COMPUTER INC. ESC8000 G3 Series/Z10PG-D24 Series, BIOS 3203 05/05/2016
Nov 20 12:04:26 g2 kernel: [170689.551160] Workqueue: fscache_operation fscache_op_work_func [fscache]
Nov 20 12:04:26 g2 kernel: [170689.551219] task: ffff882027887000 ti: ffff881429ea4000 task.ti: ffff881429ea4000
Nov 20 12:04:26 g2 kernel: [170689.551283] RIP: 0010:[<ffffffffc065df96>] [<ffffffffc065df96>] fscache_put_operation+0x1d6/0x210 [fscache]
Nov 20 12:04:26 g2 kernel: [170689.551374] RSP: 0018:ffff881429ea7de0 EFLAGS: 00010282
Nov 20 12:04:26 g2 kernel: [170689.551421] RAX: 0000000000000019 RBX: ffff882037224300 RCX: 0000000000000006
Nov 20 12:04:26 g2 kernel: [170689.551490] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff88203f28dd50
Nov 20 12:04:26 g2 kernel: [170689.551551] RBP: ffff881429ea7df8 R08: 000000000000000a R09: 00000000000009c0
Nov 20 12:04:26 g2 kernel: [170689.551611] R10: ffff88120db1b300 R11: 00000000000009c0 R12: ffff882038bf8000
Nov 20 12:04:26 g2 kernel: [170689.551672] R13: ffff88202ff08800 R14: 0000000000000c40 R15: ffff882037224300
Nov 20 12:04:26 g2 kernel: [170689.551733] FS: 0000000000000000(0000) GS:ffff88203f280000(0000) knlGS:0000000000000000
Nov 20 12:04:26 g2 kernel: [170689.551801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 20 12:04:26 g2 kernel: [170689.551850] CR2: 00007f66b2887500 CR3: 0000000001e0a000 CR4: 00000000003406e0
Nov 20 12:04:26 g2 kernel: [170689.551912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 20 12:04:26 g2 kernel: [170689.551972] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 20 12:04:26 g2 kernel: [170689.552030] Stack:
Nov 20 12:04:26 g2 kernel: [170689.552051] ffff882037224300 ffff882038bf8000 ffff88202ff08800 ffff881429ea7e10
Nov 20 12:04:26 g2 kernel: [170689.552124] ffffffffc065e1ca ffff88203439e780 ffff881429ea7e50 ffffffff8109a635
Nov 20 12:04:26 g2 kernel: [170689.552196] 0000000000000000 ffff88203439e7b0 ffff882038bf8000 0000000000000088
Nov 20 12:04:26 g2 kernel: [170689.552267] Call Trace:
Nov 20 12:04:26 g2 kernel: [170689.552299] [<ffffffffc065e1ca>] fscache_op_work_func+0x2a/0x50 [fscache]
Nov 20 12:04:26 g2 kernel: [170689.552363] [<ffffffff8109a635>] process_one_work+0x165/0x480
Nov 20 12:04:26 g2 kernel: [170689.552417] [<ffffffff8109a99b>] worker_thread+0x4b/0x4c0
Nov 20 12:04:26 g2 kernel: [170689.552468] [<ffffffff8109a950>] ? process_one_work+0x480/0x480
Nov 20 12:04:26 g2 kernel: [170689.552522] [<ffffffff810a0c75>] kthread+0xe5/0x100
Nov 20 12:04:26 g2 kernel: [170689.552567] [<ffffffff810a0b90>] ? kthread_create_on_node+0x1e0/0x1e0
Nov 20 12:04:26 g2 kernel: [170689.552626] [<ffffffff81844a4f>] ret_from_fork+0x3f/0x70
Nov 20 12:04:26 g2 kernel: [170689.552674] [<ffffffff810a0b90>] ? kthread_create_on_node+0x1e0/0x1e0
Nov 20 12:04:26 g2 kernel: [170689.552730] Code: c7 c7 fb 32 66 c0 e8 d1 02 b3 c0 48 c7 c7 09 33 66 c0 e8 c5 02 b3 c0 8b 73 40 ba 05 00 00 00 48 c7 c7 b0 1e 66 c0 e8 b1 02 b3 c0 <0f> 0b 48 c7 c7 fb 32 66 c0 e8 a3 02 b3 c0 48 c7 c7 09 33 66 c0
Nov 20 12:04:26 g2 kernel: [170689.553040] RIP [<ffffffffc065df96>] fscache_put_operation+0x1d6/0x210 [fscache]
Nov 20 12:04:26 g2 kernel: [170689.553111] RSP <ffff881429ea7de0>
Nov 20 12:04:26 g2 kernel: [170689.571801] ---[ end trace 1e9e97557a315242 ]---

# uname -a
Linux g2 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 14:24:03 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-98-generic 4.4.0-98.121
ProcVersionSignature: Ubuntu 4.4.0-98.121-generic 4.4.90
Uname: Linux 4.4.0-98-generic x86_64
NonfreeKernelModules: nvidia_uvm nvidia_drm nvidia_modeset nvidia
ApportVersion: 2.20.1-0ubuntu2.12
Architecture: amd64
Date: Mon Nov 20 19:06:29 2017
HibernationDevice: RESUME=/dev/mapper/umbocv_vg-swap_1
InstallationDate: Installed on 2017-06-14 (159 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
MachineType: ASUSTeK COMPUTER INC. ESC8000 G3 Series
ProcEnviron:
 LD_LIBRARY_PATH=<set>
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-98-generic root=/dev/mapper/umbocv_vg-root ro
PulseList:
 Error: command ['pacmd', 'list'] failed with exit code 1: Home directory not accessible: Permission denied
 No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-98-generic N/A
 linux-backports-modules-4.4.0-98-generic N/A
 linux-firmware 1.157.13
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/05/2016
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 3203
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: Z10PG-D24 Series
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 17
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3203:bd05/05/2016:svnASUSTeKCOMPUTERINC.:pnESC8000G3Series:pvrRev1.xx:rvnASUSTeKCOMPUTERINC.:rnZ10PG-D24Series:rvrRev1.xx:cvnToBeFilledByO.E.M.:ct17:cvrToBeFilledByO.E.M.:
dmi.product.name: ESC8000 G3 Series
dmi.product.version: Rev 1.xx
dmi.sys.vendor: ASUSTeK COMPUTER INC.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.14 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14

[DanielT (cooldll694)]

Hi Joseph,

Thanks for your advice.
I've installed the kernel but the machine is currently in heavy usage so I might only have a chance to reboot to about a week later. Will update once I try it.

[DanielT (cooldll694)]

Hi,

Just to update that I've tried to install and boot mainline kernel 4.14.

Unfortunately boot always fails right after "Dec 08 00:17:39 g2 systemd[1]: dev-disk-by\x2duuid-e9031178\x2d42d7\x2d46bf\x2db1b2\x2d979dc2b54c08.device: Job dev-disk-by\x2duuid-e9031178\x2d42d7\x2d46bf\x2db1b2\x2d979dc2b54c08.device/start timed out." message. I tried adding "x-systemd.device-timeout=5min" to the corresponding entries in fstab and disabled selinux but still no luck... (Downgrading the kernel back to original 4.04 works immediately though)

It'd be appreciated if there're some advices on getting it work :)

[Yuchen Fan (ychfan)]

Hi,

I met the same bugs on kernel 4.10.0-38-generic

[Ben Kantor (kkjawz)]

Same bug is occurring on 4.13.0-36-generic as well.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]