BTRFS Kernel trace - Terrible Write Performance

[jdfalk]

I am seeing this several times in the logs. I also installed a mainline upstream kernel and it still occurred in 4.5-rc7-wily.

theark kernel: INFO: task btrfs-transacti:518 blocked for more than 120 seconds.
Mar 10 14:00:44 theark kernel: Not tainted 4.2.0-27-generic #32-Ubuntu
Mar 10 14:00:44 theark kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 14:00:44 theark kernel: btrfs-transacti D ffff88041f296640 0 518 2 0x00000000
Mar 10 14:00:44 theark kernel: ffff8803e0eafdb8 0000000000000046 ffff88040cdaa940 ffff88040c8dee00
Mar 10 14:00:44 theark kernel: 0000000000000246 ffff8803e0eb0000 ffff880409d44178 ffff88040997f800
Mar 10 14:00:44 theark kernel: ffff8803e0eafdf8 ffff880409d441d8 ffff8803e0eafdd8 ffffffff817edc37
Mar 10 14:00:44 theark kernel: Call Trace:
Mar 10 14:00:44 theark kernel: [<ffffffff817edc37>] schedule+0x37/0x80
Mar 10 14:00:44 theark kernel: [<ffffffffc0324b43>] btrfs_commit_transaction+0x423/0xb20 [btrfs]
Mar 10 14:00:44 theark kernel: [<ffffffff810bd760>] ? wake_atomic_t_function+0x60/0x60
Mar 10 14:00:44 theark kernel: [<ffffffffc031fe39>] transaction_kthread+0x229/0x240 [btrfs]
Mar 10 14:00:44 theark kernel: [<ffffffffc031fc10>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
Mar 10 14:00:44 theark kernel: [<ffffffff8109ae38>] kthread+0xd8/0xf0
Mar 10 14:00:44 theark kernel: [<ffffffff8109ad60>] ? kthread_create_on_node+0x1f0/0x1f0
Mar 10 14:00:44 theark kernel: [<ffffffff817f209f>] ret_from_fork+0x3f/0x70
Mar 10 14:00:44 theark kernel: [<ffffffff8109ad60>] ? kthread_create_on_node+0x1f0/0x1f0

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: linux-image-4.2.0-27-generic 4.2.0-27.32
ProcVersionSignature: Ubuntu 4.2.0-27.32-generic 4.2.8-ckt1
Uname: Linux 4.2.0-27-generic x86_64
ApportVersion: 2.19.1-0ubuntu5
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC0D3', '/dev/snd/hwC0D2', '/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D2c', '/dev/snd/pcmC0D2p', '/dev/snd/pcmC0D1p', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Thu Mar 10 13:59:12 2016
HibernationDevice: RESUME=/dev/mapper/sda5_crypt
InstallationDate: Installed on 2015-05-10 (305 days ago)
InstallationMedia: Ubuntu-Server 15.04 "Vivid Vervet" - Release amd64 (20150422)
IwConfig:
 p1p1 no wireless extensions.

 br0 no wireless extensions.

 lo no wireless extensions.
MachineType: Gigabyte Technology Co., Ltd. To be filled by O.E.M.
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.2.0-27-generic.efi.signed root=UUID=51a981ad-aeb0-4bd7-91c6-2d493155c812 ro rootflags=subvol=@ quiet splash vt.handoff=7
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-4.2.0-27-generic N/A
 linux-backports-modules-4.2.0-27-generic N/A
 linux-firmware 1.156
RfKill:

SourcePackage: linux
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
UpgradeStatus: Upgraded to wily on 2015-10-26 (136 days ago)
dmi.bios.date: 01/06/2014
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F18i
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: Z77X-D3H
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF18i:bd01/06/2014:svnGigabyteTechnologyCo.,Ltd.:pnTobefilledbyO.E.M.:pvrTobefilledbyO.E.M.:rvnGigabyteTechnologyCo.,Ltd.:rnZ77X-D3H:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To be filled by O.E.M.
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a kernel version where you were not having this particular problem? This will help determine if the problem you are seeing is the result of a regression, and when this regression was introduced. If this is a regression, we can perform a kernel
 to identify the commit that introduced the problem.

[Joseph Salisbury (jsalisbury)]

his issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

Please follow the instructions on the wiki page[0]. The first step is to email the appropriate mailing list. If no response is received, then a bug may be opened on bugzilla.kernel.org.

Once this bug is reported upstream, please add the tag: 'kernel-bug-reported-upstream'.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[jdfalk (johnathan-falk)]

So it got much worse after 4.2.0-30-generic, the most stable of the kernels I have installed is 4.2.0-27-generic. I will try reinstalling the 4.2.0-25-generic and see how it performs.