btrfs scrub hangs

[frankster]

I have a btrfs filesystem comprised of a few discs (metadata raid1, data unduplicated). I run btrfs scrub and some checksum errors are reported, then the scrub hangs and there are some stack traces in syslog. This occurs with kernel 3.19.0-26-generic and is a regression from 3.13.0-62-generic, where the scrub reports checksum errors but completes successfully.

Aug 19 19:36:45 mythtv kernel: [ 1560.092069] INFO: task btrfs:17303 blocked for more than 120 seconds.
Aug 19 19:36:45 mythtv kernel: [ 1560.092075] Tainted: P OE 3.19.0-26-generic #28~14.04.1-Ubuntu
Aug 19 19:36:45 mythtv kernel: [ 1560.092076] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 19 19:36:45 mythtv kernel: [ 1560.092079] btrfs D ffff8800706bb828 0 17303 1 0x00000000
Aug 19 19:36:45 mythtv kernel: [ 1560.092084] ffff8800706bb828 ffff880070526bf0 0000000000013e80 ffff8800706bbfd8
Aug 19 19:36:45 mythtv kernel: [ 1560.092087] 0000000000013e80 ffff88006faa3110 ffff880070526bf0 ffff8800706bb838
Aug 19 19:36:45 mythtv kernel: [ 1560.092091] ffff8801d2742000 ffff8801d2742218 ffff8801d2742220 ffff880186bf0b00
Aug 19 19:36:45 mythtv kernel: [ 1560.092094] Call Trace:
Aug 19 19:36:45 mythtv kernel: [ 1560.092103] [<ffffffff817b24f9>] schedule+0x29/0x70
Aug 19 19:36:45 mythtv kernel: [ 1560.092139] [<ffffffffc058dfc5>] scrub_add_page_to_rd_bio+0x1c5/0x2b0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092144] [<ffffffff810b4e10>] ? prepare_to_wait_event+0x110/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092164] [<ffffffffc058f76c>] scrub_pages+0x1ec/0x290 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092183] [<ffffffffc0591498>] scrub_stripe+0xa48/0x1040 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092204] [<ffffffffc0591bab>] scrub_chunk.isra.19+0x11b/0x140 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092224] [<ffffffffc054e009>] ? btrfs_get_token_64+0x59/0xe0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092244] [<ffffffffc0591e2a>] scrub_enumerate_chunks+0x25a/0x4b0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092246] [<ffffffff810b4e00>] ? prepare_to_wait_event+0x100/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092259] [<ffffffffc05937a8>] btrfs_scrub_dev+0x1a8/0x550 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092263] [<ffffffff8120cfb5>] ? __mnt_want_write_file+0x25/0x30
Aug 19 19:36:45 mythtv kernel: [ 1560.092276] [<ffffffffc056ddaf>] btrfs_ioctl+0x138f/0x2ac0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092279] [<ffffffff811cf7d5>] ? kmem_cache_alloc_node+0x115/0x250
Aug 19 19:36:45 mythtv kernel: [ 1560.092282] [<ffffffff8137beb7>] ? create_task_io_context+0x27/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092285] [<ffffffff811ffbf8>] do_vfs_ioctl+0x2f8/0x510
Aug 19 19:36:45 mythtv kernel: [ 1560.092288] [<ffffffff8138a324>] ? set_task_ioprio+0x94/0xb0
Aug 19 19:36:45 mythtv kernel: [ 1560.092290] [<ffffffff811ffe91>] SyS_ioctl+0x81/0xa0
Aug 19 19:36:45 mythtv kernel: [ 1560.092292] [<ffffffff817b688d>] system_call_fastpath+0x16/0x1b
Aug 19 19:36:45 mythtv kernel: [ 1560.092295] INFO: task btrfs:17304 blocked for more than 120 seconds.
Aug 19 19:36:45 mythtv kernel: [ 1560.092296] Tainted: P OE 3.19.0-26-generic #28~14.04.1-Ubuntu
Aug 19 19:36:45 mythtv kernel: [ 1560.092297] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 19 19:36:45 mythtv kernel: [ 1560.092298] btrfs D ffff8800706ab8b8 0 17304 1 0x00000000
Aug 19 19:36:45 mythtv kernel: [ 1560.092300] ffff8800706ab8b8 ffff8800705275c0 0000000000013e80 ffff8800706abfd8
Aug 19 19:36:45 mythtv kernel: [ 1560.092303] 0000000000013e80 ffff88006faa3110 ffff8800705275c0 ffff8800706ab8c8
Aug 19 19:36:45 mythtv kernel: [ 1560.092305] ffff8800705821e0 ffff880070582218 ffff8800706ab8f8 ffff8801d2740800
Aug 19 19:36:45 mythtv kernel: [ 1560.092307] Call Trace:
Aug 19 19:36:45 mythtv kernel: [ 1560.092309] [<ffffffff817b24f9>] schedule+0x29/0x70
Aug 19 19:36:45 mythtv kernel: [ 1560.092322] [<ffffffffc059560d>] btrfs_reada_wait+0x6d/0xb0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092324] [<ffffffff810b4e10>] ? prepare_to_wait_event+0x110/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092337] [<ffffffffc0590cb7>] scrub_stripe+0x267/0x1040 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092346] [<ffffffffc050ed2d>] ? generic_bin_search.constprop.35+0x8d/0x1b0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092359] [<ffffffffc0591bab>] scrub_chunk.isra.19+0x11b/0x140 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092371] [<ffffffffc054e009>] ? btrfs_get_token_64+0x59/0xe0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092384] [<ffffffffc0591e2a>] scrub_enumerate_chunks+0x25a/0x4b0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092386] [<ffffffff810b4e00>] ? prepare_to_wait_event+0x100/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092399] [<ffffffffc05937a8>] btrfs_scrub_dev+0x1a8/0x550 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092413] [<ffffffff8120cfb5>] ? __mnt_want_write_file+0x25/0x30
Aug 19 19:36:45 mythtv kernel: [ 1560.092425] [<ffffffffc056ddaf>] btrfs_ioctl+0x138f/0x2ac0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092428] [<ffffffff811cf7d5>] ? kmem_cache_alloc_node+0x115/0x250
Aug 19 19:36:45 mythtv kernel: [ 1560.092430] [<ffffffff8137beb7>] ? create_task_io_context+0x27/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092432] [<ffffffff811ffbf8>] do_vfs_ioctl+0x2f8/0x510
Aug 19 19:36:45 mythtv kernel: [ 1560.092434] [<ffffffff8138a324>] ? set_task_ioprio+0x94/0xb0
Aug 19 19:36:45 mythtv kernel: [ 1560.092436] [<ffffffff811ffe91>] SyS_ioctl+0x81/0xa0
Aug 19 19:36:45 mythtv kernel: [ 1560.092438] [<ffffffff817b688d>] system_call_fastpath+0x16/0x1b
Aug 19 19:36:45 mythtv kernel: [ 1560.092440] INFO: task btrfs:17305 blocked for more than 120 seconds.
Aug 19 19:36:45 mythtv kernel: [ 1560.092442] Tainted: P OE 3.19.0-26-generic #28~14.04.1-Ubuntu
Aug 19 19:36:45 mythtv kernel: [ 1560.092443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 19 19:36:45 mythtv kernel: [ 1560.092444] btrfs D ffff8800706af8b8 0 17305 1 0x00000000
Aug 19 19:36:45 mythtv kernel: [ 1560.092446] ffff8800706af8b8 ffff8802161c1d70 0000000000013e80 ffff8800706affd8
Aug 19 19:36:45 mythtv kernel: [ 1560.092448] 0000000000013e80 ffff880188a64e80 ffff8802161c1d70 ffff8800706af8c8
Aug 19 19:36:45 mythtv kernel: [ 1560.092450] ffff88007058fd20 ffff88007058fd58 ffff8800706af8f8 ffff8801d2742c00
Aug 19 19:36:45 mythtv kernel: [ 1560.092452] Call Trace:
Aug 19 19:36:45 mythtv kernel: [ 1560.092455] [<ffffffff817b24f9>] schedule+0x29/0x70
Aug 19 19:36:45 mythtv kernel: [ 1560.092467] [<ffffffffc059560d>] btrfs_reada_wait+0x6d/0xb0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092469] [<ffffffff810b4e10>] ? prepare_to_wait_event+0x110/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092482] [<ffffffffc0590cc8>] scrub_stripe+0x278/0x1040 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092484] [<ffffffff810b4e50>] ? autoremove_wake_function+0x40/0x40
Aug 19 19:36:45 mythtv kernel: [ 1560.092498] [<ffffffffc0558e2a>] ? read_extent_buffer_pages+0x2ca/0x300 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092507] [<ffffffffc050ed2d>] ? generic_bin_search.constprop.35+0x8d/0x1b0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092520] [<ffffffffc0591bab>] scrub_chunk.isra.19+0x11b/0x140 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092532] [<ffffffffc054e009>] ? btrfs_get_token_64+0x59/0xe0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092545] [<ffffffffc0591e2a>] scrub_enumerate_chunks+0x25a/0x4b0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092547] [<ffffffff810b4e00>] ? prepare_to_wait_event+0x100/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092560] [<ffffffffc05937a8>] btrfs_scrub_dev+0x1a8/0x550 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092562] [<ffffffff8120cfb5>] ? __mnt_want_write_file+0x25/0x30
Aug 19 19:36:45 mythtv kernel: [ 1560.092574] [<ffffffffc056ddaf>] btrfs_ioctl+0x138f/0x2ac0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092577] [<ffffffff811cf7d5>] ? kmem_cache_alloc_node+0x115/0x250
Aug 19 19:36:45 mythtv kernel: [ 1560.092579] [<ffffffff8137beb7>] ? create_task_io_context+0x27/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092581] [<ffffffff811ffbf8>] do_vfs_ioctl+0x2f8/0x510
Aug 19 19:36:45 mythtv kernel: [ 1560.092583] [<ffffffff8138a324>] ? set_task_ioprio+0x94/0xb0
Aug 19 19:36:45 mythtv kernel: [ 1560.092585] [<ffffffff811ffe91>] SyS_ioctl+0x81/0xa0
Aug 19 19:36:45 mythtv kernel: [ 1560.092587] [<ffffffff817b688d>] system_call_fastpath+0x16/0x1b
Aug 19 19:36:45 mythtv kernel: [ 1560.092589] INFO: task btrfs:17306 blocked for more than 120 seconds.
Aug 19 19:36:45 mythtv kernel: [ 1560.092591] Tainted: P OE 3.19.0-26-generic #28~14.04.1-Ubuntu
Aug 19 19:36:45 mythtv kernel: [ 1560.092592] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 19 19:36:45 mythtv kernel: [ 1560.092593] btrfs D ffff8800706b38b8 0 17306 1 0x00000000
Aug 19 19:36:45 mythtv kernel: [ 1560.092595] ffff8800706b38b8 ffff8801da956bf0 0000000000013e80 ffff8800706b3fd8
Aug 19 19:36:45 mythtv kernel: [ 1560.092597] 0000000000013e80 ffff88006faa3110 ffff8801da956bf0 ffff8800706b38c8
Aug 19 19:36:45 mythtv kernel: [ 1560.092599] ffff880070582480 ffff8800705824b8 ffff8800706b38f8 ffff8801d2741c00
Aug 19 19:36:45 mythtv kernel: [ 1560.092601] Call Trace:
Aug 19 19:36:45 mythtv kernel: [ 1560.092603] [<ffffffff817b24f9>] schedule+0x29/0x70
Aug 19 19:36:45 mythtv kernel: [ 1560.092616] [<ffffffffc059560d>] btrfs_reada_wait+0x6d/0xb0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092618] [<ffffffff810b4e10>] ? prepare_to_wait_event+0x110/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092630] [<ffffffffc0590cb7>] scrub_stripe+0x267/0x1040 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092640] [<ffffffffc050ed2d>] ? generic_bin_search.constprop.35+0x8d/0x1b0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092653] [<ffffffffc0591bab>] scrub_chunk.isra.19+0x11b/0x140 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092665] [<ffffffffc054e009>] ? btrfs_get_token_64+0x59/0xe0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092678] [<ffffffffc0591e2a>] scrub_enumerate_chunks+0x25a/0x4b0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092680] [<ffffffff810b4e00>] ? prepare_to_wait_event+0x100/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092693] [<ffffffffc05937a8>] btrfs_scrub_dev+0x1a8/0x550 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092695] [<ffffffff8120cfb5>] ? __mnt_want_write_file+0x25/0x30
Aug 19 19:36:45 mythtv kernel: [ 1560.092707] [<ffffffffc056ddaf>] btrfs_ioctl+0x138f/0x2ac0 [btrfs]
Aug 19 19:36:45 mythtv kernel: [ 1560.092710] [<ffffffff811cf7d5>] ? kmem_cache_alloc_node+0x115/0x250
Aug 19 19:36:45 mythtv kernel: [ 1560.092712] [<ffffffff8137beb7>] ? create_task_io_context+0x27/0x110
Aug 19 19:36:45 mythtv kernel: [ 1560.092714] [<ffffffff811ffbf8>] do_vfs_ioctl+0x2f8/0x510
Aug 19 19:36:45 mythtv kernel: [ 1560.092716] [<ffffffff8138a324>] ? set_task_ioprio+0x94/0xb0
Aug 19 19:36:45 mythtv kernel: [ 1560.092718] [<ffffffff811ffe91>] SyS_ioctl+0x81/0xa0
Aug 19 19:36:45 mythtv kernel: [ 1560.092720] [<ffffffff817b688d>] system_call_fastpath+0x16/0x1b

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-extra-3.19.0-26-generic 3.19.0-26.28~14.04.1
ProcVersionSignature: Ubuntu 3.19.0-26.28~14.04.1-generic 3.19.8-ckt4
Uname: Linux 3.19.0-26-generic x86_64
NonfreeKernelModules: fglrx
ApportVersion: 2.14.1-0ubuntu3.11
Architecture: amd64
Date: Thu Aug 20 09:31:52 2015
InstallationDate: Installed on 2012-03-26 (1241 days ago)
InstallationMedia: Mythbuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111012)
SourcePackage: linux-lts-vivid
UpgradeStatus: Upgraded to trusty on 2014-07-20 (395 days ago)

[frankster (wtfrank)]

Problem doesn't occur with linux-image-4.0.9-040009-generic_4.0.9-040009.201507212131

[frankster (wtfrank)]

btrfs-tools 3.14.1-1