I rebooted the system in the meantime. It now had Kernel 4.2.0-30.36 from previous updates:

cat /proc/version_signature
Ubuntu 4.2.0-30.36~14.04.1-generic 4.2.8-ckt3

During the reboot, the system performed some disk I/O for several hours. Before services like sshd became available. I guess this disk operation happened before the file system was mounted in read-write mode.

What I can say is that the system tried to continue the failed balance operation during the boot (from syslog) without intervention from my side:

BTRFS info (device dm-0): continuing balance
BTRFS info (device dm-0): relocating block group 9553211031552 flags 36
found 106150 extents

The btrfs balance operation did not succeed again, but in a manner which I was used to before writing this bug report. This is MUCH better than going into an unpredictable state (ro mode with Input/Output errors). I guess the OS totally got out of sync.

BTRFS info (device dm-0): 2 enospc errors during balance

The btrfs file system was mounted in READ-WRITE-mode this time (not in read-only mode as before!). And this time, I can read the files and navigate through directories without any Input/Output errors (before the reboot, I had lots of them!).

Finally, everything looks fine for a btrfs file system with no space left on the disk.

Now that the fs is online and accessible, I can try to add another disk to the fs to have some empty space again. After that, I hopefully can delete some data and hopefully redo the btrfs balance operation.

/var/log/syslog gives me:

Mar 22 12:42:37 server kernel: [ 3481.725273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 12:42:37 server kernel: [ 3481.726122] mount           D ffff88043e296640     0   587    455 0x00000000
Mar 22 12:42:37 server kernel: [ 3481.726125]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
Mar 22 12:42:37 server kernel: [ 3481.726127]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
Mar 22 12:42:37 server kernel: [ 3481.726129]  ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
Mar 22 12:42:37 server kernel: [ 3481.726130] Call Trace:
Mar 22 12:42:37 server kernel: [ 3481.726136]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 3481.726138]  [<ffffffff817b8ac7>] schedule+0x37/0x80
Mar 22 12:42:37 server kernel: [ 3481.726140]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
Mar 22 12:42:37 server kernel: [ 3481.726143]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
Mar 22 12:42:37 server kernel: [ 3481.726146]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
Mar 22 12:42:37 server kernel: [ 3481.726147]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 3481.726149]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
Mar 22 12:42:37 server kernel: [ 3481.726151]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
Mar 22 12:42:37 server kernel: [ 3481.726152]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
Mar 22 12:42:37 server kernel: [ 3481.726155]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
Mar 22 12:42:37 server kernel: [ 3481.726157]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
Mar 22 12:42:37 server kernel: [ 3481.726159]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
Mar 22 12:42:37 server kernel: [ 3481.726160]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
Mar 22 12:42:37 server kernel: [ 3481.726175]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726184]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726190]  [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726195]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726202]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726210]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726215]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726222]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726227]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726233]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726239]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726247]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726253]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726258]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726260]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
Mar 22 12:42:37 server kernel: [ 3481.726262]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
Mar 22 12:42:37 server kernel: [ 3481.726263]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
Mar 22 12:42:37 server kernel: [ 3481.726265]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
Mar 22 12:42:37 server kernel: [ 3481.726266]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
Mar 22 12:42:37 server kernel: [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds.
Mar 22 12:42:37 server kernel: [ 4562.226559]       Tainted: G     U          4.2.0-30-generic #36~14.04.1-Ubuntu
Mar 22 12:42:37 server kernel: [ 4562.227415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[...]
Mar 22 12:42:37 server kernel: [ 3481.726122] mount           D ffff88043e296640     0   587    455 0x00000000
Mar 22 12:42:37 server kernel: [ 3481.726125]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
Mar 22 12:42:37 server kernel: [ 3481.726127]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
Mar 22 12:42:37 server kernel: [ 3481.726129]  ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
Mar 22 12:42:37 server kernel: [ 3481.726130] Call Trace:
Mar 22 12:42:37 server kernel: [ 3481.726136]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 3481.726138]  [<ffffffff817b8ac7>] schedule+0x37/0x80
Mar 22 12:42:37 server kernel: [ 3481.726140]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
Mar 22 12:42:37 server kernel: [ 3481.726143]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
Mar 22 12:42:37 server kernel: [ 3481.726146]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
Mar 22 12:42:37 server kernel: [ 3481.726147]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 3481.726149]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
Mar 22 12:42:37 server kernel: [ 3481.726151]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
Mar 22 12:42:37 server kernel: [ 3481.726152]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
Mar 22 12:42:37 server kernel: [ 3481.726155]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
Mar 22 12:42:37 server kernel: [ 3481.726157]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
Mar 22 12:42:37 server kernel: [ 3481.726159]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
Mar 22 12:42:37 server kernel: [ 3481.726160]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
Mar 22 12:42:37 server kernel: [ 3481.726175]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726184]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726190]  [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726195]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726202]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726210]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726215]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726222]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726227]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726233]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726239]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726247]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726253]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726258]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
Mar 22 12:42:37 server kernel: [ 3481.726260]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
Mar 22 12:42:37 server kernel: [ 3481.726262]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
Mar 22 12:42:37 server kernel: [ 3481.726263]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
Mar 22 12:42:37 server kernel: [ 3481.726265]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
Mar 22 12:42:37 server kernel: [ 3481.726266]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
Mar 22 12:42:37 server kernel: [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds.
[...]
Mar 22 12:42:37 server kernel: [ 4562.228280] mount           D ffff88043e296640     0   587    455 0x00000000
Mar 22 12:42:37 server kernel: [ 4562.228283]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
Mar 22 12:42:37 server kernel: [ 4562.228285]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
Mar 22 12:42:37 server kernel: [ 4562.228286]  ffff88043e5c05e8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
Mar 22 12:42:37 server kernel: [ 4562.228288] Call Trace:
Mar 22 12:42:37 server kernel: [ 4562.228294]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 4562.228296]  [<ffffffff817b8ac7>] schedule+0x37/0x80
Mar 22 12:42:37 server kernel: [ 4562.228297]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
Mar 22 12:42:37 server kernel: [ 4562.228301]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
Mar 22 12:42:37 server kernel: [ 4562.228304]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
Mar 22 12:42:37 server kernel: [ 4562.228305]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
Mar 22 12:42:37 server kernel: [ 4562.228307]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
Mar 22 12:42:37 server kernel: [ 4562.228309]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
Mar 22 12:42:37 server kernel: [ 4562.228310]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
Mar 22 12:42:37 server kernel: [ 4562.228313]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
Mar 22 12:42:37 server kernel: [ 4562.228315]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
Mar 22 12:42:37 server kernel: [ 4562.228317]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
Mar 22 12:42:37 server kernel: [ 4562.228318]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
Mar 22 12:42:37 server kernel: [ 4562.228332]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228334]  [<ffffffff81180df1>] ? set_page_dirty+0x41/0x70
Mar 22 12:42:37 server kernel: [ 4562.228343]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228348]  [<ffffffffc0498071>] ? btrfs_release_path+0x21/0xa0 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228354]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228361]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228369]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228374]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228381]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228386]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228392]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228398]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228405]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228412]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228417]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
Mar 22 12:42:37 server kernel: [ 4562.228419]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
Mar 22 12:42:37 server kernel: [ 4562.228421]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
Mar 22 12:42:37 server kernel: [ 4562.228422]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
Mar 22 12:42:37 server kernel: [ 4562.228424]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
Mar 22 12:42:37 server kernel: [ 4562.228425]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
Mar 22 12:42:37 server kernel: [ 7975.560908] BTRFS info (device dm-0): continuing balance
[...]
Mar 22 12:44:03 server kernel: [ 8073.441009] BTRFS info (device dm-0): relocating block group 9553211031552 flags 36
[...]
Mar 22 12:51:33 server kernel: [ 8524.324198] btrfs-transacti D ffff88043e3d6640     0   409      2 0x00000000
Mar 22 12:51:33 server kernel: [ 8524.324204]  ffff880424983d68 0000000000000046 ffff88042b9b8000 ffff880425be5280
Mar 22 12:51:33 server kernel: [ 8524.324208]  0000000000000246 ffff880424984000 ffff8803ff9884e0 ffff8804265801b0
Mar 22 12:51:33 server kernel: [ 8524.324211]  ffff88042bbe5000 0000000000000000 ffff880424983d88 ffffffff817b8ac7
Mar 22 12:51:33 server kernel: [ 8524.324215] Call Trace:
Mar 22 12:51:33 server kernel: [ 8524.324225]  [<ffffffff817b8ac7>] schedule+0x37/0x80
Mar 22 12:51:33 server kernel: [ 8524.324247]  [<ffffffffc04c06d0>] wait_for_commit.isra.17+0x40/0x70 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324252]  [<ffffffff810b7670>] ? prepare_to_wait_event+0xf0/0xf0
Mar 22 12:51:33 server kernel: [ 8524.324267]  [<ffffffffc04c149c>] btrfs_commit_transaction+0x15c/0xb10 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324281]  [<ffffffffc04c1ee3>] ? start_transaction+0x93/0x580 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324295]  [<ffffffffc04bc99a>] transaction_kthread+0x1ba/0x240 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324307]  [<ffffffffc04bc7e0>] ? btrfs_cleanup_transaction+0x540/0x540 [btrfs]
Mar 22 12:51:33 server kernel: [ 8524.324311]  [<ffffffff81095499>] kthread+0xc9/0xe0
Mar 22 12:51:33 server kernel: [ 8524.324315]  [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0
Mar 22 12:51:33 server kernel: [ 8524.324318]  [<ffffffff817bc75f>] ret_from_fork+0x3f/0x70
Mar 22 12:51:33 server kernel: [ 8524.324322]  [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0
Mar 22 12:52:23 server kernel: [ 8574.196463] init: ureadahead main process (449) killed by KILL signal
Mar 22 12:59:29 server kernel: [ 9000.470433] BTRFS info (device dm-0): found 106150 extents
Mar 22 12:59:54 server kernel: [ 9025.568882] BTRFS info (device dm-0): 2 enospc errors during balance



dmesg gives me

[ 3481.723605] INFO: task mount:587 blocked for more than 120 seconds.
[ 3481.724438]       Tainted: G     U          4.2.0-30-generic #36~14.04.1-Ubuntu
[ 3481.725273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3481.726122] mount           D ffff88043e296640     0   587    455 0x00000000
[ 3481.726125]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
[ 3481.726127]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
[ 3481.726129]  ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
[ 3481.726130] Call Trace:
[ 3481.726136]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
[ 3481.726138]  [<ffffffff817b8ac7>] schedule+0x37/0x80
[ 3481.726140]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
[ 3481.726143]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
[ 3481.726146]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
[ 3481.726147]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
[ 3481.726149]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
[ 3481.726151]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
[ 3481.726152]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
[ 3481.726155]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
[ 3481.726157]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
[ 3481.726159]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
[ 3481.726160]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
[ 3481.726175]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
[ 3481.726184]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
[ 3481.726190]  [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs]
[ 3481.726195]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
[ 3481.726202]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
[ 3481.726210]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
[ 3481.726215]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
[ 3481.726222]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
[ 3481.726227]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
[ 3481.726233]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
[ 3481.726239]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
[ 3481.726247]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
[ 3481.726253]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
[ 3481.726258]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
[ 3481.726260]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
[ 3481.726262]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
[ 3481.726263]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
[ 3481.726265]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
[ 3481.726266]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds.
[ 4562.226559]       Tainted: G     U          4.2.0-30-generic #36~14.04.1-Ubuntu
[ 4562.227415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4562.228280] mount           D ffff88043e296640     0   587    455 0x00000000
[ 4562.228283]  ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00
[ 4562.228285]  ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff
[ 4562.228286]  ffff88043e5c05e8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7
[ 4562.228288] Call Trace:
[ 4562.228294]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
[ 4562.228296]  [<ffffffff817b8ac7>] schedule+0x37/0x80
[ 4562.228297]  [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0
[ 4562.228301]  [<ffffffff8101d749>] ? read_tsc+0x9/0x10
[ 4562.228304]  [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0
[ 4562.228305]  [<ffffffff817b9240>] ? bit_wait+0x60/0x60
[ 4562.228307]  [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110
[ 4562.228309]  [<ffffffff817b925f>] bit_wait_io+0x1f/0x70
[ 4562.228310]  [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90
[ 4562.228313]  [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170
[ 4562.228315]  [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0
[ 4562.228317]  [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40
[ 4562.228318]  [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180
[ 4562.228332]  [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs]
[ 4562.228334]  [<ffffffff81180df1>] ? set_page_dirty+0x41/0x70
[ 4562.228343]  [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs]
[ 4562.228348]  [<ffffffffc0498071>] ? btrfs_release_path+0x21/0xa0 [btrfs]
[ 4562.228354]  [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs]
[ 4562.228361]  [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs]
[ 4562.228369]  [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs]
[ 4562.228374]  [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs]
[ 4562.228381]  [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs]
[ 4562.228386]  [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs]
[ 4562.228392]  [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs]
[ 4562.228398]  [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs]
[ 4562.228405]  [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs]
[ 4562.228412]  [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs]
[ 4562.228417]  [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs]
[ 4562.228419]  [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0
[ 4562.228421]  [<ffffffff8120c413>] do_mount+0x843/0xcf0
[ 4562.228422]  [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150
[ 4562.228424]  [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0
[ 4562.228425]  [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 7975.560908] BTRFS info (device dm-0): continuing balance
[ 7977.590090] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
[ 7978.881574] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null)
[ 7982.053560] init: failsafe main process (1047) killed by TERM signal
[...]
[ 8073.441009] BTRFS info (device dm-0): relocating block group 9553211031552 flags 36
[ 8081.239604] audit: type=1400 audit(1458647050.858:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=1299 comm="apparmor_parser"
[ 8171.758863] init: mysql post-start process (1322) terminated with status 1
[ 8190.153034] audit: type=1400 audit(1458647159.668:23): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/var/tmp/mysql/" pid=1321 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=106 ouid=0
[ 8216.355398] init: plymouth-upstart-bridge main process ended, respawning
[ 8524.324045] INFO: task btrfs-transacti:409 blocked for more than 120 seconds.
[ 8524.324102]       Tainted: G     U          4.2.0-30-generic #36~14.04.1-Ubuntu
[ 8524.324149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 8524.324198] btrfs-transacti D ffff88043e3d6640     0   409      2 0x00000000
[ 8524.324204]  ffff880424983d68 0000000000000046 ffff88042b9b8000 ffff880425be5280
[ 8524.324208]  0000000000000246 ffff880424984000 ffff8803ff9884e0 ffff8804265801b0
[ 8524.324211]  ffff88042bbe5000 0000000000000000 ffff880424983d88 ffffffff817b8ac7
[ 8524.324215] Call Trace:
[ 8524.324225]  [<ffffffff817b8ac7>] schedule+0x37/0x80
[ 8524.324247]  [<ffffffffc04c06d0>] wait_for_commit.isra.17+0x40/0x70 [btrfs]
[ 8524.324252]  [<ffffffff810b7670>] ? prepare_to_wait_event+0xf0/0xf0
[ 8524.324267]  [<ffffffffc04c149c>] btrfs_commit_transaction+0x15c/0xb10 [btrfs]
[ 8524.324281]  [<ffffffffc04c1ee3>] ? start_transaction+0x93/0x580 [btrfs]
[ 8524.324295]  [<ffffffffc04bc99a>] transaction_kthread+0x1ba/0x240 [btrfs]
[ 8524.324307]  [<ffffffffc04bc7e0>] ? btrfs_cleanup_transaction+0x540/0x540 [btrfs]
[ 8524.324311]  [<ffffffff81095499>] kthread+0xc9/0xe0
[ 8524.324315]  [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0
[ 8524.324318]  [<ffffffff817bc75f>] ret_from_fork+0x3f/0x70
[ 8524.324322]  [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0
[ 8574.196463] init: ureadahead main process (449) killed by KILL signal
[ 9000.470433] BTRFS info (device dm-0): found 106150 extents
[ 9025.568882] BTRFS info (device dm-0): 2 enospc errors during balance