Ubuntu
linux package

BTRFS Page allocation failure with Bacula.

Bug #1576297 reported by MikeB on 2016-04-28

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	linux (Ubuntu)	Confirmed	Medium	Unassigned

Bug Description

As soon as bacula starts running I see the below errors in the kernel log, and the server eventually hangs at some point. I'm backing up our servers to a local BTRFS volume on a local drive, nothing fancy and it used to work really well in Ubuntu 14.04.

Apr 27 21:05:18 backup kernel: [343463.805173] kworker/u4:5: page allocation failure: order:0, mode:0x2204020
Apr 27 21:05:18 backup kernel: [343463.805178] CPU: 1 PID: 19811 Comm: kworker/u4:5 Not tainted 4.4.0-21-generic #37-Ubuntu
Apr 27 21:05:18 backup kernel: [343463.805179] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Apr 27 21:05:18 backup kernel: [343463.805199] Workqueue: btrfs-submit btrfs_submit_helper [btrfs]
Apr 27 21:05:18 backup kernel: [343463.805201] 0000000000000086 000000005236f502 ffff880019f2b548 ffffffff813e93c3
Apr 27 21:05:18 backup kernel: [343463.805203] 0000000002204020 0000000000000000 ffff880019f2b5d8 ffffffff81192fba
Apr 27 21:05:18 backup kernel: [343463.805205] ffff88007fff9d40 ffff88007fffb6e0 0000000000000000 ffffffff00000060
Apr 27 21:05:18 backup kernel: [343463.805206] Call Trace:
Apr 27 21:05:18 backup kernel: [343463.805211] [<ffffffff813e93c3>] dump_stack+0x63/0x90
Apr 27 21:05:18 backup kernel: [343463.805214] [<ffffffff81192fba>] warn_alloc_failed+0xfa/0x150
Apr 27 21:05:18 backup kernel: [343463.805217] [<ffffffff81196b01>] __alloc_pages_nodemask+0x341/0xb60
Apr 27 21:05:18 backup kernel: [343463.805220] [<ffffffff811e084c>] alloc_pages_current+0x8c/0x110
Apr 27 21:05:18 backup kernel: [343463.805222] [<ffffffff811e992f>] new_slab+0x28f/0x490
Apr 27 21:05:18 backup kernel: [343463.805224] [<ffffffff811ea97b>] ___slab_alloc+0x22b/0x460
Apr 27 21:05:18 backup kernel: [343463.805227] [<ffffffff814bf3cd>] ? alloc_indirect.isra.4+0x1d/0x50
Apr 27 21:05:18 backup kernel: [343463.805228] [<ffffffff814bf3cd>] ? alloc_indirect.isra.4+0x1d/0x50
Apr 27 21:05:18 backup kernel: [343463.805230] [<ffffffff811eabd0>] __slab_alloc+0x20/0x40
Apr 27 21:05:18 backup kernel: [343463.805231] [<ffffffff811ec555>] __kmalloc+0x1d5/0x250
Apr 27 21:05:18 backup kernel: [343463.805233] [<ffffffff814bf3cd>] alloc_indirect.isra.4+0x1d/0x50
Apr 27 21:05:18 backup kernel: [343463.805234] [<ffffffff814bfaaa>] virtqueue_add_sgs+0x2ca/0x3a0
Apr 27 21:05:18 backup kernel: [343463.805237] [<ffffffff81571b9d>] __virtblk_add_req+0xbd/0x220
Apr 27 21:05:18 backup kernel: [343463.805239] [<ffffffff8118fe05>] ? mempool_alloc_slab+0x15/0x20
Apr 27 21:05:18 backup kernel: [343463.805242] [<ffffffff813c0113>] ? blk_rq_map_sg+0x223/0x570
Apr 27 21:05:18 backup kernel: [343463.805243] [<ffffffff81571e82>] virtio_queue_rq+0x112/0x280
Apr 27 21:05:18 backup kernel: [343463.805245] [<ffffffff813c48c6>] __blk_mq_run_hw_queue+0x1d6/0x380
Apr 27 21:05:18 backup kernel: [343463.805247] [<ffffffff813c46ce>] blk_mq_run_hw_queue+0x8e/0xb0
Apr 27 21:05:18 backup kernel: [343463.805248] [<ffffffff813c5c46>] blk_mq_insert_requests+0xf6/0x1c0
Apr 27 21:05:18 backup kernel: [343463.805250] [<ffffffff813c6958>] blk_mq_flush_plug_list+0x128/0x150
Apr 27 21:05:18 backup kernel: [343463.805251] [<ffffffff813bb606>] blk_flush_plug_list+0xd6/0x240
Apr 27 21:05:18 backup kernel: [343463.805253] [<ffffffff813bbb7c>] blk_finish_plug+0x2c/0x40
Apr 27 21:05:18 backup kernel: [343463.805264] [<ffffffffc01a7b13>] run_scheduled_bios+0x363/0x530 [btrfs]
Apr 27 21:05:18 backup kernel: [343463.805275] [<ffffffffc01a7cf5>] pending_bios_fn+0x15/0x20 [btrfs]
Apr 27 21:05:18 backup kernel: [343463.805285] [<ffffffffc01b2e8a>] btrfs_scrubparity_helper+0xca/0x2f0 [btrfs]
Apr 27 21:05:18 backup kernel: [343463.805294] [<ffffffffc01b31be>] btrfs_submit_helper+0xe/0x10 [btrfs]
Apr 27 21:05:18 backup kernel: [343463.805297] [<ffffffff81099ff2>] process_one_work+0x162/0x480
Apr 27 21:05:18 backup kernel: [343463.805298] [<ffffffff8109a35b>] worker_thread+0x4b/0x4c0
Apr 27 21:05:18 backup kernel: [343463.805300] [<ffffffff8109a310>] ? process_one_work+0x480/0x480
Apr 27 21:05:18 backup kernel: [343463.805301] [<ffffffff8109a310>] ? process_one_work+0x480/0x480
Apr 27 21:05:18 backup kernel: [343463.805303] [<ffffffff810a0528>] kthread+0xd8/0xf0
Apr 27 21:05:18 backup kernel: [343463.805305] [<ffffffff810a0450>] ? kthread_create_on_node+0x1e0/0x1e0
Apr 27 21:05:18 backup kernel: [343463.805308] [<ffffffff8182488f>] ret_from_fork+0x3f/0x70
Apr 27 21:05:18 backup kernel: [343463.805309] [<ffffffff810a0450>] ? kthread_create_on_node+0x1e0/0x1e0
Apr 27 21:05:18 backup kernel: [343463.805310] Mem-Info:
Apr 27 21:05:18 backup kernel: [343463.805313] active_anon:43184 inactive_anon:51940 isolated_anon:0
Apr 27 21:05:18 backup kernel: [343463.805313] active_file:123451 inactive_file:249283 isolated_file:0
Apr 27 21:05:18 backup kernel: [343463.805313] unevictable:1752 dirty:32219 writeback:6149 unstable:0
Apr 27 21:05:18 backup kernel: [343463.805313] slab_reclaimable:18341 slab_unreclaimable:4095
Apr 27 21:05:18 backup kernel: [343463.805313] mapped:58659 shmem:79658 pagetables:1755 bounce:0
Apr 27 21:05:18 backup kernel: [343463.805313] free:12908 free_pcp:138 free_cma:0
Apr 27 21:05:18 backup kernel: [343463.805315] Node 0 DMA free:9992kB min:40kB low:48kB high:60kB active_anon:28kB inactive_anon:396kB active_file:256kB inactive_file:884kB unevictable:92kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:92kB dirty:936kB writeback:0kB mapped:60kB shmem:316kB slab_reclaimable:3424kB slab_unreclaimable:308kB kernel_stack:192kB pagetables:60kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2580 all_unreclaimable? no
Apr 27 21:05:18 backup kernel: [343463.805319] lowmem_reserve[]: 0 1981 1981 1981 1981
Apr 27 21:05:18 backup kernel: [343463.805321] Node 0 DMA32 free:41640kB min:5580kB low:6972kB high:8368kB active_anon:172708kB inactive_anon:207364kB active_file:493548kB inactive_file:996248kB unevictable:6916kB isolated(anon):0kB isolated(file):0kB present:2080760kB managed:2032544kB mlocked:6916kB dirty:127940kB writeback:24596kB mapped:234576kB shmem:318316kB slab_reclaimable:69940kB slab_unreclaimable:16072kB kernel_stack:3184kB pagetables:6960kB unstable:0kB bounce:0kB free_pcp:552kB local_pcp:224kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 27 21:05:18 backup kernel: [343463.805325] lowmem_reserve[]: 0 0 0 0 0
Apr 27 21:05:18 backup kernel: [343463.805327] Node 0 DMA: 17*4kB (UEH) 65*8kB (UMEH) 8*16kB (UMH) 24*32kB (UMEH) 21*64kB (UMEH) 14*128kB (UMH) 7*256kB (UMH) 1*512kB (H) 1*1024kB (H) 1*2048kB (H) 0*4096kB = 9996kB
Apr 27 21:05:18 backup kernel: [343463.805335] Node 0 DMA32: 60*4kB (H) 75*8kB (H) 64*16kB (H) 33*32kB (H) 44*64kB (H) 33*128kB (H) 20*256kB (H) 14*512kB (H) 7*1024kB (H) 6*2048kB (H) 0*4096kB = 41704kB
Apr 27 21:05:18 backup kernel: [343463.805343] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 27 21:05:18 backup kernel: [343463.805343] 453607 total pagecache pages
Apr 27 21:05:18 backup kernel: [343463.805345] 317 pages in swap cache
Apr 27 21:05:18 backup kernel: [343463.805346] Swap cache stats: add 4177, delete 3860, find 161061/161539
Apr 27 21:05:18 backup kernel: [343463.805346] Free swap = 1043312kB
Apr 27 21:05:18 backup kernel: [343463.805347] Total swap = 1046524kB
Apr 27 21:05:18 backup kernel: [343463.805348] 524188 pages RAM
Apr 27 21:05:18 backup kernel: [343463.805349] 0 pages HighMem/MovableOnly
Apr 27 21:05:18 backup kernel: [343463.805349] 12075 pages reserved
Apr 27 21:05:18 backup kernel: [343463.805350] 0 pages cma reserved
Apr 27 21:05:18 backup kernel: [343463.805350] 0 pages hwpoisoned
Apr 27 21:05:18 backup kernel: [343463.805352] SLUB: Unable to allocate memory on node -1 (gfp=0x2080020)
Apr 27 21:05:18 backup kernel: [343463.805353] cache: kmalloc-2048, object size: 2048, buffer size: 2048, default order: 3, min order: 0
Apr 27 21:05:18 backup kernel: [343463.805354] node 0: slabs: 39, objs: 470, free: 0
Apr 27 21:05:18 backup kernel: [343463.816748] kworker/1:1H: page allocation failure: order:0, mode:0x2204020
Apr 27 21:05:18 backup kernel: [343463.816752] CPU: 1 PID: 154 Comm: kworker/1:1H Not tainted 4.4.0-21-generic #37-Ubuntu
Apr 27 21:05:18 backup kernel: [343463.816753] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Apr 27 21:05:18 backup kernel: [343463.816758] Workqueue: kblockd blk_mq_run_work_fn
Apr 27 21:05:18 backup kernel: [343463.816759] 0000000000000086 00000000010bc024 ffff88003528f7e8 ffffffff813e93c3
Apr 27 21:05:18 backup kernel: [343463.816761] 0000000002204020 0000000000000000 ffff88003528f878 ffffffff81192fba
Apr 27 21:05:18 backup kernel: [343463.816763] ffff88007fff9d40 ffff88007fffb6e0 0000000000000000 ffffffff00000060
Apr 27 21:05:18 backup kernel: [343463.816764] Call Trace:
Apr 27 21:05:18 backup kernel: [343463.816768] [<ffffffff813e93c3>] dump_stack+0x63/0x90
Apr 27 21:05:18 backup kernel: [343463.816771] [<ffffffff81192fba>] warn_alloc_failed+0xfa/0x150
Apr 27 21:05:18 backup kernel: [343463.816774] [<ffffffff81196b01>] __alloc_pages_nodemask+0x341/0xb60
Apr 27 21:05:18 backup kernel: [343463.816777] [<ffffffff810d8787>] ? vprintk_emit+0x2d7/0x520
Apr 27 21:05:18 backup kernel: [343463.816780] [<ffffffff811e084c>] alloc_pages_current+0x8c/0x110
Apr 27 21:05:18 backup kernel: [343463.816782] [<ffffffff811e992f>] new_slab+0x28f/0x490
Apr 27 21:05:18 backup kernel: [343463.816783] [<ffffffff811ea97b>] ___slab_alloc+0x22b/0x460
Apr 27 21:05:18 backup kernel: [343463.816786] [<ffffffff814bf3cd>] ? alloc_indirect.isra.4+0x1d/0x50
Apr 27 21:05:18 backup kernel: [343463.816788] [<ffffffff810bc8d5>] ? update_sd_lb_stats+0x115/0x520
Apr 27 21:05:18 backup kernel: [343463.816790] [<ffffffff814bf3cd>] ? alloc_indirect.isra.4+0x1d/0x50
Apr 27 21:05:18 backup kernel: [343463.816791] [<ffffffff811eabd0>] __slab_alloc+0x20/0x40
Apr 27 21:05:18 backup kernel: [343463.816793] [<ffffffff811ec555>] __kmalloc+0x1d5/0x250
Apr 27 21:05:18 backup kernel: [343463.816794] [<ffffffff814bf3cd>] alloc_indirect.isra.4+0x1d/0x50
Apr 27 21:05:18 backup kernel: [343463.816796] [<ffffffff814bfaaa>] virtqueue_add_sgs+0x2ca/0x3a0
Apr 27 21:05:18 backup kernel: [343463.816798] [<ffffffff81571b9d>] __virtblk_add_req+0xbd/0x220
Apr 27 21:05:18 backup kernel: [343463.816801] [<ffffffff813c0113>] ? blk_rq_map_sg+0x223/0x570
Apr 27 21:05:18 backup kernel: [343463.816803] [<ffffffff81571e82>] virtio_queue_rq+0x112/0x280
Apr 27 21:05:18 backup kernel: [343463.816805] [<ffffffff813c48c6>] __blk_mq_run_hw_queue+0x1d6/0x380
Apr 27 21:05:18 backup kernel: [343463.816806] [<ffffffff813c4df2>] blk_mq_run_work_fn+0x12/0x20
Apr 27 21:05:18 backup kernel: [343463.816809] [<ffffffff81099ff2>] process_one_work+0x162/0x480
Apr 27 21:05:18 backup kernel: [343463.816810] [<ffffffff8109a35b>] worker_thread+0x4b/0x4c0
Apr 27 21:05:18 backup kernel: [343463.816812] [<ffffffff8109a310>] ? process_one_work+0x480/0x480
Apr 27 21:05:18 backup kernel: [343463.816813] [<ffffffff8109a310>] ? process_one_work+0x480/0x480
Apr 27 21:05:18 backup kernel: [343463.816815] [<ffffffff810a0528>] kthread+0xd8/0xf0
Apr 27 21:05:18 backup kernel: [343463.816816] [<ffffffff810a0450>] ? kthread_create_on_node+0x1e0/0x1e0
Apr 27 21:05:18 backup kernel: [343463.816819] [<ffffffff8182488f>] ret_from_fork+0x3f/0x70
Apr 27 21:05:18 backup kernel: [343463.816821] [<ffffffff810a0450>] ? kthread_create_on_node+0x1e0/0x1e0
Apr 27 21:05:18 backup kernel: [343463.816822] Mem-Info:
Apr 27 21:05:18 backup kernel: [343463.816825] active_anon:43184 inactive_anon:51940 isolated_anon:0
Apr 27 21:05:18 backup kernel: [343463.816825] active_file:123441 inactive_file:249312 isolated_file:0
Apr 27 21:05:18 backup kernel: [343463.816825] unevictable:1752 dirty:32397 writeback:5123 unstable:0
Apr 27 21:05:18 backup kernel: [343463.816825] slab_reclaimable:18360 slab_unreclaimable:4097
Apr 27 21:05:18 backup kernel: [343463.816825] mapped:58673 shmem:79658 pagetables:1755 bounce:0
Apr 27 21:05:18 backup kernel: [343463.816825] free:12867 free_pcp:107 free_cma:0
Apr 27 21:05:18 backup kernel: [343463.816827] Node 0 DMA free:9988kB min:40kB low:48kB high:60kB active_anon:28kB inactive_anon:396kB active_file:256kB inactive_file:884kB unevictable:92kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:92kB dirty:940kB writeback:0kB mapped:60kB shmem:316kB slab_reclaimable:3424kB slab_unreclaimable:316kB kernel_stack:192kB pagetables:60kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:532 all_unreclaimable? no
Apr 27 21:05:18 backup kernel: [343463.816831] lowmem_reserve[]: 0 1981 1981 1981 1981
Apr 27 21:05:18 backup kernel: [343463.816833] Node 0 DMA32 free:41480kB min:5580kB low:6972kB high:8368kB active_anon:172708kB inactive_anon:207364kB active_file:493508kB inactive_file:996364kB unevictable:6916kB isolated(anon):0kB isolated(file):0kB present:2080760kB managed:2032544kB mlocked:6916kB dirty:128648kB writeback:20492kB mapped:234632kB shmem:318316kB slab_reclaimable:70016kB slab_unreclaimable:16072kB kernel_stack:3184kB pagetables:6960kB unstable:0kB bounce:0kB free_pcp:428kB local_pcp:160kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 27 21:05:18 backup kernel: [343463.816837] lowmem_reserve[]: 0 0 0 0 0
Apr 27 21:05:18 backup kernel: [343463.816839] Node 0 DMA: 15*4kB (UEH) 65*8kB (UMEH) 8*16kB (UMH) 24*32kB (UMEH) 21*64kB (UMEH) 14*128kB (UMH) 7*256kB (UMH) 1*512kB (H) 1*1024kB (H) 1*2048kB (H) 0*4096kB = 9988kB
Apr 27 21:05:18 backup kernel: [343463.816847] Node 0 DMA32: 60*4kB (H) 75*8kB (H) 64*16kB (H) 29*32kB (H) 44*64kB (H) 33*128kB (H) 20*256kB (H) 14*512kB (H) 7*1024kB (H) 6*2048kB (H) 0*4096kB = 41576kB
Apr 27 21:05:18 backup kernel: [343463.816854] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 27 21:05:18 backup kernel: [343463.816855] 453618 total pagecache pages
Apr 27 21:05:18 backup kernel: [343463.816856] 317 pages in swap cache
Apr 27 21:05:18 backup kernel: [343463.816857] Swap cache stats: add 4177, delete 3860, find 161061/161539
Apr 27 21:05:18 backup kernel: [343463.816858] Free swap = 1043312kB
Apr 27 21:05:18 backup kernel: [343463.816859] Total swap = 1046524kB
Apr 27 21:05:18 backup kernel: [343463.816860] 524188 pages RAM
Apr 27 21:05:18 backup kernel: [343463.816860] 0 pages HighMem/MovableOnly
Apr 27 21:05:18 backup kernel: [343463.816861] 12075 pages reserved
Apr 27 21:05:18 backup kernel: [343463.816861] 0 pages cma reserved
Apr 27 21:05:18 backup kernel: [343463.816862] 0 pages hwpoisoned
Apr 27 21:05:18 backup kernel: [343463.816863] SLUB: Unable to allocate memory on node -1 (gfp=0x2080020)
Apr 27 21:05:18 backup kernel: [343463.816865] cache: kmalloc-2048, object size: 2048, buffer size: 2048, default order: 3, min order: 0
Apr 27 21:05:18 backup kernel: [343463.816866] node 0: slabs: 41, objs: 474, free: 0
Apr 27 21:05:18 backup kernel: [343463.816872] kworker/1:1H: page allocation failure: order:0, mode:0x2204020
Apr 27 21:05:18 backup kernel: [343463.816873] CPU: 1 PID: 154 Comm: kworker/1:1H Not tainted 4.4.0-21-generic #37-Ubuntu
Apr 27 21:05:18 backup kernel: [343463.816874] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Apr 27 21:05:18 backup kernel: [343463.816875] Workqueue: kblockd blk_mq_run_work_fn
Apr 27 21:05:18 backup kernel: [343463.816876] 0000000000000086 00000000010bc024 ffff88003528f7e8 ffffffff813e93c3
Apr 27 21:05:18 backup kernel: [343463.816878] 0000000002204020 0000000000000000 ffff88003528f878 ffffffff81192fba
Apr 27 21:05:18 backup kernel: [343463.816879] ffff88007fff9d40 ffff88007fffb6e0 0000000000000000 ffffffff00000060
Apr 27 21:05:18 backup kernel: [343463.816881] Call Trace:
Apr 27 21:05:18 backup kernel: [343463.816882] [<ffffffff813e93c3>] dump_stack+0x63/0x90
Apr 27 21:05:18 backup kernel: [343463.816884] [<ffffffff81192fba>] warn_alloc_failed+0xfa/0x150
Apr 27 21:05:18 backup kernel: [343463.816885] [<ffffffff81196b01>] __alloc_pages_nodemask+0x341/0xb60
Apr 27 21:05:18 backup kernel: [343463.816888] [<ffffffff810d8787>] ? vprintk_emit+0x2d7/0x520
Apr 27 21:05:18 backup kernel: [343463.816890] [<ffffffff811e084c>] alloc_pages_current+0x8c/0x110
Apr 27 21:05:18 backup kernel: [343463.816891] [<ffffffff811e992f>] new_slab+0x28f/0x490
Apr 27 21:05:18 backup kernel: [343463.816893] [<ffffffff811ea97b>] ___slab_alloc+0x22b/0x460
Apr 27 21:05:18 backup kernel: [343463.816894] [<ffffffff814bf3cd>] ? alloc_indirect.isra.4+0x1d/0x50
Apr 27 21:05:18 backup kernel: [343463.816896] [<ffffffff810bc8d5>] ? update_sd_lb_stats+0x115/0x520
Apr 27 21:05:18 backup kernel: [343463.816897] [<ffffffff814bf3cd>] ? alloc_indirect.isra.4+0x1d/0x50
Apr 27 21:05:18 backup kernel: [343463.816899] [<ffffffff811eabd0>] __slab_alloc+0x20/0x40
Apr 27 21:05:18 backup kernel: [343463.816900] [<ffffffff811ec555>] __kmalloc+0x1d5/0x250
Apr 27 21:05:18 backup kernel: [343463.816901] [<ffffffff814bf3cd>] alloc_indirect.isra.4+0x1d/0x50
Apr 27 21:05:18 backup kernel: [343463.816903] [<ffffffff814bfaaa>] virtqueue_add_sgs+0x2ca/0x3a0
Apr 27 21:05:18 backup kernel: [343463.816905] [<ffffffff81571b9d>] __virtblk_add_req+0xbd/0x220
Apr 27 21:05:18 backup kernel: [343463.816907] [<ffffffff813c0113>] ? blk_rq_map_sg+0x223/0x570
Apr 27 21:05:18 backup kernel: [343463.816908] [<ffffffff81571e82>] virtio_queue_rq+0x112/0x280
Apr 27 21:05:18 backup kernel: [343463.816910] [<ffffffff813c48c6>] __blk_mq_run_hw_queue+0x1d6/0x380
Apr 27 21:05:18 backup kernel: [343463.816911] [<ffffffff813c4df2>] blk_mq_run_work_fn+0x12/0x20
Apr 27 21:05:18 backup kernel: [343463.816915] [<ffffffff81099ff2>] process_one_work+0x162/0x480
Apr 27 21:05:18 backup kernel: [343463.816916] [<ffffffff8109a35b>] worker_thread+0x4b/0x4c0
Apr 27 21:05:18 backup kernel: [343463.816918] [<ffffffff8109a310>] ? process_one_work+0x480/0x480
Apr 27 21:05:18 backup kernel: [343463.816919] [<ffffffff8109a310>] ? process_one_work+0x480/0x480
Apr 27 21:05:18 backup kernel: [343463.816921] [<ffffffff810a0528>] kthread+0xd8/0xf0
Apr 27 21:05:18 backup kernel: [343463.816922] [<ffffffff810a0450>] ? kthread_create_on_node+0x1e0/0x1e0
Apr 27 21:05:18 backup kernel: [343463.816924] [<ffffffff8182488f>] ret_from_fork+0x3f/0x70
Apr 27 21:05:18 backup kernel: [343463.816925] [<ffffffff810a0450>] ? kthread_create_on_node+0x1e0/0x1e0
Apr 27 21:05:18 backup kernel: [343463.816926] Mem-Info:
Apr 27 21:05:18 backup kernel: [343463.816929] active_anon:43184 inactive_anon:51940 isolated_anon:0
Apr 27 21:05:18 backup kernel: [343463.816929] active_file:123441 inactive_file:249312 isolated_file:0
Apr 27 21:05:18 backup kernel: [343463.816929] unevictable:1752 dirty:32397 writeback:5123 unstable:0
Apr 27 21:05:18 backup kernel: [343463.816929] slab_reclaimable:18360 slab_unreclaimable:4097
Apr 27 21:05:18 backup kernel: [343463.816929] mapped:58673 shmem:79658 pagetables:1755 bounce:0
Apr 27 21:05:18 backup kernel: [343463.816929] free:12867 free_pcp:107 free_cma:0
Apr 27 21:05:18 backup kernel: [343463.816931] Node 0 DMA free:9988kB min:40kB low:48kB high:60kB active_anon:28kB inactive_anon:396kB active_file:256kB inactive_file:884kB unevictable:92kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:92kB dirty:940kB writeback:0kB mapped:60kB shmem:316kB slab_reclaimable:3424kB slab_unreclaimable:316kB kernel_stack:192kB pagetables:60kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:532 all_unreclaimable? no
Apr 27 21:05:18 backup kernel: [343463.816938] lowmem_reserve[]: 0 1981 1981 1981 1981
Apr 27 21:05:18 backup kernel: [343463.816940] Node 0 DMA32 free:41480kB min:5580kB low:6972kB high:8368kB active_anon:172708kB inactive_anon:207364kB active_file:493508kB inactive_file:996364kB unevictable:6916kB isolated(anon):0kB isolated(file):0kB present:2080760kB managed:2032544kB mlocked:6916kB dirty:128648kB writeback:20492kB mapped:234632kB shmem:318316kB slab_reclaimable:70016kB slab_unreclaimable:16072kB kernel_stack:3184kB pagetables:6960kB unstable:0kB bounce:0kB free_pcp:428kB local_pcp:160kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 27 21:05:18 backup kernel: [343463.816943] lowmem_reserve[]: 0 0 0 0 0
Apr 27 21:05:18 backup kernel: [343463.816945] Node 0 DMA: 15*4kB (UEH) 65*8kB (UMEH) 8*16kB (UMH) 24*32kB (UMEH) 21*64kB (UMEH) 14*128kB (UMH) 7*256kB (UMH) 1*512kB (H) 1*1024kB (H) 1*2048kB (H) 0*4096kB = 9988kB
Apr 27 21:05:18 backup kernel: [343463.816952] Node 0 DMA32: 60*4kB (H) 75*8kB (H) 64*16kB (H) 29*32kB (H) 44*64kB (H) 33*128kB (H) 20*256kB (H) 14*512kB (H) 7*1024kB (H) 6*2048kB (H) 0*4096kB = 41576kB
Apr 27 21:05:18 backup kernel: [343463.816960] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 27 21:05:18 backup kernel: [343463.816960] 453618 total pagecache pages
Apr 27 21:05:18 backup kernel: [343463.816961] 317 pages in swap cache
Apr 27 21:05:18 backup kernel: [343463.816962] Swap cache stats: add 4177, delete 3860, find 161061/161539
Apr 27 21:05:18 backup kernel: [343463.816963] Free swap = 1043312kB
Apr 27 21:05:18 backup kernel: [343463.816963] Total swap = 1046524kB
Apr 27 21:05:18 backup kernel: [343463.816964] 524188 pages RAM
Apr 27 21:05:18 backup kernel: [343463.816965] 0 pages HighMem/MovableOnly
Apr 27 21:05:18 backup kernel: [343463.816965] 12075 pages reserved
Apr 27 21:05:18 backup kernel: [343463.816966] 0 pages cma reserved
Apr 27 21:05:18 backup kernel: [343463.816966] 0 pages hwpoisoned

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-21-generic 4.4.0-21.37
ProcVersionSignature: Ubuntu 4.4.0-21.37-generic 4.4.6
Uname: Linux 4.4.0-21-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version k4.4.0-21-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
Date: Thu Apr 28 08:53:40 2016
HibernationDevice: RESUME=UUID=b13cd3f3-9607-4bcc-8ce7-78efe89b82b6
IwConfig:
eth0 no wireless extensions.

lo no wireless extensions.

tun0 no wireless extensions.
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Bochs Bochs
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 qxldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=UUID=0fdb77e4-84b1-4b4c-b681-f6c7b0fab4d2 ro elevator=deadline
RelatedPackageVersions:
linux-restricted-modules-4.4.0-21-generic N/A
linux-backports-modules-4.4.0-21-generic N/A
linux-firmware 1.157
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to xenial on 2016-04-21 (7 days ago)
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnBochs:pnBochs:pvr:cvnBochs:ct1:cvr:
dmi.product.name: Bochs
dmi.sys.vendor: Bochs

Tags:

Revision history for this message

MikeB (mikeb-timetrex) wrote on 2016-04-28:

AlsaDevices.txt Edit (375 bytes, text/plain; charset="utf-8")
CRDA.txt Edit (422 bytes, text/plain; charset="utf-8")
Card0.Codecs.codec.0.txt Edit (2.0 KiB, text/plain; charset="utf-8")
CurrentDmesg.txt Edit (35.6 KiB, text/plain; charset="utf-8")
Dependencies.txt Edit (2.6 KiB, text/plain; charset="utf-8")
JournalErrors.txt Edit (3.5 KiB, text/plain; charset="utf-8")
Lspci.txt Edit (7.1 KiB, text/plain; charset="utf-8")
PciMultimedia.txt Edit (700 bytes, text/plain; charset="utf-8")
ProcCpuinfo.txt Edit (1.2 KiB, text/plain; charset="utf-8")
ProcInterrupts.txt Edit (2.1 KiB, text/plain; charset="utf-8")
ProcModules.txt Edit (2.0 KiB, text/plain; charset="utf-8")
UdevDb.txt Edit (73.6 KiB, text/plain; charset="utf-8")
WifiSyslog.txt Edit (40.0 KiB, text/plain; charset="utf-8")

Revision history for this message

Brad Figg (brad-figg) wrote on 2016-04-28: Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status:	New → Confirmed

Revision history for this message

Joseph Salisbury (jsalisbury) wrote on 2016-05-05:

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.6 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-rc6-wily/

Changed in linux (Ubuntu):
importance:	Undecided → Medium
status:	Confirmed → Incomplete

Revision history for this message

MikeB (mikeb-timetrex) wrote on 2016-05-05:

Okay, I have installed 4.6 and ran a manual backup job, no error messages yet.

I will let it run overnight and let you know the results tomorrow.

Revision history for this message

MikeB (mikeb-timetrex) wrote on 2016-05-07:

I can confirm after a couple days running 4.6.0-040600rc6-generic, the error messages no longer appear.

Revision history for this message

MikeB (mikeb-timetrex) wrote on 2016-05-11:

Download full text (21.1 KiB)

Spoke too soon, though it may be a slightly different issue, but still appears to be BTRFS related. This is still on 4.6.0-040600rc6-generic, the memory on this server is 1GB.

[121025.879323] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC)
[121025.879324]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[121025.879324]   node 0: slabs: 148, objs: 2368, free: 0
[121025.879517] kworker/u4:2: page allocation failure: order:0, mode:0x2200020(GFP_NOWAIT|__GFP_HIGH|__GFP_NOTRACK)
[121025.879519] CPU: 1 PID: 9038 Comm: kworker/u4:2 Not tainted 4.6.0-040600rc6-generic #201605012031
[121025.879520] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[121025.879572] Workqueue: btrfs-worker btrfs_worker_helper [btrfs]
[121025.879574]  0000000000000086 00000000d14aff48 ffff88003fd03960 ffffffff813ef3b3
[121025.879576]  0000000000000000 0000000000000000 ffff88003fd039f0 ffffffff8119b3d4
[121025.879577]  0220002079d790e3 ffff88003fffb6e0 ffffffff82110ae0 ffffffff00000060
[121025.879579] Call Trace:
[121025.879580]  <IRQ>  [<ffffffff813ef3b3>] dump_stack+0x63/0x90
[121025.879584]  [<ffffffff8119b3d4>] warn_alloc_failed+0x104/0x160
[121025.879586]  [<ffffffff8119f059>] __alloc_pages_nodemask+0x4f9/0xc30
[121025.879588]  [<ffffffff810dc3f1>] ? vprintk_emit+0x2a1/0x4d0
[121025.879590]  [<ffffffff811ef0bc>] alloc_pages_current+0x8c/0x110
[121025.879592]  [<ffffffff811f9252>] new_slab+0x452/0x5c0
[121025.879593]  [<ffffffff811fa2c9>] ___slab_alloc+0x249/0x480
[121025.879595]  [<ffffffff81723aea>] ? __build_skb+0x2a/0xe0
[121025.879596]  [<ffffffff81723aea>] ? __build_skb+0x2a/0xe0
[121025.879598]  [<ffffffff811fa520>] __slab_alloc+0x20/0x40
[121025.879599]  [<ffffffff811fa6d2>] kmem_cache_alloc+0x192/0x1d0
[121025.879600]  [<ffffffff81723aea>] ? __build_skb+0x2a/0xe0
[121025.879601]  [<ffffffff81723aea>] __build_skb+0x2a/0xe0
[121025.879603]  [<ffffffff81723dc8>] __napi_alloc_skb+0x98/0xd0
[121025.879605]  [<ffffffff81603da6>] page_to_skb.isra.28+0x66/0x310
[121025.879607]  [<ffffffff8160617f>] virtnet_receive+0x2df/0xa00
[121025.879609]  [<ffffffff810a8b28>] ? update_rq_clock.part.81+0x18/0x40
[121025.879610]  [<ffffffff816068bd>] virtnet_poll+0x1d/0x80
[121025.879612]  [<ffffffff81733705>] net_rx_action+0x225/0x360
[121025.879615]  [<ffffffff818433a6>] __do_softirq+0xf6/0x27e
[121025.879616]  [<ffffffff81088483>] irq_exit+0xa3/0xb0
[121025.879617]  [<ffffffff81843104>] do_IRQ+0x54/0xd0
[121025.879619]  [<ffffffff81841202>] common_interrupt+0x82/0x82
[121025.879619]  <EOI>  [<ffffffff811ad389>] ? shrink_inactive_list+0x199/0x500
[121025.879623]  [<ffffffff811ad5f5>] ? shrink_inactive_list+0x405/0x500
[121025.879626]  [<ffffffff811ae09d>] shrink_zone_memcg+0x5ad/0x7a0
[121025.879627]  [<ffffffff811a9500>] ? shrink_slab.part.39+0x240/0x3e0
[121025.879634]  [<ffffffff8111975c>] ? css_next_descendant_pre+0x4c/0x60
[121025.879636]  [<ffffffff8111975c>] ? css_next_descendant_pre+0x4c/0x60
[121025.879638]  [<ffffffff811ae370>] shrink_zone+0xe0/0x300
[121025.879640]  [<ffffffff811ae70e>] do_try_to_free_pages+0x17e/0x430
[121025.879642]  [<ffffffff811aea8e>] try_to_free_pages+0xce/0x180
[121025.879644]  [<ffffffff8119f124>] __alloc_pages_nodemask+0x5c4/0xc30
[121025.879646]  [<ffffffff810b7e09>] ? set_next_entity+0x49/0xd0
[121025.879647]  [<ffffffff811ef0bc>] alloc_pages_current+0x8c/0x110
[121025.879649]  [<ffffffff811f9252>] new_slab+0x452/0x5c0
[121025.879651]  [<ffffffff811fa2c9>] ___slab_alloc+0x249/0x480
[121025.879661]  [<ffffffffc016482a>] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs]
[121025.879668]  [<ffffffff813afdd7>] ? chksum_update+0x17/0x20
[121025.879669]  [<ffffffff813a6910>] ? crypto_shash_update+0x30/0xe0
[121025.879679]  [<ffffffffc016482a>] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs]
[121025.879681]  [<ffffffff811fa520>] __slab_alloc+0x20/0x40
[121025.879682]  [<ffffffff811fbea4>] __kmalloc+0x1c4/0x230
[121025.879691]  [<ffffffffc0167500>] ? run_one_async_free+0x20/0x20 [btrfs]
[121025.879700]  [<ffffffffc016482a>] btrfs_csum_one_bio+0x6a/0x430 [btrfs]
[121025.879709]  [<ffffffffc0167500>] ? run_one_async_free+0x20/0x20 [btrfs]
[121025.879718]  [<ffffffffc017426d>] __btrfs_submit_bio_start+0x1d/0x30 [btrfs]
[121025.879727]  [<ffffffffc01672a6>] run_one_async_start+0x26/0x30 [btrfs]
[121025.879738]  [<ffffffffc01a489a>] btrfs_scrubparity_helper+0xca/0x2e0 [btrfs]
[121025.879749]  [<ffffffffc01a4bfe>] btrfs_worker_helper+0xe/0x10 [btrfs]
[121025.879753]  [<ffffffff8109cad8>] process_one_work+0x168/0x480
[121025.879755]  [<ffffffff8109ce3b>] worker_thread+0x4b/0x500
[121025.879757]  [<ffffffff8109cdf0>] ? process_one_work+0x480/0x480
[121025.879758]  [<ffffffff810a3228>] kthread+0xd8/0xf0
[121025.879760]  [<ffffffff81840ae2>] ret_from_fork+0x22/0x40
[121025.879763]  [<ffffffff810a3150>] ? kthread_create_on_node+0x1a0/0x1a0
[121025.879764] Mem-Info:
[121025.879766] active_anon:18131 inactive_anon:23781 isolated_anon:0
                 active_file:84105 inactive_file:84587 isolated_file:19
                 unevictable:1762 dirty:869 writeback:23497 unstable:0
                 slab_reclaimable:25801 slab_unreclaimable:5224
                 mapped:33650 shmem:34217 pagetables:1605 bounce:0
                 free:1897 free_pcp:20 free_cma:0
[121025.879769] Node 0 DMA free:3920kB min:60kB low:72kB high:84kB active_anon:188kB inactive_anon:1324kB active_file:2540kB inactive_file:2664kB unevictable:112kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:112kB dirty:0kB writeback:80kB mapped:884kB shmem:1356kB slab_reclaimable:2296kB slab_unreclaimable:284kB kernel_stack:64kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[121025.879772] lowmem_reserve[]: 0 973 973 973 973
[121025.879775] Node 0 DMA32 free:3668kB min:3824kB low:4820kB high:5816kB active_anon:72336kB inactive_anon:93800kB active_file:333880kB inactive_file:335684kB unevictable:6936kB isolated(anon):0kB isolated(file):76kB present:1032184kB managed:1000480kB mlocked:6936kB dirty:3476kB writeback:93908kB mapped:133716kB shmem:135512kB slab_reclaimable:100908kB slab_unreclaimable:20612kB kernel_stack:3296kB pagetables:6340kB unstable:0kB bounce:0kB free_pcp:80kB local_pcp:80kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
[121025.879779] lowmem_reserve[]: 0 0 0 0 0
[121025.879781] Node 0 DMA: 361*4kB (UME) 59*8kB (UEH) 1*16kB (H) 0*32kB 1*64kB (H) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 3916kB
[121025.879788] Node 0 DMA32: 836*4kB (UMH) 22*8kB (UH) 16*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3776kB
[121025.879794] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[121025.879795] 204488 total pagecache pages
[121025.879796] 675 pages in swap cache
[121025.879796] Swap cache stats: add 42167, delete 41492, find 1069663/1070758
[121025.879797] Free swap  = 897344kB
[121025.879798] Total swap = 1046524kB
[121025.879799] 262044 pages RAM
[121025.879800] 0 pages HighMem/MovableOnly
[121025.879800] 7947 pages reserved
[121025.879801] 0 pages cma reserved
[121025.879801] 0 pages hwpoisoned
[121025.879803] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC)
[121025.879804]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[121025.879804]   node 0: slabs: 148, objs: 2368, free: 0
[121025.879807] kworker/u4:2: page allocation failure: order:0, mode:0x2200020(GFP_NOWAIT|__GFP_HIGH|__GFP_NOTRACK)
[121025.879808] CPU: 1 PID: 9038 Comm: kworker/u4:2 Not tainted 4.6.0-040600rc6-generic #201605012031
[121025.879809] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[121025.879819] Workqueue: btrfs-worker btrfs_worker_helper [btrfs]
[121025.879820]  0000000000000086 00000000d14aff48 ffff88003fd03960 ffffffff813ef3b3
[121025.879822]  0000000000000000 0000000000000000 ffff88003fd039f0 ffffffff8119b3d4
[121025.879823]  02200020d14aff48 ffff88003fffb6e0 ffffffff82110ae0 ffffffff00000060
[121025.879824] Call Trace:
[121025.879825]  <IRQ>  [<ffffffff813ef3b3>] dump_stack+0x63/0x90
[121025.879829]  [<ffffffff8119b3d4>] warn_alloc_failed+0x104/0x160
[121025.879831]  [<ffffffff8119f059>] __alloc_pages_nodemask+0x4f9/0xc30
[121025.879833]  [<ffffffff810dc3f1>] ? vprintk_emit+0x2a1/0x4d0
[121025.879834]  [<ffffffff811ef0bc>] alloc_pages_current+0x8c/0x110
[121025.879836]  [<ffffffff811f9252>] new_slab+0x452/0x5c0
[121025.879837]  [<ffffffff811fa2c9>] ___slab_alloc+0x249/0x480
[121025.879839]  [<ffffffff81723aea>] ? __build_skb+0x2a/0xe0
[121025.879841]  [<ffffffff81723aea>] ? __build_skb+0x2a/0xe0
[121025.879842]  [<ffffffff811fa520>] __slab_alloc+0x20/0x40
[121025.879843]  [<ffffffff811fa6d2>] kmem_cache_alloc+0x192/0x1d0
[121025.879844]  [<ffffffff81723aea>] ? __build_skb+0x2a/0xe0
[121025.879846]  [<ffffffff81723aea>] __build_skb+0x2a/0xe0
[121025.879849]  [<ffffffff81723dc8>] __napi_alloc_skb+0x98/0xd0
[121025.879857]  [<ffffffff81603da6>] page_to_skb.isra.28+0x66/0x310
[121025.879859]  [<ffffffff8160617f>] virtnet_receive+0x2df/0xa00
[121025.879861]  [<ffffffff810a8b28>] ? update_rq_clock.part.81+0x18/0x40
[121025.879863]  [<ffffffff816068bd>] virtnet_poll+0x1d/0x80
[121025.879864]  [<ffffffff81733705>] net_rx_action+0x225/0x360
[121025.879866]  [<ffffffff818433a6>] __do_softirq+0xf6/0x27e
[121025.879867]  [<ffffffff81088483>] irq_exit+0xa3/0xb0
[121025.879869]  [<ffffffff81843104>] do_IRQ+0x54/0xd0
[121025.879871]  [<ffffffff81841202>] common_interrupt+0x82/0x82
[121025.879871]  <EOI>  [<ffffffff811ad389>] ? shrink_inactive_list+0x199/0x500
[121025.879875]  [<ffffffff811ad5f5>] ? shrink_inactive_list+0x405/0x500
[121025.879877]  [<ffffffff811ae09d>] shrink_zone_memcg+0x5ad/0x7a0
[121025.879879]  [<ffffffff811a9500>] ? shrink_slab.part.39+0x240/0x3e0
[121025.879881]  [<ffffffff8111975c>] ? css_next_descendant_pre+0x4c/0x60
[121025.879882]  [<ffffffff8111975c>] ? css_next_descendant_pre+0x4c/0x60
[121025.879885]  [<ffffffff811ae370>] shrink_zone+0xe0/0x300
[121025.879886]  [<ffffffff811ae70e>] do_try_to_free_pages+0x17e/0x430
[121025.879888]  [<ffffffff811aea8e>] try_to_free_pages+0xce/0x180
[121025.879890]  [<ffffffff8119f124>] __alloc_pages_nodemask+0x5c4/0xc30
[121025.879892]  [<ffffffff810b7e09>] ? set_next_entity+0x49/0xd0
[121025.879894]  [<ffffffff811ef0bc>] alloc_pages_current+0x8c/0x110
[121025.879895]  [<ffffffff811f9252>] new_slab+0x452/0x5c0
[121025.879896]  [<ffffffff811fa2c9>] ___slab_alloc+0x249/0x480
[121025.879905]  [<ffffffffc016482a>] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs]
[121025.879908]  [<ffffffff813afdd7>] ? chksum_update+0x17/0x20
[121025.879909]  [<ffffffff813a6910>] ? crypto_shash_update+0x30/0xe0
[121025.879917]  [<ffffffffc016482a>] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs]
[121025.879919]  [<ffffffff811fa520>] __slab_alloc+0x20/0x40
[121025.879921]  [<ffffffff811fbea4>] __kmalloc+0x1c4/0x230
[121025.879929]  [<ffffffffc0167500>] ? run_one_async_free+0x20/0x20 [btrfs]
[121025.879938]  [<ffffffffc016482a>] btrfs_csum_one_bio+0x6a/0x430 [btrfs]
[121025.879947]  [<ffffffffc0167500>] ? run_one_async_free+0x20/0x20 [btrfs]
[121025.879956]  [<ffffffffc017426d>] __btrfs_submit_bio_start+0x1d/0x30 [btrfs]
[121025.879965]  [<ffffffffc01672a6>] run_one_async_start+0x26/0x30 [btrfs]
[121025.879975]  [<ffffffffc01a489a>] btrfs_scrubparity_helper+0xca/0x2e0 [btrfs]
[121025.879985]  [<ffffffffc01a4bfe>] btrfs_worker_helper+0xe/0x10 [btrfs]
[121025.879988]  [<ffffffff8109cad8>] process_one_work+0x168/0x480
[121025.879990]  [<ffffffff8109ce3b>] worker_thread+0x4b/0x500
[121025.879992]  [<ffffffff8109cdf0>] ? process_one_work+0x480/0x480
[121025.879993]  [<ffffffff810a3228>] kthread+0xd8/0xf0
[121025.879995]  [<ffffffff81840ae2>] ret_from_fork+0x22/0x40
[121025.879996]  [<ffffffff810a3150>] ? kthread_create_on_node+0x1a0/0x1a0
[121025.879998] Mem-Info:
[121025.880000] active_anon:18131 inactive_anon:23781 isolated_anon:0
                 active_file:84105 inactive_file:84587 isolated_file:19
                 unevictable:1762 dirty:869 writeback:23497 unstable:0
                 slab_reclaimable:25801 slab_unreclaimable:5224
                 mapped:33650 shmem:34217 pagetables:1605 bounce:0
                 free:1897 free_pcp:20 free_cma:0
[121025.880002] Node 0 DMA free:3920kB min:60kB low:72kB high:84kB active_anon:188kB inactive_anon:1324kB active_file:2540kB inactive_file:2664kB unevictable:112kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:112kB dirty:0kB writeback:80kB mapped:884kB shmem:1356kB slab_reclaimable:2296kB slab_unreclaimable:284kB kernel_stack:64kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[121025.880006] lowmem_reserve[]: 0 973 973 973 973
[121025.880008] Node 0 DMA32 free:3668kB min:3824kB low:4820kB high:5816kB active_anon:72336kB inactive_anon:93800kB active_file:333880kB inactive_file:335684kB unevictable:6936kB isolated(anon):0kB isolated(file):76kB present:1032184kB managed:1000480kB mlocked:6936kB dirty:3476kB writeback:93908kB mapped:133716kB shmem:135512kB slab_reclaimable:100908kB slab_unreclaimable:20612kB kernel_stack:3296kB pagetables:6340kB unstable:0kB bounce:0kB free_pcp:80kB local_pcp:80kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
[121025.880014] lowmem_reserve[]: 0 0 0 0 0
[121025.880016] Node 0 DMA: 361*4kB (UME) 59*8kB (UEH) 1*16kB (H) 0*32kB 1*64kB (H) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 3916kB
[121025.880023] Node 0 DMA32: 836*4kB (UMH) 22*8kB (UH) 16*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3776kB
[121025.880029] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[121025.880030] 204488 total pagecache pages
[121025.880030] 675 pages in swap cache
[121025.880031] Swap cache stats: add 42167, delete 41492, find 1069663/1070758
[121025.880032] Free swap  = 897344kB
[121025.880033] Total swap = 1046524kB
[121025.880034] 262044 pages RAM
[121025.880035] 0 pages HighMem/MovableOnly
[121025.880035] 7947 pages reserved
[121025.880036] 0 pages cma reserved
[121025.880036] 0 pages hwpoisoned
[121025.880038] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC)
[121025.880039]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[121025.880040]   node 0: slabs: 148, objs: 2368, free: 0
[121025.880042] kworker/u4:2: page allocation failure: order:0, mode:0x2200020(GFP_NOWAIT|__GFP_HIGH|__GFP_NOTRACK)
[121025.880044] CPU: 1 PID: 9038 Comm: kworker/u4:2 Not tainted 4.6.0-040600rc6-generic #201605012031
[121025.880044] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[121025.880054] Workqueue: btrfs-worker btrfs_worker_helper [btrfs]
[121025.880056]  0000000000000086 00000000d14aff48 ffff88003fd03960 ffffffff813ef3b3
[121025.880057]  0000000000000000 0000000000000000 ffff88003fd039f0 ffffffff8119b3d4
[121025.880059]  02200020d14aff48 ffff88003fffb6e0 ffffffff82110ae0 ffffffff00000060
[121025.880060] Call Trace:
[121025.880061]  <IRQ>  [<ffffffff813ef3b3>] dump_stack+0x63/0x90
[121025.880064]  [<ffffffff8119b3d4>] warn_alloc_failed+0x104/0x160
[121025.880066]  [<ffffffff8119f059>] __alloc_pages_nodemask+0x4f9/0xc30
[121025.880068]  [<ffffffff810dc3f1>] ? vprintk_emit+0x2a1/0x4d0
[121025.880070]  [<ffffffff811ef0bc>] alloc_pages_current+0x8c/0x110
[121025.880071]  [<ffffffff811f9252>] new_slab+0x452/0x5c0
[121025.880073]  [<ffffffff811fa2c9>] ___slab_alloc+0x249/0x480
[121025.880075]  [<ffffffff81723aea>] ? __build_skb+0x2a/0xe0
[121025.880076]  [<ffffffff81723aea>] ? __build_skb+0x2a/0xe0
[121025.880077]  [<ffffffff811fa520>] __slab_alloc+0x20/0x40
[121025.880079]  [<ffffffff811fa6d2>] kmem_cache_alloc+0x192/0x1d0
[121025.880080]  [<ffffffff81723aea>] ? __build_skb+0x2a/0xe0
[121025.880082]  [<ffffffff81723aea>] __build_skb+0x2a/0xe0
[121025.880083]  [<ffffffff81723dc8>] __napi_alloc_skb+0x98/0xd0
[121025.880085]  [<ffffffff81603da6>] page_to_skb.isra.28+0x66/0x310
[121025.880087]  [<ffffffff8160617f>] virtnet_receive+0x2df/0xa00
[121025.880088]  [<ffffffff810a8b28>] ? update_rq_clock.part.81+0x18/0x40
[121025.880091]  [<ffffffff816068bd>] virtnet_poll+0x1d/0x80
[121025.880092]  [<ffffffff81733705>] net_rx_action+0x225/0x360
[121025.880094]  [<ffffffff818433a6>] __do_softirq+0xf6/0x27e
[121025.880095]  [<ffffffff81088483>] irq_exit+0xa3/0xb0
[121025.880097]  [<ffffffff81843104>] do_IRQ+0x54/0xd0
[121025.880098]  [<ffffffff81841202>] common_interrupt+0x82/0x82
[121025.880099]  <EOI>  [<ffffffff811ad389>] ? shrink_inactive_list+0x199/0x500
[121025.880103]  [<ffffffff811ad5f5>] ? shrink_inactive_list+0x405/0x500
[121025.880105]  [<ffffffff811ae09d>] shrink_zone_memcg+0x5ad/0x7a0
[121025.880106]  [<ffffffff811a9500>] ? shrink_slab.part.39+0x240/0x3e0
[121025.880109]  [<ffffffff8111975c>] ? css_next_descendant_pre+0x4c/0x60
[121025.880110]  [<ffffffff8111975c>] ? css_next_descendant_pre+0x4c/0x60
[121025.880112]  [<ffffffff811ae370>] shrink_zone+0xe0/0x300
[121025.880116]  [<ffffffff811ae70e>] do_try_to_free_pages+0x17e/0x430
[121025.880119]  [<ffffffff811aea8e>] try_to_free_pages+0xce/0x180
[121025.880120]  [<ffffffff8119f124>] __alloc_pages_nodemask+0x5c4/0xc30
[121025.880122]  [<ffffffff810b7e09>] ? set_next_entity+0x49/0xd0
[121025.880123]  [<ffffffff811ef0bc>] alloc_pages_current+0x8c/0x110
[121025.880125]  [<ffffffff811f9252>] new_slab+0x452/0x5c0
[121025.880127]  [<ffffffff811fa2c9>] ___slab_alloc+0x249/0x480
[121025.880136]  [<ffffffffc016482a>] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs]
[121025.880138]  [<ffffffff813afdd7>] ? chksum_update+0x17/0x20
[121025.880139]  [<ffffffff813a6910>] ? crypto_shash_update+0x30/0xe0
[121025.880148]  [<ffffffffc016482a>] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs]
[121025.880149]  [<ffffffff811fa520>] __slab_alloc+0x20/0x40
[121025.880151]  [<ffffffff811fbea4>] __kmalloc+0x1c4/0x230
[121025.880159]  [<ffffffffc0167500>] ? run_one_async_free+0x20/0x20 [btrfs]
[121025.880168]  [<ffffffffc016482a>] btrfs_csum_one_bio+0x6a/0x430 [btrfs]
[121025.880177]  [<ffffffffc0167500>] ? run_one_async_free+0x20/0x20 [btrfs]
[121025.880186]  [<ffffffffc017426d>] __btrfs_submit_bio_start+0x1d/0x30 [btrfs]
[121025.880195]  [<ffffffffc01672a6>] run_one_async_start+0x26/0x30 [btrfs]
[121025.880205]  [<ffffffffc01a489a>] btrfs_scrubparity_helper+0xca/0x2e0 [btrfs]
[121025.880215]  [<ffffffffc01a4bfe>] btrfs_worker_helper+0xe/0x10 [btrfs]
[121025.880217]  [<ffffffff8109cad8>] process_one_work+0x168/0x480
[121025.880219]  [<ffffffff8109ce3b>] worker_thread+0x4b/0x500
[121025.880221]  [<ffffffff8109cdf0>] ? process_one_work+0x480/0x480
[121025.880222]  [<ffffffff810a3228>] kthread+0xd8/0xf0
[121025.880224]  [<ffffffff81840ae2>] ret_from_fork+0x22/0x40
[121025.880226]  [<ffffffff810a3150>] ? kthread_create_on_node+0x1a0/0x1a0
[121025.880228] Mem-Info:
[121025.880230] active_anon:18131 inactive_anon:23781 isolated_anon:0
                 active_file:84105 inactive_file:84587 isolated_file:19
                 unevictable:1762 dirty:869 writeback:23497 unstable:0
                 slab_reclaimable:25801 slab_unreclaimable:5224
                 mapped:33650 shmem:34217 pagetables:1605 bounce:0
                 free:1897 free_pcp:21 free_cma:0
[121025.880232] Node 0 DMA free:3920kB min:60kB low:72kB high:84kB active_anon:188kB inactive_anon:1324kB active_file:2540kB inactive_file:2664kB unevictable:112kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:112kB dirty:0kB writeback:80kB mapped:884kB shmem:1356kB slab_reclaimable:2296kB slab_unreclaimable:284kB kernel_stack:64kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[121025.880236] lowmem_reserve[]: 0 973 973 973 973
[121025.880239] Node 0 DMA32 free:3668kB min:3824kB low:4820kB high:5816kB active_anon:72336kB inactive_anon:93800kB active_file:333880kB inactive_file:335684kB unevictable:6936kB isolated(anon):0kB isolated(file):76kB present:1032184kB managed:1000480kB mlocked:6936kB dirty:3476kB writeback:93908kB mapped:133716kB shmem:135512kB slab_reclaimable:100908kB slab_unreclaimable:20612kB kernel_stack:3296kB pagetables:6340kB unstable:0kB bounce:0kB free_pcp:84kB local_pcp:84kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
[121025.880242] lowmem_reserve[]: 0 0 0 0 0
[121025.880244] Node 0 DMA: 361*4kB (UME) 59*8kB (UEH) 1*16kB (H) 0*32kB 1*64kB (H) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 3916kB
[121025.880251] Node 0 DMA32: 836*4kB (UMH) 22*8kB (UH) 16*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3776kB
[121025.880258] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[121025.880258] 204488 total pagecache pages
[121025.880259] 675 pages in swap cache
[121025.880260] Swap cache stats: add 42167, delete 41492, find 1069663/1070758
[121025.880261] Free swap  = 897344kB
[121025.880261] Total swap = 1046524kB
[121025.880262] 262044 pages RAM
[121025.880262] 0 pages HighMem/MovableOnly
[121025.880263] 7947 pages reserved
[121025.880264] 0 pages cma reserved
[121025.880266] 0 pages hwpoisoned
[121025.880267] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC)
[121025.880268]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0
[121025.880269]   node 0: slabs: 148, objs: 2368, free: 0

Revision history for this message

Thomas Mayer (thomas303) wrote on 2016-06-23:

As MikeB pointed out, the issue still exists with latest mainline kernel. Therefore, I added Tag "kernel-bug-exists-upstream" and changed status to "confirmed".

I know of another user with the same problem. He was also testing against latest mainline kernel with no success.

tags:	added: kernel-bug-exists-upstream
Changed in linux (Ubuntu):
status:	Incomplete → Confirmed

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulinux package

BTRFS Page allocation failure with Bacula.

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
linux package