Spoke too soon, though it may be a slightly different issue, but still appears to be BTRFS related. This is still on 4.6.0-040600rc6-generic, the memory on this server is 1GB. [121025.879323] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [121025.879324] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [121025.879324] node 0: slabs: 148, objs: 2368, free: 0 [121025.879517] kworker/u4:2: page allocation failure: order:0, mode:0x2200020(GFP_NOWAIT|__GFP_HIGH|__GFP_NOTRACK) [121025.879519] CPU: 1 PID: 9038 Comm: kworker/u4:2 Not tainted 4.6.0-040600rc6-generic #201605012031 [121025.879520] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [121025.879572] Workqueue: btrfs-worker btrfs_worker_helper [btrfs] [121025.879574] 0000000000000086 00000000d14aff48 ffff88003fd03960 ffffffff813ef3b3 [121025.879576] 0000000000000000 0000000000000000 ffff88003fd039f0 ffffffff8119b3d4 [121025.879577] 0220002079d790e3 ffff88003fffb6e0 ffffffff82110ae0 ffffffff00000060 [121025.879579] Call Trace: [121025.879580] [] dump_stack+0x63/0x90 [121025.879584] [] warn_alloc_failed+0x104/0x160 [121025.879586] [] __alloc_pages_nodemask+0x4f9/0xc30 [121025.879588] [] ? vprintk_emit+0x2a1/0x4d0 [121025.879590] [] alloc_pages_current+0x8c/0x110 [121025.879592] [] new_slab+0x452/0x5c0 [121025.879593] [] ___slab_alloc+0x249/0x480 [121025.879595] [] ? __build_skb+0x2a/0xe0 [121025.879596] [] ? __build_skb+0x2a/0xe0 [121025.879598] [] __slab_alloc+0x20/0x40 [121025.879599] [] kmem_cache_alloc+0x192/0x1d0 [121025.879600] [] ? __build_skb+0x2a/0xe0 [121025.879601] [] __build_skb+0x2a/0xe0 [121025.879603] [] __napi_alloc_skb+0x98/0xd0 [121025.879605] [] page_to_skb.isra.28+0x66/0x310 [121025.879607] [] virtnet_receive+0x2df/0xa00 [121025.879609] [] ? update_rq_clock.part.81+0x18/0x40 [121025.879610] [] virtnet_poll+0x1d/0x80 [121025.879612] [] net_rx_action+0x225/0x360 [121025.879615] [] __do_softirq+0xf6/0x27e [121025.879616] [] irq_exit+0xa3/0xb0 [121025.879617] [] do_IRQ+0x54/0xd0 [121025.879619] [] common_interrupt+0x82/0x82 [121025.879619] [] ? shrink_inactive_list+0x199/0x500 [121025.879623] [] ? shrink_inactive_list+0x405/0x500 [121025.879626] [] shrink_zone_memcg+0x5ad/0x7a0 [121025.879627] [] ? shrink_slab.part.39+0x240/0x3e0 [121025.879634] [] ? css_next_descendant_pre+0x4c/0x60 [121025.879636] [] ? css_next_descendant_pre+0x4c/0x60 [121025.879638] [] shrink_zone+0xe0/0x300 [121025.879640] [] do_try_to_free_pages+0x17e/0x430 [121025.879642] [] try_to_free_pages+0xce/0x180 [121025.879644] [] __alloc_pages_nodemask+0x5c4/0xc30 [121025.879646] [] ? set_next_entity+0x49/0xd0 [121025.879647] [] alloc_pages_current+0x8c/0x110 [121025.879649] [] new_slab+0x452/0x5c0 [121025.879651] [] ___slab_alloc+0x249/0x480 [121025.879661] [] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs] [121025.879668] [] ? chksum_update+0x17/0x20 [121025.879669] [] ? crypto_shash_update+0x30/0xe0 [121025.879679] [] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs] [121025.879681] [] __slab_alloc+0x20/0x40 [121025.879682] [] __kmalloc+0x1c4/0x230 [121025.879691] [] ? run_one_async_free+0x20/0x20 [btrfs] [121025.879700] [] btrfs_csum_one_bio+0x6a/0x430 [btrfs] [121025.879709] [] ? run_one_async_free+0x20/0x20 [btrfs] [121025.879718] [] __btrfs_submit_bio_start+0x1d/0x30 [btrfs] [121025.879727] [] run_one_async_start+0x26/0x30 [btrfs] [121025.879738] [] btrfs_scrubparity_helper+0xca/0x2e0 [btrfs] [121025.879749] [] btrfs_worker_helper+0xe/0x10 [btrfs] [121025.879753] [] process_one_work+0x168/0x480 [121025.879755] [] worker_thread+0x4b/0x500 [121025.879757] [] ? process_one_work+0x480/0x480 [121025.879758] [] kthread+0xd8/0xf0 [121025.879760] [] ret_from_fork+0x22/0x40 [121025.879763] [] ? kthread_create_on_node+0x1a0/0x1a0 [121025.879764] Mem-Info: [121025.879766] active_anon:18131 inactive_anon:23781 isolated_anon:0 active_file:84105 inactive_file:84587 isolated_file:19 unevictable:1762 dirty:869 writeback:23497 unstable:0 slab_reclaimable:25801 slab_unreclaimable:5224 mapped:33650 shmem:34217 pagetables:1605 bounce:0 free:1897 free_pcp:20 free_cma:0 [121025.879769] Node 0 DMA free:3920kB min:60kB low:72kB high:84kB active_anon:188kB inactive_anon:1324kB active_file:2540kB inactive_file:2664kB unevictable:112kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:112kB dirty:0kB writeback:80kB mapped:884kB shmem:1356kB slab_reclaimable:2296kB slab_unreclaimable:284kB kernel_stack:64kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [121025.879772] lowmem_reserve[]: 0 973 973 973 973 [121025.879775] Node 0 DMA32 free:3668kB min:3824kB low:4820kB high:5816kB active_anon:72336kB inactive_anon:93800kB active_file:333880kB inactive_file:335684kB unevictable:6936kB isolated(anon):0kB isolated(file):76kB present:1032184kB managed:1000480kB mlocked:6936kB dirty:3476kB writeback:93908kB mapped:133716kB shmem:135512kB slab_reclaimable:100908kB slab_unreclaimable:20612kB kernel_stack:3296kB pagetables:6340kB unstable:0kB bounce:0kB free_pcp:80kB local_pcp:80kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no [121025.879779] lowmem_reserve[]: 0 0 0 0 0 [121025.879781] Node 0 DMA: 361*4kB (UME) 59*8kB (UEH) 1*16kB (H) 0*32kB 1*64kB (H) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 3916kB [121025.879788] Node 0 DMA32: 836*4kB (UMH) 22*8kB (UH) 16*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3776kB [121025.879794] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [121025.879795] 204488 total pagecache pages [121025.879796] 675 pages in swap cache [121025.879796] Swap cache stats: add 42167, delete 41492, find 1069663/1070758 [121025.879797] Free swap = 897344kB [121025.879798] Total swap = 1046524kB [121025.879799] 262044 pages RAM [121025.879800] 0 pages HighMem/MovableOnly [121025.879800] 7947 pages reserved [121025.879801] 0 pages cma reserved [121025.879801] 0 pages hwpoisoned [121025.879803] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [121025.879804] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [121025.879804] node 0: slabs: 148, objs: 2368, free: 0 [121025.879807] kworker/u4:2: page allocation failure: order:0, mode:0x2200020(GFP_NOWAIT|__GFP_HIGH|__GFP_NOTRACK) [121025.879808] CPU: 1 PID: 9038 Comm: kworker/u4:2 Not tainted 4.6.0-040600rc6-generic #201605012031 [121025.879809] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [121025.879819] Workqueue: btrfs-worker btrfs_worker_helper [btrfs] [121025.879820] 0000000000000086 00000000d14aff48 ffff88003fd03960 ffffffff813ef3b3 [121025.879822] 0000000000000000 0000000000000000 ffff88003fd039f0 ffffffff8119b3d4 [121025.879823] 02200020d14aff48 ffff88003fffb6e0 ffffffff82110ae0 ffffffff00000060 [121025.879824] Call Trace: [121025.879825] [] dump_stack+0x63/0x90 [121025.879829] [] warn_alloc_failed+0x104/0x160 [121025.879831] [] __alloc_pages_nodemask+0x4f9/0xc30 [121025.879833] [] ? vprintk_emit+0x2a1/0x4d0 [121025.879834] [] alloc_pages_current+0x8c/0x110 [121025.879836] [] new_slab+0x452/0x5c0 [121025.879837] [] ___slab_alloc+0x249/0x480 [121025.879839] [] ? __build_skb+0x2a/0xe0 [121025.879841] [] ? __build_skb+0x2a/0xe0 [121025.879842] [] __slab_alloc+0x20/0x40 [121025.879843] [] kmem_cache_alloc+0x192/0x1d0 [121025.879844] [] ? __build_skb+0x2a/0xe0 [121025.879846] [] __build_skb+0x2a/0xe0 [121025.879849] [] __napi_alloc_skb+0x98/0xd0 [121025.879857] [] page_to_skb.isra.28+0x66/0x310 [121025.879859] [] virtnet_receive+0x2df/0xa00 [121025.879861] [] ? update_rq_clock.part.81+0x18/0x40 [121025.879863] [] virtnet_poll+0x1d/0x80 [121025.879864] [] net_rx_action+0x225/0x360 [121025.879866] [] __do_softirq+0xf6/0x27e [121025.879867] [] irq_exit+0xa3/0xb0 [121025.879869] [] do_IRQ+0x54/0xd0 [121025.879871] [] common_interrupt+0x82/0x82 [121025.879871] [] ? shrink_inactive_list+0x199/0x500 [121025.879875] [] ? shrink_inactive_list+0x405/0x500 [121025.879877] [] shrink_zone_memcg+0x5ad/0x7a0 [121025.879879] [] ? shrink_slab.part.39+0x240/0x3e0 [121025.879881] [] ? css_next_descendant_pre+0x4c/0x60 [121025.879882] [] ? css_next_descendant_pre+0x4c/0x60 [121025.879885] [] shrink_zone+0xe0/0x300 [121025.879886] [] do_try_to_free_pages+0x17e/0x430 [121025.879888] [] try_to_free_pages+0xce/0x180 [121025.879890] [] __alloc_pages_nodemask+0x5c4/0xc30 [121025.879892] [] ? set_next_entity+0x49/0xd0 [121025.879894] [] alloc_pages_current+0x8c/0x110 [121025.879895] [] new_slab+0x452/0x5c0 [121025.879896] [] ___slab_alloc+0x249/0x480 [121025.879905] [] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs] [121025.879908] [] ? chksum_update+0x17/0x20 [121025.879909] [] ? crypto_shash_update+0x30/0xe0 [121025.879917] [] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs] [121025.879919] [] __slab_alloc+0x20/0x40 [121025.879921] [] __kmalloc+0x1c4/0x230 [121025.879929] [] ? run_one_async_free+0x20/0x20 [btrfs] [121025.879938] [] btrfs_csum_one_bio+0x6a/0x430 [btrfs] [121025.879947] [] ? run_one_async_free+0x20/0x20 [btrfs] [121025.879956] [] __btrfs_submit_bio_start+0x1d/0x30 [btrfs] [121025.879965] [] run_one_async_start+0x26/0x30 [btrfs] [121025.879975] [] btrfs_scrubparity_helper+0xca/0x2e0 [btrfs] [121025.879985] [] btrfs_worker_helper+0xe/0x10 [btrfs] [121025.879988] [] process_one_work+0x168/0x480 [121025.879990] [] worker_thread+0x4b/0x500 [121025.879992] [] ? process_one_work+0x480/0x480 [121025.879993] [] kthread+0xd8/0xf0 [121025.879995] [] ret_from_fork+0x22/0x40 [121025.879996] [] ? kthread_create_on_node+0x1a0/0x1a0 [121025.879998] Mem-Info: [121025.880000] active_anon:18131 inactive_anon:23781 isolated_anon:0 active_file:84105 inactive_file:84587 isolated_file:19 unevictable:1762 dirty:869 writeback:23497 unstable:0 slab_reclaimable:25801 slab_unreclaimable:5224 mapped:33650 shmem:34217 pagetables:1605 bounce:0 free:1897 free_pcp:20 free_cma:0 [121025.880002] Node 0 DMA free:3920kB min:60kB low:72kB high:84kB active_anon:188kB inactive_anon:1324kB active_file:2540kB inactive_file:2664kB unevictable:112kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:112kB dirty:0kB writeback:80kB mapped:884kB shmem:1356kB slab_reclaimable:2296kB slab_unreclaimable:284kB kernel_stack:64kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [121025.880006] lowmem_reserve[]: 0 973 973 973 973 [121025.880008] Node 0 DMA32 free:3668kB min:3824kB low:4820kB high:5816kB active_anon:72336kB inactive_anon:93800kB active_file:333880kB inactive_file:335684kB unevictable:6936kB isolated(anon):0kB isolated(file):76kB present:1032184kB managed:1000480kB mlocked:6936kB dirty:3476kB writeback:93908kB mapped:133716kB shmem:135512kB slab_reclaimable:100908kB slab_unreclaimable:20612kB kernel_stack:3296kB pagetables:6340kB unstable:0kB bounce:0kB free_pcp:80kB local_pcp:80kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no [121025.880014] lowmem_reserve[]: 0 0 0 0 0 [121025.880016] Node 0 DMA: 361*4kB (UME) 59*8kB (UEH) 1*16kB (H) 0*32kB 1*64kB (H) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 3916kB [121025.880023] Node 0 DMA32: 836*4kB (UMH) 22*8kB (UH) 16*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3776kB [121025.880029] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [121025.880030] 204488 total pagecache pages [121025.880030] 675 pages in swap cache [121025.880031] Swap cache stats: add 42167, delete 41492, find 1069663/1070758 [121025.880032] Free swap = 897344kB [121025.880033] Total swap = 1046524kB [121025.880034] 262044 pages RAM [121025.880035] 0 pages HighMem/MovableOnly [121025.880035] 7947 pages reserved [121025.880036] 0 pages cma reserved [121025.880036] 0 pages hwpoisoned [121025.880038] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [121025.880039] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [121025.880040] node 0: slabs: 148, objs: 2368, free: 0 [121025.880042] kworker/u4:2: page allocation failure: order:0, mode:0x2200020(GFP_NOWAIT|__GFP_HIGH|__GFP_NOTRACK) [121025.880044] CPU: 1 PID: 9038 Comm: kworker/u4:2 Not tainted 4.6.0-040600rc6-generic #201605012031 [121025.880044] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [121025.880054] Workqueue: btrfs-worker btrfs_worker_helper [btrfs] [121025.880056] 0000000000000086 00000000d14aff48 ffff88003fd03960 ffffffff813ef3b3 [121025.880057] 0000000000000000 0000000000000000 ffff88003fd039f0 ffffffff8119b3d4 [121025.880059] 02200020d14aff48 ffff88003fffb6e0 ffffffff82110ae0 ffffffff00000060 [121025.880060] Call Trace: [121025.880061] [] dump_stack+0x63/0x90 [121025.880064] [] warn_alloc_failed+0x104/0x160 [121025.880066] [] __alloc_pages_nodemask+0x4f9/0xc30 [121025.880068] [] ? vprintk_emit+0x2a1/0x4d0 [121025.880070] [] alloc_pages_current+0x8c/0x110 [121025.880071] [] new_slab+0x452/0x5c0 [121025.880073] [] ___slab_alloc+0x249/0x480 [121025.880075] [] ? __build_skb+0x2a/0xe0 [121025.880076] [] ? __build_skb+0x2a/0xe0 [121025.880077] [] __slab_alloc+0x20/0x40 [121025.880079] [] kmem_cache_alloc+0x192/0x1d0 [121025.880080] [] ? __build_skb+0x2a/0xe0 [121025.880082] [] __build_skb+0x2a/0xe0 [121025.880083] [] __napi_alloc_skb+0x98/0xd0 [121025.880085] [] page_to_skb.isra.28+0x66/0x310 [121025.880087] [] virtnet_receive+0x2df/0xa00 [121025.880088] [] ? update_rq_clock.part.81+0x18/0x40 [121025.880091] [] virtnet_poll+0x1d/0x80 [121025.880092] [] net_rx_action+0x225/0x360 [121025.880094] [] __do_softirq+0xf6/0x27e [121025.880095] [] irq_exit+0xa3/0xb0 [121025.880097] [] do_IRQ+0x54/0xd0 [121025.880098] [] common_interrupt+0x82/0x82 [121025.880099] [] ? shrink_inactive_list+0x199/0x500 [121025.880103] [] ? shrink_inactive_list+0x405/0x500 [121025.880105] [] shrink_zone_memcg+0x5ad/0x7a0 [121025.880106] [] ? shrink_slab.part.39+0x240/0x3e0 [121025.880109] [] ? css_next_descendant_pre+0x4c/0x60 [121025.880110] [] ? css_next_descendant_pre+0x4c/0x60 [121025.880112] [] shrink_zone+0xe0/0x300 [121025.880116] [] do_try_to_free_pages+0x17e/0x430 [121025.880119] [] try_to_free_pages+0xce/0x180 [121025.880120] [] __alloc_pages_nodemask+0x5c4/0xc30 [121025.880122] [] ? set_next_entity+0x49/0xd0 [121025.880123] [] alloc_pages_current+0x8c/0x110 [121025.880125] [] new_slab+0x452/0x5c0 [121025.880127] [] ___slab_alloc+0x249/0x480 [121025.880136] [] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs] [121025.880138] [] ? chksum_update+0x17/0x20 [121025.880139] [] ? crypto_shash_update+0x30/0xe0 [121025.880148] [] ? btrfs_csum_one_bio+0x6a/0x430 [btrfs] [121025.880149] [] __slab_alloc+0x20/0x40 [121025.880151] [] __kmalloc+0x1c4/0x230 [121025.880159] [] ? run_one_async_free+0x20/0x20 [btrfs] [121025.880168] [] btrfs_csum_one_bio+0x6a/0x430 [btrfs] [121025.880177] [] ? run_one_async_free+0x20/0x20 [btrfs] [121025.880186] [] __btrfs_submit_bio_start+0x1d/0x30 [btrfs] [121025.880195] [] run_one_async_start+0x26/0x30 [btrfs] [121025.880205] [] btrfs_scrubparity_helper+0xca/0x2e0 [btrfs] [121025.880215] [] btrfs_worker_helper+0xe/0x10 [btrfs] [121025.880217] [] process_one_work+0x168/0x480 [121025.880219] [] worker_thread+0x4b/0x500 [121025.880221] [] ? process_one_work+0x480/0x480 [121025.880222] [] kthread+0xd8/0xf0 [121025.880224] [] ret_from_fork+0x22/0x40 [121025.880226] [] ? kthread_create_on_node+0x1a0/0x1a0 [121025.880228] Mem-Info: [121025.880230] active_anon:18131 inactive_anon:23781 isolated_anon:0 active_file:84105 inactive_file:84587 isolated_file:19 unevictable:1762 dirty:869 writeback:23497 unstable:0 slab_reclaimable:25801 slab_unreclaimable:5224 mapped:33650 shmem:34217 pagetables:1605 bounce:0 free:1897 free_pcp:21 free_cma:0 [121025.880232] Node 0 DMA free:3920kB min:60kB low:72kB high:84kB active_anon:188kB inactive_anon:1324kB active_file:2540kB inactive_file:2664kB unevictable:112kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:112kB dirty:0kB writeback:80kB mapped:884kB shmem:1356kB slab_reclaimable:2296kB slab_unreclaimable:284kB kernel_stack:64kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [121025.880236] lowmem_reserve[]: 0 973 973 973 973 [121025.880239] Node 0 DMA32 free:3668kB min:3824kB low:4820kB high:5816kB active_anon:72336kB inactive_anon:93800kB active_file:333880kB inactive_file:335684kB unevictable:6936kB isolated(anon):0kB isolated(file):76kB present:1032184kB managed:1000480kB mlocked:6936kB dirty:3476kB writeback:93908kB mapped:133716kB shmem:135512kB slab_reclaimable:100908kB slab_unreclaimable:20612kB kernel_stack:3296kB pagetables:6340kB unstable:0kB bounce:0kB free_pcp:84kB local_pcp:84kB free_cma:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no [121025.880242] lowmem_reserve[]: 0 0 0 0 0 [121025.880244] Node 0 DMA: 361*4kB (UME) 59*8kB (UEH) 1*16kB (H) 0*32kB 1*64kB (H) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 3916kB [121025.880251] Node 0 DMA32: 836*4kB (UMH) 22*8kB (UH) 16*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3776kB [121025.880258] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [121025.880258] 204488 total pagecache pages [121025.880259] 675 pages in swap cache [121025.880260] Swap cache stats: add 42167, delete 41492, find 1069663/1070758 [121025.880261] Free swap = 897344kB [121025.880261] Total swap = 1046524kB [121025.880262] 262044 pages RAM [121025.880262] 0 pages HighMem/MovableOnly [121025.880263] 7947 pages reserved [121025.880264] 0 pages cma reserved [121025.880266] 0 pages hwpoisoned [121025.880267] SLUB: Unable to allocate memory on node -1, gfp=0x2080020(GFP_ATOMIC) [121025.880268] cache: kmalloc-256, object size: 256, buffer size: 256, default order: 0, min order: 0 [121025.880269] node 0: slabs: 148, objs: 2368, free: 0