BTRFS Page allocation failure with Bacula.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Confirmed
|
Medium
|
Unassigned |
Bug Description
As soon as bacula starts running I see the below errors in the kernel log, and the server eventually hangs at some point. I'm backing up our servers to a local BTRFS volume on a local drive, nothing fancy and it used to work really well in Ubuntu 14.04.
Apr 27 21:05:18 backup kernel: [343463.805173] kworker/u4:5: page allocation failure: order:0, mode:0x2204020
Apr 27 21:05:18 backup kernel: [343463.805178] CPU: 1 PID: 19811 Comm: kworker/u4:5 Not tainted 4.4.0-21-generic #37-Ubuntu
Apr 27 21:05:18 backup kernel: [343463.805179] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Apr 27 21:05:18 backup kernel: [343463.805199] Workqueue: btrfs-submit btrfs_submit_helper [btrfs]
Apr 27 21:05:18 backup kernel: [343463.805201] 0000000000000086 000000005236f502 ffff880019f2b548 ffffffff813e93c3
Apr 27 21:05:18 backup kernel: [343463.805203] 0000000002204020 0000000000000000 ffff880019f2b5d8 ffffffff81192fba
Apr 27 21:05:18 backup kernel: [343463.805205] ffff88007fff9d40 ffff88007fffb6e0 0000000000000000 ffffffff00000060
Apr 27 21:05:18 backup kernel: [343463.805206] Call Trace:
Apr 27 21:05:18 backup kernel: [343463.805211] [<ffffffff813e9
Apr 27 21:05:18 backup kernel: [343463.805214] [<ffffffff81192
Apr 27 21:05:18 backup kernel: [343463.805217] [<ffffffff81196
Apr 27 21:05:18 backup kernel: [343463.805220] [<ffffffff811e0
Apr 27 21:05:18 backup kernel: [343463.805222] [<ffffffff811e9
Apr 27 21:05:18 backup kernel: [343463.805224] [<ffffffff811ea
Apr 27 21:05:18 backup kernel: [343463.805227] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.805228] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.805230] [<ffffffff811ea
Apr 27 21:05:18 backup kernel: [343463.805231] [<ffffffff811ec
Apr 27 21:05:18 backup kernel: [343463.805233] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.805234] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.805237] [<ffffffff81571
Apr 27 21:05:18 backup kernel: [343463.805239] [<ffffffff8118f
Apr 27 21:05:18 backup kernel: [343463.805242] [<ffffffff813c0
Apr 27 21:05:18 backup kernel: [343463.805243] [<ffffffff81571
Apr 27 21:05:18 backup kernel: [343463.805245] [<ffffffff813c4
Apr 27 21:05:18 backup kernel: [343463.805247] [<ffffffff813c4
Apr 27 21:05:18 backup kernel: [343463.805248] [<ffffffff813c5
Apr 27 21:05:18 backup kernel: [343463.805250] [<ffffffff813c6
Apr 27 21:05:18 backup kernel: [343463.805251] [<ffffffff813bb
Apr 27 21:05:18 backup kernel: [343463.805253] [<ffffffff813bb
Apr 27 21:05:18 backup kernel: [343463.805264] [<ffffffffc01a7
Apr 27 21:05:18 backup kernel: [343463.805275] [<ffffffffc01a7
Apr 27 21:05:18 backup kernel: [343463.805285] [<ffffffffc01b2
Apr 27 21:05:18 backup kernel: [343463.805294] [<ffffffffc01b3
Apr 27 21:05:18 backup kernel: [343463.805297] [<ffffffff81099
Apr 27 21:05:18 backup kernel: [343463.805298] [<ffffffff8109a
Apr 27 21:05:18 backup kernel: [343463.805300] [<ffffffff8109a
Apr 27 21:05:18 backup kernel: [343463.805301] [<ffffffff8109a
Apr 27 21:05:18 backup kernel: [343463.805303] [<ffffffff810a0
Apr 27 21:05:18 backup kernel: [343463.805305] [<ffffffff810a0
Apr 27 21:05:18 backup kernel: [343463.805308] [<ffffffff81824
Apr 27 21:05:18 backup kernel: [343463.805309] [<ffffffff810a0
Apr 27 21:05:18 backup kernel: [343463.805310] Mem-Info:
Apr 27 21:05:18 backup kernel: [343463.805313] active_anon:43184 inactive_anon:51940 isolated_anon:0
Apr 27 21:05:18 backup kernel: [343463.805313] active_file:123451 inactive_
Apr 27 21:05:18 backup kernel: [343463.805313] unevictable:1752 dirty:32219 writeback:6149 unstable:0
Apr 27 21:05:18 backup kernel: [343463.805313] slab_reclaimabl
Apr 27 21:05:18 backup kernel: [343463.805313] mapped:58659 shmem:79658 pagetables:1755 bounce:0
Apr 27 21:05:18 backup kernel: [343463.805313] free:12908 free_pcp:138 free_cma:0
Apr 27 21:05:18 backup kernel: [343463.805315] Node 0 DMA free:9992kB min:40kB low:48kB high:60kB active_anon:28kB inactive_anon:396kB active_file:256kB inactive_file:884kB unevictable:92kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:92kB dirty:936kB writeback:0kB mapped:60kB shmem:316kB slab_reclaimabl
Apr 27 21:05:18 backup kernel: [343463.805319] lowmem_reserve[]: 0 1981 1981 1981 1981
Apr 27 21:05:18 backup kernel: [343463.805321] Node 0 DMA32 free:41640kB min:5580kB low:6972kB high:8368kB active_
Apr 27 21:05:18 backup kernel: [343463.805325] lowmem_reserve[]: 0 0 0 0 0
Apr 27 21:05:18 backup kernel: [343463.805327] Node 0 DMA: 17*4kB (UEH) 65*8kB (UMEH) 8*16kB (UMH) 24*32kB (UMEH) 21*64kB (UMEH) 14*128kB (UMH) 7*256kB (UMH) 1*512kB (H) 1*1024kB (H) 1*2048kB (H) 0*4096kB = 9996kB
Apr 27 21:05:18 backup kernel: [343463.805335] Node 0 DMA32: 60*4kB (H) 75*8kB (H) 64*16kB (H) 33*32kB (H) 44*64kB (H) 33*128kB (H) 20*256kB (H) 14*512kB (H) 7*1024kB (H) 6*2048kB (H) 0*4096kB = 41704kB
Apr 27 21:05:18 backup kernel: [343463.805343] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_
Apr 27 21:05:18 backup kernel: [343463.805343] 453607 total pagecache pages
Apr 27 21:05:18 backup kernel: [343463.805345] 317 pages in swap cache
Apr 27 21:05:18 backup kernel: [343463.805346] Swap cache stats: add 4177, delete 3860, find 161061/161539
Apr 27 21:05:18 backup kernel: [343463.805346] Free swap = 1043312kB
Apr 27 21:05:18 backup kernel: [343463.805347] Total swap = 1046524kB
Apr 27 21:05:18 backup kernel: [343463.805348] 524188 pages RAM
Apr 27 21:05:18 backup kernel: [343463.805349] 0 pages HighMem/MovableOnly
Apr 27 21:05:18 backup kernel: [343463.805349] 12075 pages reserved
Apr 27 21:05:18 backup kernel: [343463.805350] 0 pages cma reserved
Apr 27 21:05:18 backup kernel: [343463.805350] 0 pages hwpoisoned
Apr 27 21:05:18 backup kernel: [343463.805352] SLUB: Unable to allocate memory on node -1 (gfp=0x2080020)
Apr 27 21:05:18 backup kernel: [343463.805353] cache: kmalloc-2048, object size: 2048, buffer size: 2048, default order: 3, min order: 0
Apr 27 21:05:18 backup kernel: [343463.805354] node 0: slabs: 39, objs: 470, free: 0
Apr 27 21:05:18 backup kernel: [343463.816748] kworker/1:1H: page allocation failure: order:0, mode:0x2204020
Apr 27 21:05:18 backup kernel: [343463.816752] CPU: 1 PID: 154 Comm: kworker/1:1H Not tainted 4.4.0-21-generic #37-Ubuntu
Apr 27 21:05:18 backup kernel: [343463.816753] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Apr 27 21:05:18 backup kernel: [343463.816758] Workqueue: kblockd blk_mq_run_work_fn
Apr 27 21:05:18 backup kernel: [343463.816759] 0000000000000086 00000000010bc024 ffff88003528f7e8 ffffffff813e93c3
Apr 27 21:05:18 backup kernel: [343463.816761] 0000000002204020 0000000000000000 ffff88003528f878 ffffffff81192fba
Apr 27 21:05:18 backup kernel: [343463.816763] ffff88007fff9d40 ffff88007fffb6e0 0000000000000000 ffffffff00000060
Apr 27 21:05:18 backup kernel: [343463.816764] Call Trace:
Apr 27 21:05:18 backup kernel: [343463.816768] [<ffffffff813e9
Apr 27 21:05:18 backup kernel: [343463.816771] [<ffffffff81192
Apr 27 21:05:18 backup kernel: [343463.816774] [<ffffffff81196
Apr 27 21:05:18 backup kernel: [343463.816777] [<ffffffff810d8
Apr 27 21:05:18 backup kernel: [343463.816780] [<ffffffff811e0
Apr 27 21:05:18 backup kernel: [343463.816782] [<ffffffff811e9
Apr 27 21:05:18 backup kernel: [343463.816783] [<ffffffff811ea
Apr 27 21:05:18 backup kernel: [343463.816786] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.816788] [<ffffffff810bc
Apr 27 21:05:18 backup kernel: [343463.816790] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.816791] [<ffffffff811ea
Apr 27 21:05:18 backup kernel: [343463.816793] [<ffffffff811ec
Apr 27 21:05:18 backup kernel: [343463.816794] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.816796] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.816798] [<ffffffff81571
Apr 27 21:05:18 backup kernel: [343463.816801] [<ffffffff813c0
Apr 27 21:05:18 backup kernel: [343463.816803] [<ffffffff81571
Apr 27 21:05:18 backup kernel: [343463.816805] [<ffffffff813c4
Apr 27 21:05:18 backup kernel: [343463.816806] [<ffffffff813c4
Apr 27 21:05:18 backup kernel: [343463.816809] [<ffffffff81099
Apr 27 21:05:18 backup kernel: [343463.816810] [<ffffffff8109a
Apr 27 21:05:18 backup kernel: [343463.816812] [<ffffffff8109a
Apr 27 21:05:18 backup kernel: [343463.816813] [<ffffffff8109a
Apr 27 21:05:18 backup kernel: [343463.816815] [<ffffffff810a0
Apr 27 21:05:18 backup kernel: [343463.816816] [<ffffffff810a0
Apr 27 21:05:18 backup kernel: [343463.816819] [<ffffffff81824
Apr 27 21:05:18 backup kernel: [343463.816821] [<ffffffff810a0
Apr 27 21:05:18 backup kernel: [343463.816822] Mem-Info:
Apr 27 21:05:18 backup kernel: [343463.816825] active_anon:43184 inactive_anon:51940 isolated_anon:0
Apr 27 21:05:18 backup kernel: [343463.816825] active_file:123441 inactive_
Apr 27 21:05:18 backup kernel: [343463.816825] unevictable:1752 dirty:32397 writeback:5123 unstable:0
Apr 27 21:05:18 backup kernel: [343463.816825] slab_reclaimabl
Apr 27 21:05:18 backup kernel: [343463.816825] mapped:58673 shmem:79658 pagetables:1755 bounce:0
Apr 27 21:05:18 backup kernel: [343463.816825] free:12867 free_pcp:107 free_cma:0
Apr 27 21:05:18 backup kernel: [343463.816827] Node 0 DMA free:9988kB min:40kB low:48kB high:60kB active_anon:28kB inactive_anon:396kB active_file:256kB inactive_file:884kB unevictable:92kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:92kB dirty:940kB writeback:0kB mapped:60kB shmem:316kB slab_reclaimabl
Apr 27 21:05:18 backup kernel: [343463.816831] lowmem_reserve[]: 0 1981 1981 1981 1981
Apr 27 21:05:18 backup kernel: [343463.816833] Node 0 DMA32 free:41480kB min:5580kB low:6972kB high:8368kB active_
Apr 27 21:05:18 backup kernel: [343463.816837] lowmem_reserve[]: 0 0 0 0 0
Apr 27 21:05:18 backup kernel: [343463.816839] Node 0 DMA: 15*4kB (UEH) 65*8kB (UMEH) 8*16kB (UMH) 24*32kB (UMEH) 21*64kB (UMEH) 14*128kB (UMH) 7*256kB (UMH) 1*512kB (H) 1*1024kB (H) 1*2048kB (H) 0*4096kB = 9988kB
Apr 27 21:05:18 backup kernel: [343463.816847] Node 0 DMA32: 60*4kB (H) 75*8kB (H) 64*16kB (H) 29*32kB (H) 44*64kB (H) 33*128kB (H) 20*256kB (H) 14*512kB (H) 7*1024kB (H) 6*2048kB (H) 0*4096kB = 41576kB
Apr 27 21:05:18 backup kernel: [343463.816854] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_
Apr 27 21:05:18 backup kernel: [343463.816855] 453618 total pagecache pages
Apr 27 21:05:18 backup kernel: [343463.816856] 317 pages in swap cache
Apr 27 21:05:18 backup kernel: [343463.816857] Swap cache stats: add 4177, delete 3860, find 161061/161539
Apr 27 21:05:18 backup kernel: [343463.816858] Free swap = 1043312kB
Apr 27 21:05:18 backup kernel: [343463.816859] Total swap = 1046524kB
Apr 27 21:05:18 backup kernel: [343463.816860] 524188 pages RAM
Apr 27 21:05:18 backup kernel: [343463.816860] 0 pages HighMem/MovableOnly
Apr 27 21:05:18 backup kernel: [343463.816861] 12075 pages reserved
Apr 27 21:05:18 backup kernel: [343463.816861] 0 pages cma reserved
Apr 27 21:05:18 backup kernel: [343463.816862] 0 pages hwpoisoned
Apr 27 21:05:18 backup kernel: [343463.816863] SLUB: Unable to allocate memory on node -1 (gfp=0x2080020)
Apr 27 21:05:18 backup kernel: [343463.816865] cache: kmalloc-2048, object size: 2048, buffer size: 2048, default order: 3, min order: 0
Apr 27 21:05:18 backup kernel: [343463.816866] node 0: slabs: 41, objs: 474, free: 0
Apr 27 21:05:18 backup kernel: [343463.816872] kworker/1:1H: page allocation failure: order:0, mode:0x2204020
Apr 27 21:05:18 backup kernel: [343463.816873] CPU: 1 PID: 154 Comm: kworker/1:1H Not tainted 4.4.0-21-generic #37-Ubuntu
Apr 27 21:05:18 backup kernel: [343463.816874] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Apr 27 21:05:18 backup kernel: [343463.816875] Workqueue: kblockd blk_mq_run_work_fn
Apr 27 21:05:18 backup kernel: [343463.816876] 0000000000000086 00000000010bc024 ffff88003528f7e8 ffffffff813e93c3
Apr 27 21:05:18 backup kernel: [343463.816878] 0000000002204020 0000000000000000 ffff88003528f878 ffffffff81192fba
Apr 27 21:05:18 backup kernel: [343463.816879] ffff88007fff9d40 ffff88007fffb6e0 0000000000000000 ffffffff00000060
Apr 27 21:05:18 backup kernel: [343463.816881] Call Trace:
Apr 27 21:05:18 backup kernel: [343463.816882] [<ffffffff813e9
Apr 27 21:05:18 backup kernel: [343463.816884] [<ffffffff81192
Apr 27 21:05:18 backup kernel: [343463.816885] [<ffffffff81196
Apr 27 21:05:18 backup kernel: [343463.816888] [<ffffffff810d8
Apr 27 21:05:18 backup kernel: [343463.816890] [<ffffffff811e0
Apr 27 21:05:18 backup kernel: [343463.816891] [<ffffffff811e9
Apr 27 21:05:18 backup kernel: [343463.816893] [<ffffffff811ea
Apr 27 21:05:18 backup kernel: [343463.816894] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.816896] [<ffffffff810bc
Apr 27 21:05:18 backup kernel: [343463.816897] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.816899] [<ffffffff811ea
Apr 27 21:05:18 backup kernel: [343463.816900] [<ffffffff811ec
Apr 27 21:05:18 backup kernel: [343463.816901] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.816903] [<ffffffff814bf
Apr 27 21:05:18 backup kernel: [343463.816905] [<ffffffff81571
Apr 27 21:05:18 backup kernel: [343463.816907] [<ffffffff813c0
Apr 27 21:05:18 backup kernel: [343463.816908] [<ffffffff81571
Apr 27 21:05:18 backup kernel: [343463.816910] [<ffffffff813c4
Apr 27 21:05:18 backup kernel: [343463.816911] [<ffffffff813c4
Apr 27 21:05:18 backup kernel: [343463.816915] [<ffffffff81099
Apr 27 21:05:18 backup kernel: [343463.816916] [<ffffffff8109a
Apr 27 21:05:18 backup kernel: [343463.816918] [<ffffffff8109a
Apr 27 21:05:18 backup kernel: [343463.816919] [<ffffffff8109a
Apr 27 21:05:18 backup kernel: [343463.816921] [<ffffffff810a0
Apr 27 21:05:18 backup kernel: [343463.816922] [<ffffffff810a0
Apr 27 21:05:18 backup kernel: [343463.816924] [<ffffffff81824
Apr 27 21:05:18 backup kernel: [343463.816925] [<ffffffff810a0
Apr 27 21:05:18 backup kernel: [343463.816926] Mem-Info:
Apr 27 21:05:18 backup kernel: [343463.816929] active_anon:43184 inactive_anon:51940 isolated_anon:0
Apr 27 21:05:18 backup kernel: [343463.816929] active_file:123441 inactive_
Apr 27 21:05:18 backup kernel: [343463.816929] unevictable:1752 dirty:32397 writeback:5123 unstable:0
Apr 27 21:05:18 backup kernel: [343463.816929] slab_reclaimabl
Apr 27 21:05:18 backup kernel: [343463.816929] mapped:58673 shmem:79658 pagetables:1755 bounce:0
Apr 27 21:05:18 backup kernel: [343463.816929] free:12867 free_pcp:107 free_cma:0
Apr 27 21:05:18 backup kernel: [343463.816931] Node 0 DMA free:9988kB min:40kB low:48kB high:60kB active_anon:28kB inactive_anon:396kB active_file:256kB inactive_file:884kB unevictable:92kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:92kB dirty:940kB writeback:0kB mapped:60kB shmem:316kB slab_reclaimabl
Apr 27 21:05:18 backup kernel: [343463.816938] lowmem_reserve[]: 0 1981 1981 1981 1981
Apr 27 21:05:18 backup kernel: [343463.816940] Node 0 DMA32 free:41480kB min:5580kB low:6972kB high:8368kB active_
Apr 27 21:05:18 backup kernel: [343463.816943] lowmem_reserve[]: 0 0 0 0 0
Apr 27 21:05:18 backup kernel: [343463.816945] Node 0 DMA: 15*4kB (UEH) 65*8kB (UMEH) 8*16kB (UMH) 24*32kB (UMEH) 21*64kB (UMEH) 14*128kB (UMH) 7*256kB (UMH) 1*512kB (H) 1*1024kB (H) 1*2048kB (H) 0*4096kB = 9988kB
Apr 27 21:05:18 backup kernel: [343463.816952] Node 0 DMA32: 60*4kB (H) 75*8kB (H) 64*16kB (H) 29*32kB (H) 44*64kB (H) 33*128kB (H) 20*256kB (H) 14*512kB (H) 7*1024kB (H) 6*2048kB (H) 0*4096kB = 41576kB
Apr 27 21:05:18 backup kernel: [343463.816960] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_
Apr 27 21:05:18 backup kernel: [343463.816960] 453618 total pagecache pages
Apr 27 21:05:18 backup kernel: [343463.816961] 317 pages in swap cache
Apr 27 21:05:18 backup kernel: [343463.816962] Swap cache stats: add 4177, delete 3860, find 161061/161539
Apr 27 21:05:18 backup kernel: [343463.816963] Free swap = 1043312kB
Apr 27 21:05:18 backup kernel: [343463.816963] Total swap = 1046524kB
Apr 27 21:05:18 backup kernel: [343463.816964] 524188 pages RAM
Apr 27 21:05:18 backup kernel: [343463.816965] 0 pages HighMem/MovableOnly
Apr 27 21:05:18 backup kernel: [343463.816965] 12075 pages reserved
Apr 27 21:05:18 backup kernel: [343463.816966] 0 pages cma reserved
Apr 27 21:05:18 backup kernel: [343463.816966] 0 pages hwpoisoned
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-
ProcVersionSign
Uname: Linux 4.4.0-21-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version k4.4.0-21-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC0D0', '/dev/snd/
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.
Date: Thu Apr 28 08:53:40 2016
HibernationDevice: RESUME=
IwConfig:
eth0 no wireless extensions.
lo no wireless extensions.
tun0 no wireless extensions.
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Bochs Bochs
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
XDG_RUNTIME_
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 qxldrmfb
ProcKernelCmdLine: BOOT_IMAGE=
RelatedPackageV
linux-
linux-
linux-firmware 1.157
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to xenial on 2016-04-21 (7 days ago)
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:
dmi.product.name: Bochs
dmi.sys.vendor: Bochs
This change was made by a bot.