"cannot set up guest memory" b/c no automatic clearing of Linux' cache

[Celmor]

Version: qemu-2.6.0-1
Kernel: 4.4.13-1-MANJARO
Full script (shouldn't matter though): https://pastebin.com/Hp24PWNE

Problem:
When host has been up and used for a while cache has been filled as much that guest can't be started without droping caches.

Expected behavior:
Qemu should be able to request as much Memory as it needs and cause Linux to drop cache pages if needed. A user shouldn't be required to have to come to this conclusion and having to drop caches to start Qemu with the required amount of memory.

My fix:
Following command (as root) required before qemu start:
# sync && echo 3 > /proc/sys/vm/drop_caches

Example:
$ sudo qemu.sh -m 10240 && echo success || echo failed
qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory
failed
$ free
              total used free shared buff/cache available
Mem: 16379476 9126884 3462688 148480 3789904 5123572
Swap: 0 0 0
$ sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
$ free
              total used free shared buff/cache available
Mem: 16379476 1694528 14106552 149772 578396 14256428
Swap: 0 0 0
$ sudo qemu.sh -m 10240 && echo success || echo failed
success

[Dr. David Alan Gilbert (dgilbert-h)]

Hi Celmor,
  That shouldn't happen! QEMU's allocation of memory is pretty normal, so really the question is for the kernel guys.
  Having chatted to a collague, two questions:
     a) Does it still happen if you set /proc/sys/vm/overcommit_memory to 1?
     b) What filesystem are you using (some have different behaviour when dealing with the caches).

[Celmor (celmor)]

@dgilbert-h / Dr. David Alan Gilbert
Thanks for your answer.

b)
Mounted/used block devices:
NAME MOUNTPOINT TYPE FSTYPE
sda disk crypto_LUKS
└─Data1 crypt zfs_member
├─sdb5 / part ext4
└─sdb6 /boot part vfat
sdd disk crypto_LUKS
└─Data2 crypt zfs_member
ZFS file system for extra space, also ZFS is the reason I'm not running the latest kernel...

a)
$ free
              total used free shared buff/cache available
Mem: 16379476 7879216 1867196 188180 6633064 3587620
Swap: 0 0 0
$ qemu.sh -m 10240 && echo success || echo failed
qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory
failed
$ sudo sh -c 'echo 1 > /proc/sys/vm/overcommit_memory'
$ qemu.sh -m 10240 && echo success || echo failed
success

So setting /proc/sys/vm/overcommit_memory to 1 works, so I guess I'm gonna need to execute
sudo sh -c 'echo 1 > /proc/sys/vm/overcommit_memory'
at start of my qemu.sh script instead of the 'drop_caches' part.
I still think the kernel should do whatever function overcommit_memory is for automatically, bit it seams to be the fault of the kernel of my distribution then, thanks for your help.

[Dr. David Alan Gilbert (dgilbert-h)]

Thanks Celmor; that might be one for the zfs guys then if that's what's holding onto the caches; I suspect their caching is a bit different from the rest of the Linux filesystems.

I'm going to close this as 'invalid' because I'm pretty sure this isn't a qemu bug; the kernel should be giving us the RAM we asked for if it's got it.