memory overhead of qemu-kvm with ceph rbd and ram-allocation-ratio=0.9 leads to memory starvation
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Nova Cloud Controller Charm |
Invalid
|
Undecided
|
Unassigned | ||
ceph (Ubuntu) |
Incomplete
|
Medium
|
Unassigned | ||
qemu (Ubuntu) |
Confirmed
|
Medium
|
Unassigned |
Bug Description
We have observed up to 20% memory overhead on several 18GB nova instances packed per node taking up to 22GB resident memory when fully utilized. Our standard ram-allocation-
The nodes have 512 GB ram and were experiencing less than 8GB free ram with 8GB swap utilized. High consumption CICD environment with ceph ephemeral disk.
We have since worked around memory starvation by adding nodes and reducing ram_allocation_
Does this resident memory overhead seem abnormally high? Is there anything in the below qemu process that may be causing this that we need to account for in the ram-allocation-
Specs:
Xenial series
nova-cloud-
nova-compute 13.1.2 charm rev. 135
Note the -m 18432(MB) argument and 22049 MB RSS of this typical high-use instance's process:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
libvirt+ 2020854 208 4.2 49713524 22579016 ? Sl 19:06 235:28 /usr/bin/
accel=kvm
usb=off -cpu Haswell-noTSX -m 18432 -realtime mlock=off -smp 8
sockets=8
cores=1
threads=1 -uuid XXXXXXXX -smbios type=1
manufacturer=
product=OpenStack Nova
version=13.1.2
serial=XXXXXXXXXXXX
uuid=XXXXXXXXX
family=Virtual Machine -no-user-config -nodefaults -chardev socket
id=charmonitor
path=/var/
server
nowait -mon chardev=charmonitor
id=monitor
mode=control -rtc base=utc
driftfix=slew -global kvm-pit.
id=usb
bus=pci.0
addr=0x1.0x2 -drive file=rbd:
format=raw
if=none
id=drive-
cache=none -device virtio-blk-pci
scsi=off
bus=pci.0
addr=0x4
drive=drive-
id=virtio-disk0
bootindex=1 -netdev tap
fd=26
id=hostnet0
vhost=on
vhostfd=34 -device virtio-net-pci
netdev=hostnet0
id=net0
mac=XX:
bus=pci.0
addr=0x3 -chardev file
id=charserial0
path=/var/
chardev=charserial0
id=serial0 -chardev pty
id=charserial1 -device isa-serial
chardev=charserial1
id=serial1 -device usb-tablet
id=input0 -vnc 0.0.0.0:0 -k en-us -device cirrus-vga
id=video0
bus=pci.0
addr=0x2 -device virtio-balloon-pci
id=balloon0
bus=pci.0
addr=0x5 -msg timestamp=on
tags: | added: canonical-bootstack |
Changed in qemu (Ubuntu): | |
status: | Expired → New |
Changed in ceph (Ubuntu): | |
status: | Expired → New |
I am not using openstack, but have the same problem with QEMU/KVM virtual machines that have Ceph RBD disks attached. There is an issue on the Ceph tracker (http:// tracker. ceph.com/ issues/ 20054#change- 93573) but it isn't clear if the bug is on their side or if it is in QEMU.