Nova doesn't account for hugepages when scheduling VMs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Won't Fix
|
Undecided
|
Unassigned |
Bug Description
Description
===========
When hugepages are enabled on the host it's possible to schedule VMs using more RAM than available.
On the node with memory usage presented below it was possible to schedule 6 instances using a total of 140G of memory and a non-hugepages-
After running memory intensive operations on the VMs, some of them got OOM killed.
$ cat /proc/meminfo | egrep "^(Mem|Huge)" # on the compute node
MemTotal: 197784792 kB
MemFree: 115005288 kB
MemAvailable: 116745612 kB
HugePages_Total: 64
HugePages_Free: 64
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 67108864 kB
$ os hypervisor show copmute1 -c memory_mb -c memory_mb_used -c free_ram_mb
+------
| Field | Value |
+------
| free_ram_mb | 29309 |
| memory_mb | 193149 |
| memory_mb_used | 163840 |
+------
$ os host show compute1
+------
| Host | Project | CPU | Memory MB | Disk GB |
+------
| compute1 | (total) | 0 | 193149 | 893 |
| compute1 | (used_now) | 72 | 163840 | 460 |
| compute1 | (used_max) | 72 | 147456 | 460 |
| compute1 | some_project_
| compute1 | another_
+------
$ os resource provider inventory list uuid_of_
+------
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total |
+------
| MEMORY_MB | 1.0 | 1 | 193149 | 16384 | 1 | 193149 |
| DISK_GB | 1.0 | 1 | 893 | 0 | 1 | 893 |
| PCPU | 1.0 | 1 | 72 | 0 | 1 | 72 |
+------
Steps to reproduce
==================
1. Reserve a large part of memory for hugepages on the hypervisor.
2. Create VMs using a flavor that uses a lot of memory that isn't backed by hugepages.
3. Start memory intensive operations on the VMs, e.g.:
stress-ng --vm-bytes $(awk '/MemAvailable/
Expected result
===============
Nova should not allow overcommitment and should be able to differentiate between hugepages and "normal" memory.
Actual result
=============
Overcommitment resulting in OOM kills.
Environment
===========
nova-api-metadata 2:21.2.
nova-common 2:21.2.
nova-compute 2:21.2.
nova-compute-kvm 2:21.2.
nova-compute-
python3-nova 2:21.2.
python3-novaclient 2:17.0.
OS: Ubuntu 18.04.5 LTS
Hypervisor: libvirt + KVM
This can be reproduced on Focal/ussuri:
############ Computes: ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- --+---- ------- -+----- ------- ------- ------- ------- -----+- ------- ------- ------- + provider_ uuid | ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- --+---- ------- -+----- ------- ------- ------- ------- -----+- ------- ------- ------- + 7e60-4365- 9cc8-7afc78b530 05 | juju-98fb61- zaza-d6f2c78250 43-9.project. serverstack | 5 | ca3fa736- 7e60-4365- 9cc8-7afc78b530 05 | None | 71d5-40ed- ab8f-eceeaaac59 b5 | juju-98fb61- zaza-d6f2c78250 43-8.project. serverstack | 4 | 0605bd29- 71d5-40ed- ab8f-eceeaaac59 b5 | None | ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- --+---- ------- -+----- ------- ------- ------- ------- -----+- ------- ------- ------- +
$ os resource provider list
+------
| uuid | name | generation | root_provider_uuid | parent_
+------
| ca3fa736-
| 0605bd29-
+------
############ Mem Allocation ratio is 1: 7e60-4365- 9cc8-7afc78b530 05 ------- ---+--- ------- ------- -+----- -----+- ------- --+---- ------+ ------- ----+-- -----+- ------+ ------- ---+--- ------- ------- -+----- -----+- ------- --+---- ------+ ------- ----+-- -----+- ------+ ------- ---+--- ------- ------- -+----- -----+- ------- --+---- ------+ ------- ----+-- -----+- ------+
$ openstack resource provider inventory list ca3fa736-
+------
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+------
| VCPU | 16.0 | 1 | 8 | 0 | 1 | 8 | 2 |
| MEMORY_MB | 1.0 | 1 | 16008 | 2048 | 1 | 16008 | 13960 |
| DISK_GB | 1.0 | 1 | 77 | 0 | 1 | 77 | 20 |
+------
$ openstack resource provider inventory list 0605bd29- 71d5-40ed- ab8f-eceeaaac59 b5 ------- ---+--- ------- ------- -+----- -----+- ------- --+---- ------+ ------- ----+-- -----+- -----+ ------- ---+--- ------- ------- -+----- -----+- ------- --+---- ------+ ------- ----+-- -----+- -----+ ------- ---+--- ------- ------- -+----- -----+- ------- --+---- ------+ ------- ----+-- -----+- -----+
+------
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+------
| VCPU | 16.0 | 1 | 8 | 0 | 1 | 8 | 0 |
| MEMORY_MB | 1.0 | 1 | 16008 | 2048 | 1 | 16008 | 0 |
| DISK_GB | 1.0 | 1 | 77 | 0 | 1 | 77 | 0 |
+------
######## Hugepages: 1000 * 2M 98fb61- zaza-d6f2c78250 43-9:~# cat /proc/meminfo | grep -i huge
root@juju-
AnonHugePages: 622592 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 1000
HugePages_Free: 1000
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 2048000 kB
root@juju- 98fb61- zaza-d6f2c78250 43-9:~# free -mh
total used free shared buff/cache available
Mem: 15Gi ...