Previously (for example, in RHOSP 13) libvirt-daemon RPM was installed on compute nodes and it provided /usr/lib/sysctl.d/60-libvirtd.conf configuration file [1] to increase fs.aio-max-nr from 65536 (default) to 1048576. libvirt-daemon RPM is no longer installed on RHOSP 16.1 nodes and default value is not changed there.
Documentation tells us that this number should be calculated as number of virtual disks on the host * 128. As a result default limit leaves us with 512 virtual disks, but KCS providing deployment limitations https://access.redhat.com/articles/1436373 tells us about 16384 KVM guests per hypervisor.
I am not sure how default value would behave in containerized environment, but this looks like a possible problem that requires second look.
[1]
[root@compute-1 ~]# cat /usr/lib/sysctl.d/60-libvirtd.conf
# The kernel allocates aio memory on demand, and this number limits the
# number of parallel aio requests; the only drawback of a larger limit is
# that a malicious guest could issue parallel requests to cause the kernel
# to set aside memory. Set this number at least as large as
# 128 * (number of virtual disks on the host)
# Libvirt uses a default of 1M requests to allow 8k disks, with at most
# 64M of kernel memory if all disks hit an aio request at the same time.
fs.aio-max-nr = 1048576
Fix proposed to branch: stable/train
Review: https:/