Simultaneous allocation of 1Gb and 2 M HugePages break VMs launching

[Ksenia Svechnikova]

ISO 9.0 150 - version http://paste.openstack.org/show/492973/

Steps to reproduce:

    Prepare 2+1 nodes

[root@fuel ~]# fuel node --env 2
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------|------------------|---------|-----------|-------------------|-----------------|---------------|--------|---------
5 | ready | Untitled (58:06) | 2 | 10.20.0.6 | 0c:c4:7a:6c:58:06 | cinder, compute | | True | 2
2 | ready | Untitled (55:2c) | 2 | 10.20.0.5 | 0c:c4:7a:34:55:2c | cinder, compute | | True | 2
6 | ready | Untitled (53:8e) | 2 | 10.20.0.4 | 0c:c4:7a:34:53:8e | controller | | True | 2

    There are 4 NUMA nodes on compute node-2. Allocate Nova Huge pages: 2.0 MB - 2030 and 1.0 GB - 10
    Deploy env
    Verify HP are present on node-2
    Create an aggregate for compute nodes with huge pages and CPU pinning:

nova aggregate-create performance_3_cpu
nova aggregate-set-metadata performance_3_cpur hpgs=true
nova aggregate-set-metadata performance_3_cpu pinned=true

    Add hosts to them:

nova aggregate-add-host performance_3_cpu node-2.domain.tld

    Create new flavors for instances with hugepages for each HP on the hosts:

nova flavor-create h1.huge.hpgs auto 512 1 1
nova flavor-create h1.small.hpgs auto 1024 1 1

nova flavor-key h1.huge.hpgs set hw:mem_page_size=1048576
nova flavor-key h1.huge.hpgs set aggregate_instance_extra_specs:hpgs=true

nova flavor-key h1.small.hpgs set hw:mem_page_size=2048
nova flavor-key h1.small.hpgs set aggregate_instance_extra_specs:hpgs=true

    Add to the flavors requirement for the CPU pinning:

nova flavor-key h1.huge.hpgs set hw:cpu_policy=dedicated
nova flavor-key h1.huge.hpgs set aggregate_instance_extra_specs:pinned=true

nova flavor-key h1.small.hpgs set hw:cpu_policy=dedicated
nova flavor-key h1.small.hpgs set aggregate_instance_extra_specs:pinned=true

    Create instance with 2M flavor HP size
    Delete instace
    Create instance with 1Gb flavor size

Expected result:

   VMs are created

Actual result:

Vm with 1Gb is in Error state with "No valid host" error.

From libvirt log (node-2): qemuBuildNumaArgStr:6760 : internal error: Unable to find any usable hugetlbfs mount for 1048576 KiB

No mount point for 1Gb HugePages are present

[Arthur Svechnikov (asvechnikov)]

From node-2:

 root@node-2:~# mount
 /dev/mapper/os-root on / type ext4 (rw,errors=panic)
 proc on /proc type proc (rw,noexec,nosuid,nodev)
 sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
 none on /sys/fs/cgroup type tmpfs (rw)
 none on /sys/fs/fuse/connections type fusectl (rw)
 none on /sys/kernel/debug type debugfs (rw)
 none on /sys/kernel/security type securityfs (rw)
 udev on /dev type devtmpfs (rw,mode=0755)
 devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
 tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
 none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
 none on /run/shm type tmpfs (rw,nosuid,nodev)
 none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
 none on /sys/fs/pstore type pstore (rw)
 /dev/sda3 on /boot type ext2 (rw)
 /dev/mapper/vm-nova on /var/lib/nova type xfs (rw)
 systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
 hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,mode=775,gid=114)
 none on /sys/kernel/config type configfs (rw)

Libvirt configuration file /etc/libvirt/qemu.com contains line:

 # If provided by the host and a hugetlbfs mount point is configured,
 # a guest may request huge page backing. When this mount point is
 # unspecified here, determination of a host mount point in /proc/mounts
 # will be attempted. Specifying an explicit mount overrides detection
 # of the same in /proc/mounts. Setting the mount point to "" will
 # disable guest hugepage backing. If desired, multiple mount points can
 # be specified at once, separated by comma and enclosed in square
 # brackets, for example:
 #
 # hugetlbfs_mount = ["/dev/hugepages2M", "/dev/hugepages1G"]
 #
 # The size of huge page served by specific mount point is determined by
 # libvirt at the daemon startup.
 #
 # NB, within these mount points, guests will create memory backing
 # files in a location of $MOUNTPOINT/libvirt/qemu
 #
 #hugetlbfs_mount = "/dev/hugepages"

 hugetlbfs_mount = "/run/hugepages/kvm"

Seems that there should be 2 mount points for 2M and 1G. However when hugetlbfs_mount is set to ["/run/hugepages/kvm_2M", "/run/hugepages/kvm_1G"], the same (/run/hugepages/kvm) mount point is created after reboot.

[Ksenia Svechnikova (kdemina)]

Snapshot: https://drive.google.com/file/d/0B2v38w72jlwTajNGTmJGN2VQVFk/view?usp=sharing

[Atsuko Ito (yottatsa)]

/run/hugepages/kvm is predefined for /etc/init*/qemu-kvm.

We're explicitly specifying hugetlbfs_mount in /etc/libvirt/qemu.conf to disable autodiscovery of mountpoints (that will lead to bug if dpdk is used).

[Arthur Svechnikov (asvechnikov)]

Additional configuration should be applied:

Add creation of mount point to

file /usr/share/qemu/init/qemu-kvm-init

            mkdir -p /run/hugepages/kvm
            mount -t hugetlbfs hugetlbfs-kvm -o mode=775,gid=kvm /run/hugepages/kvm

changed to

            mkdir -p /run/hugepages/kvm
            mount -t hugetlbfs hugetlbfs-kvm -o mode=775,gid=kvm /run/hugepages/kvm
            mkdir -p /run/hugepages/kvm_1GB
            mount -t hugetlbfs hugetlbfs-kvm -o mode=775,gid=kvm,pagesize=1GB /run/hugepages/kvm_1GB

Grant the privilege to mount point

file /etc/apparmor.d/abstractions/libvirt-qemu

  # for access to hugepages
  owner "/run/hugepages/kvm/libvirt/qemu/**" rw,

changed to

  # for access to hugepages
  owner "/run/hugepages/kvm/libvirt/qemu/**" rw,
  owner "/run/hugepages/kvm_1GB/libvirt/qemu/**" rw,

Add mount point to libvirt config

file /etc/libvirt/qemu.conf

 hugetlbfs_mount = "/run/hugepages/kvm"

changed to

 hugetlbfs_mount = ["/run/hugepages/kvm", "/run/hugepages/kvm_1GB"]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/303580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/303580
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=10c3d88b55fbcd4427c4cb39690af9e0ae60175b
Submitter: Jenkins
Branch: master

commit 10c3d88b55fbcd4427c4cb39690af9e0ae60175b
Author: Sergey Kolekonov <email address hidden>
Date: Fri Apr 8 16:11:47 2016 +0300

    Fix simultaneous usage of 1Gb and 2Mb huge pages

    Currently it's impossible to use 1Gb and 2Mb huge pages on one compute node
    as a helper script from qemu package allows to use only one mount point
    for huge pages, so 1 Gb huge pages support has to be managed explicitly.

    This patch adds the following steps:
    - create a mount point for 1 Gb huge pages and mount pages in runtime
    - create a persistent record in /etc/fstab to mount huge pages on boot
    - add related settings for libvirt
    - update apparmor settings to allow an additional mount point

    Due to LP #1560532 Nailgun doesn't provide information about allocated
    huge pages when 1 Gb huge pages are used, so this information is computed
    in a custom fact.

    Closes-bug: #1566293

    Change-Id: Iba421abdc354afa7d89f6f10a94c6ba3edb99148

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/305741

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/305741
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c2b9eb320798357dc77557d638f995254edea2a9
Submitter: Jenkins
Branch: stable/mitaka

commit c2b9eb320798357dc77557d638f995254edea2a9
Author: Sergey Kolekonov <email address hidden>
Date: Fri Apr 8 16:11:47 2016 +0300

    Fix simultaneous usage of 1Gb and 2Mb huge pages

    Currently it's impossible to use 1Gb and 2Mb huge pages on one compute node
    as a helper script from qemu package allows to use only one mount point
    for huge pages, so 1 Gb huge pages support has to be managed explicitly.

    This patch adds the following steps:
    - create a mount point for 1 Gb huge pages and mount pages in runtime
    - create a persistent record in /etc/fstab to mount huge pages on boot
    - add related settings for libvirt
    - update apparmor settings to allow an additional mount point

    Due to LP #1560532 Nailgun doesn't provide information about allocated
    huge pages when 1 Gb huge pages are used, so this information is computed
    in a custom fact.

    Closes-bug: #1566293

    Change-Id: Iba421abdc354afa7d89f6f10a94c6ba3edb99148
    (cherry picked from commit 10c3d88b55fbcd4427c4cb39690af9e0ae60175b)

[Mikhail Chernik (mchernik)]

Verified on MOS 9.0 ISO 217, fixed.