'hw:cpu_thread_policy=require' does not function correctly if NUMATopologyFilter is disabled

Bug #1550269 reported by Stephen Finucane on 2016-02-26
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Stephen Finucane

Bug Description

The 'require' policy is supposed to restrict instances to hosts that support hardware threads, e.g. HyperThreading. However, this filtering is done as part of the NUMATopologyFilter. If this filter is disabled, the host boots just fine. This should not be the case and needs to be fixed. Findings below.

---

Testing was conducted on a single-node, Fedora 23-based (4.3.5-300.fc23.x86_64)
OpenStack instance (built with devstack). The system is a dual-socket, ten core,
HT-enabled system (2 sockets * 10 cores * 2 threads = 40 "pCPUs".
0-9,20-29 = node0, 10-19,30-39 = node1).

Commit '8bafc9' of Nova was used.

# Steps

## Create flavors

    $ openstack flavor create pinned.require \
        --id 102 --ram 2048 --disk 0 --vcpus 4
    $ openstack flavor set pinned.require \
        --property "hw:cpu_policy=dedicated" \
        --property "hw:cpu_thread_policy=require"

## Validate a HT-enabled node

The 'require' case is a stricter version of the `prefer` case, in that it
should fail if we have HyperThreading disabled, do not have enough free
sibling sets, or have no HyperThreading support at all. However, since we're
not hitting any of these conditions on this host, things should function just
like they do for the `prefer` case. Therefore, the guest should see a two
sockets with one core per socket and two threads per core.

    $ openstack server create --flavor=pinned.require \
        --image=cirros-0.3.4-x86_64-uec --wait test1

    $ sudo virsh list
     Id Name State
    ----------------------------------------------------
     2 instance-00000002 running

    $ sudo virsh dumpxml 2
    <domain type='kvm' id='2'>
      <name>instance-00000002</name>
      ...
      <vcpu placement='static'>4</vcpu>
      <cputune>
        <shares>4096</shares>
        <vcpupin vcpu='0' cpuset='1'/>
        <vcpupin vcpu='1' cpuset='21'/>
        <vcpupin vcpu='2' cpuset='0'/>
        <vcpupin vcpu='3' cpuset='20'/>
        <emulatorpin cpuset='0-1,20-21'/>
      </cputune>
      <numatune>
        <memory mode='strict' nodeset='0'/>
        <memnode cellid='0' mode='strict' nodeset='0'/>
      </numatune>
      ...
      <cpu>
        <topology sockets='2' cores='1' threads='2'/>
        <numa>
          <cell id='0' cpus='0-3' memory='2097152' unit='KiB'/>
        </numa>
      </cpu>
      ...
    </domain>

    $ openstack server delete test1

No issues here.

## Validate a HT-disabled node

This policy "requires" HyperThreading or similar on the host, so it shouldn't
work here.

    $ openstack server create --flavor=pinned.require \
        --image=cirros-0.3.4-x86_64-uec --wait test1

    $ sudo virsh list
     Id Name State
    ----------------------------------------------------
     2 instance-00000002 running

    $ sudo virsh dumpxml 2
    <domain type='kvm' id='2'>
      <name>instance-00000002</name>
      ...
      <vcpu placement='static'>4</vcpu>
      <cputune>
        <shares>4096</shares>
        <vcpupin vcpu='0' cpuset='0'/>
        <vcpupin vcpu='1' cpuset='1'/>
        <vcpupin vcpu='2' cpuset='2'/>
        <vcpupin vcpu='3' cpuset='3'/>
        <emulatorpin cpuset='0-3'/>
      </cputune>
      <numatune>
        <memory mode='strict' nodeset='0'/>
        <memnode cellid='0' mode='strict' nodeset='0'/>
      </numatune>
      ...
      <cpu>
        <topology sockets='2' cores='1' threads='2'/>
        <numa>
          <cell id='0' cpus='0-3' memory='2097152' unit='KiB'/>
        </numa>
      </cpu>
      ...
    </domain>

    $ openstack server delete test1

This is a problem, but we do not currently have the filter activated:

    $ cat /etc/nova/nova.conf | grep scheduler_default_filters
    scheduler_default_filters = RetryFilter,AvailabilityZoneFilter,RamFilter,\
    DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,\
    ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,\
    DifferentHostFilter

Let's activate this:

    $ cat /etc/nova/nova.conf | grep scheduler_default_filters
    scheduler_default_filters = RetryFilter,AvailabilityZoneFilter,RamFilter,\
    DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,\
    ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,\
    DifferentHostFilter,NUMATopologyFilter

And try again:

    $ openstack server create --flavor=pinned.require \
        --image=cirros-0.3.4-x86_64-uec --wait test1
    Error creating server: test1

    Error creating server

That's more like it, but it shouldn't be necessary.

Fix proposed to branch: master
Review: https://review.openstack.org/285232

Changed in nova:
assignee: nobody → Stephen Finucane (sfinucan)
status: New → In Progress
tags: added: mitaka-backport-potential

Reviewed: https://review.openstack.org/285232
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ad0047e97b2847412ee28bad6a3bfb48395add35
Submitter: Jenkins
Branch: master

commit ad0047e97b2847412ee28bad6a3bfb48395add35
Author: Stephen Finucane <email address hidden>
Date: Fri Feb 26 10:55:55 2016 +0000

    virt/hardware: Check for threads when "required"

    The 'require' case "requires" the presence of hardware threads on a
    host. At present, this check is done using the NUMATopology filter.
    Unfortunately, this means that if this filter is disabled then
    instances can be scheduled on invalid hosts. Resolve this by adding a
    new check to be run when hosts are actually scheduling.

    Change-Id: Ia9e4784e02ca9ce7a3d81c962b95bee100f6db42
    Closes-bug: #1550269

Changed in nova:
status: In Progress → Fix Released

This issue was fixed in the openstack/nova 14.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers