Nova filter rejects with unexpected filter reason from CoreFilter stating "Insufficient vcpus: req:6, avail:10.0"

[Wendy Mitchell]

Brief Description
-----------------
STX: Nova filter rejects with unexpected filter reason from CoreFilter

Severity
--------
Major

Steps to Reproduce
------------------

1: Create a flavor with 6 vcpus
eg. nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd::2]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne flavor-create cpu_rt_6 auto 1024 2 6

2: Set flavor extra specs with isolate thread policy
eg. {'hw:cpu_policy': 'dedicated', 'hw:cpu_realtime_mask': '^2-3', 'hw:numa_nodes': 1, 'hw:cpu_thread_policy': 'isolate', 'hw:cpu_realtime': 'yes'}

3: Boot a vm with the flavor containing isolate thread policy
eg. nova --os-username 'tenant2' --os-password 'Li69nux*' --os-project-name tenant2 --os-auth-url http://[abcd::2]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne boot --boot-volume f6ce6f6b-b364-4418-8d96-19bd4a9406d8 --key-name keypair-tenant2 --flavor 168aa10f-1962-4d4b-a902-db522ba611b1 --nic vif-model=virtio,net-id=5b9a95d2-fc95-4d43-bc9e-bdb72f5c1fc4 --nic vif-model=virtio,net-id=2b1398c3-3967-43e5-99ba-66fdd1738c0d tenant2-rt-yes_mask-^2-3_6vcpu-26 --poll'

Expected Behavior
------------------
The host, if being rejected by the corefilter, should provide the appropriate reason.
If there are really 10.0 vcpus available and only 6 are being requested, then would not expect it to be rejected for this reason:
Insufficient vcpus: req:6, avail:10.0

Actual Behavior
----------------
nova.utils corefilter rejects with an reason that does not make sense ie. Insufficient vcpus: req:6, avail:10.0

see reject reason in the nova-scheduler.log (doesn't make sense as it is requesting 6 and there is 10 available according to the reason nova.filters)

Reproducibility
---------------
Reproducible

System Configuration
--------------------
2 node (single processor)

Branch/Pull Time/Commit
-----------------------
Master as of date: 2018-09-19_21-38-00

Timestamp/Logs
--------------
see below nova-scheduler.log

2018-09-25 00:59:30.986 26167 INFO nova.utils [req-faa80ca1-ca50-400a-97cb-687cb1158bc6 14585a485444451b9b689db246d0395a c077c17d2f894a5e8cd3f101e9f6f9a9 - default default] CoreFilter: (controller-0) REJECT: Insufficient vcpus: req:6, avail:10.0
2018-09-25 00:59:30.988 26167 INFO nova.utils [req-faa80ca1-ca50-400a-97cb-687cb1158bc6 14585a485444451b9b689db246d0395a c077c17d2f894a5e8cd3f101e9f6f9a9 - default default] CoreFilter: (controller-1) REJECT: Insufficient vcpus: req:6, avail:10.0
2018-09-25 00:59:30.989 26167 INFO nova.filters [req-faa80ca1-ca50-400a-97cb-687cb1158bc6 14585a485444451b9b689db246d0395a c077c17d2f894a5e8cd3f101e9f6f9a9 - default default] Filter CoreFilter returned 0 hosts
2018-09-25 00:59:30.989 26167 INFO nova.filters [req-faa80ca1-ca50-400a-97cb-687cb1158bc6 14585a485444451b9b689db246d0395a c077c17d2f894a5e8cd3f101e9f6f9a9 - default default] Filtering removed all hosts for the request with instance ID 'bd43e689-5e94-4fe5-9c67-d5d9cfe10d02'. Filter results: ['RetryFilter: (start: 2, end: 2)', 'ComputeFilter: (start: 2, end: 2)', 'BaremetalFilter: (start: 2, end: 2)', 'AvailabilityZoneFilter: (start: 2, end: 2)', 'AggregateInstanceExtraSpecsFilter: (start: 2, end: 2)', 'RamFilter: (start: 2, end: 2)', 'ComputeCapabilitiesFilter: (start: 2, end: 2)', 'ImagePropertiesFilter: (start: 2, end: 2)', 'CoreFilter: (start: 2, end: 0)']
2018-09-25 00:59:31.043 26167 INFO nova.scheduler.host_manager [req-faa80ca1-ca50-400a-97cb-687cb1158bc6 14585a485444451b9b689db246d0395a c077c17d2f894a5e8cd3f101e9f6f9a9 - default default] metrics(controller-0): cpu.frequency=2509, cpu.idle.percent=81.0, cpu.idle.time=243683140000000, cpu.iowait.percent=0.0, cpu.iowait.time=749040000000, cpu.kernel.percent=1.0, cpu.kernel.time=3350150000000, cpu.percent=18.0, cpu.user.percent=16.0, cpu.user.time=47993650000000
2018-09-25 00:59:31.051 26167 INFO nova.scheduler.host_manager [req-faa80ca1-ca50-400a-97cb-687cb1158bc6 14585a485444451b9b689db246d0395a c077c17d2f894a5e8cd3f101e9f6f9a9 - default default] metrics(controller-1): cpu.frequency=2601, cpu.idle.percent=85.0, cpu.idle.time=175510790000000, cpu.iowait.percent=0.0, cpu.iowait.time=50030000000, cpu.kernel.percent=0.0, cpu.kernel.time=1196690000000, cpu.percent=14.0, cpu.user.percent=13.0, cpu.user.time=27979070000000
2018-09-25 00:59:31.054 26167 WARNING nova.scheduler.filter_scheduler [req-faa80ca1-ca50-400a-97cb-687cb1158bc6 14585a485444451b9b689db246d0395a c077c17d2f894a5e8cd3f101e9f6f9a9 - default default] CANNOT SCHEDULE: 0 available out of 1 requested. spec_obj=

[Ghada Khalil (gkhalil)]

From Daniel Chavolla:
This problem occurs when the flavor asks for 'hw:cpu_thread_policy': 'isolate' in a Hyper-Threaded lab, which in effect makes the requested number of cores to double, possibly requesting more cpus than available. Note that the problem is the poor error message, not that the VM would not schedule.

[Ghada Khalil (gkhalil)]

Low priority issue (nova inaccurate error msg); not targeted towards any release.

[Frank Miller (sensfan22)]

StarlingX is now aligned with OpenStack master. This issue should be re-tested and if the issue is reproduced then a nova launchpad created for the upstream nova team to decide if they will pursue a fix or not.