dedicated instance on low latency worker node not appearing in C1 state

Bug #1838524 reported by Wendy Mitchell on 2019-07-31
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Medium
Jim Gauld

Bug Description

Brief Description
-----------------
Instance launched with dedicated cpu policy on worker with Lowlatency subfunction unexpectedly does not transition into the C1 idle-state

Severity
--------
Standard

Steps to Reproduce
------------------
1. On the worker node, run the followng to confirm C6 idle-stats are ~100%
$sudo cpupower monitor
2. Launch an instance with eg. 4 vcpu and hw:cpu_policy=dedicated
3. The instance lands on the worker node under test (eg. compute-0) which has lowlatency subfunction
4. Run the following command to confirm to get a report of the idle power state statistics for the dedicated cpus for the new instance.
$sudo virsh list
$sudo virsh dumpxml
$sudo cpupower monitor

Expected Behavior
------------------
Expect in step 4, that for the dedicated cpus (in this case 13, 19,57, 63) the C1 stat should have appeared to be roughly 100%

Actual Behavior
---------------
C6 state still appeared roughly 100% instead of C1 for the dedicated cpus under test.

compute-0:~$ sudo virsh list
 Id Name State
----------------------------------
 2 instance-00000007 paused

compute-0:~$ sudo virsh dumpxml instance-00000007
<domain type='kvm' id='2'>
  <name>instance-00000007</name>
  <uuid>44a4e835-d64b-4381-bcf5-ff2a9fb3f16b</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="19.0.2"/>
      <nova:name>fdafds</nova:name>
      <nova:creationTime>2019-07-31 12:09:55</nova:creationTime>
      <nova:flavor name="only4vcpu">
        <nova:memory>1024</nova:memory>
        <nova:disk>10</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="08df4dc23d0e40eb812297da8d043614">admin</nova:user>
        <nova:project uuid="847a07d6f6964049bca9f5984b011760">tenant1</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="9d1a2657-92f5-42d5-9a5a-ace5e87ae5b6"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='19'/>
    <vcpupin vcpu='1' cpuset='63'/>
    <vcpupin vcpu='2' cpuset='57'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <emulatorpin cpuset='13,19,57,63'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>19.0.2</entry>
      <entry name='serial'>44a4e835-d64b-4381-bcf5-ff2a9fb3f16b</entry>
      <entry name='uuid'>44a4e835-d64b-4381-bcf5-ff2a9fb3f16b</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client</model>
    <vendor>Intel</vendor>
    <topology sockets='2' cores='1' threads='2'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='topoext'/>
    <feature policy='disable' name='mpx'/>
    <feature policy='disable' name='xsavec'/>
    <feature policy='disable' name='xgetbv1'/>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576' unit='KiB'/>
    </numa>
  </cpu>

compute-0:~$ sudo cpupower monitor
Password:
Sorry, try again.
Password:
              |Nehalem || Mperf || Idle_Stats
PKG |CORE|CPU | C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1-B | C1E- | C3-B | C6-B
   0| 0| 0| 20.13| 13.33| 0.00| 0.00|| 20.15| 79.85| 3370|| 0.00| 2.46| 3.76| 44.34| 29.24
   0| 0| 44| 20.13| 13.33| 0.00| 0.00|| 18.92| 81.08| 3387|| 0.00| 2.08| 4.75| 11.85| 62.34
   0| 1| 1| 0.00| 0.00| 0.00| 0.00|| 96.90| 3.10| 3452|| 0.00| 0.00| 0.00| 0.00| 0.00
   0| 1| 45| 0.00| 0.00| 0.00| 0.00|| 0.00|100.00| 3384|| 0.00|100.00| 0.00| 0.00| 0.00
   0| 2| 2| 0.00| 0.00| 0.00| 0.00|| 96.87| 3.13| 3452|| 0.00| 0.00| 0.00| 0.00| 0.00
   0| 2| 46| 0.00| 0.00| 0.00| 0.00|| 0.00|100.00| 3400|| 0.00|100.00| 0.00| 0.00| 0.00
   0| 3| 3| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 3215|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 3| 47| 0.00| 99.66| 0.00| 0.00|| 0.01| 99.99| 3191|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 4| 4| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 3101|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 4| 48| 0.00| 99.69| 0.00| 0.00|| 0.01| 99.99| 3102|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 5| 5| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 3023|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 5| 49| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 3017|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 8| 6| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2954|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 8| 50| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2950|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 9| 7| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2876|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 9| 51| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2878|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 10| 8| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2802|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 10| 52| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2800|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 11| 9| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2878|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 11| 53| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2796|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 12| 10| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2798|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 12| 54| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2796|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 16| 11| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2798|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 16| 55| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2799|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 17| 12| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2799|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 17| 56| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2797|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 18| 13| 0.03| 99.55| 0.00| 0.00|| 0.03| 99.97| 3225|| 0.00| 0.00| 0.00| 0.08| 99.89
   0| 18| 57| 0.03| 99.52| 0.00| 0.00|| 0.01| 99.99| 2856|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 19| 14| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2800|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 19| 58| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2795|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 20| 15| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2799|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 20| 59| 0.00| 99.69| 0.00| 0.00|| 0.01| 99.99| 2797|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 21| 16| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2797|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 21| 60| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2797|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 24| 17| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2820|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 24| 61| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2816|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 25| 18| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2842|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 25| 62| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2837|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 26| 19| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2855|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 26| 63| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2864|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 27| 20| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2868|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 27| 64| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2884|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 28| 21| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2891|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 28| 65| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2903|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 0| 22| 0.00| 99.73| 0.00| 0.00|| 0.02| 99.98| 1790|| 0.00| 0.00| 0.00| 0.00| 99.98
   1| 0| 66| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1650|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 1| 23| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 1944|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 1| 67| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1600|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 2| 24| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2328|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 2| 68| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1913|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 3| 25| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2455|| 0.00| 0.00| 0.00| 0.00| 99.98
   1| 3| 69| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1819|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 4| 26| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2854|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 4| 70| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1716|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 5| 27| 0.00| 99.75| 0.00| 0.00|| 0.01| 99.99| 2548|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 5| 71| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2318|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 8| 28| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2485|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 8| 72| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2637|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 9| 29| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2675|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 9| 73| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2934|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 10| 30| 0.00| 99.72| 0.00| 0.00|| 0.01| 99.99| 2858|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 10| 74| 0.00| 99.72| 0.00| 0.00|| 0.01| 99.99| 2768|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 11| 31| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2970|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 11| 75| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 3009|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 12| 32| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2821|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 12| 76| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 3243|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 16| 33| 0.00| 99.72| 0.00| 0.00|| 0.01| 99.99| 2932|| 0.00| 0.00| 0.00| 0.00| 99.98
   1| 16| 77| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 3273|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 17| 34| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2673|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 17| 78| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 3122|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 18| 35| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2937|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 18| 79| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 3280|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 19| 36| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2885|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 19| 80| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 3163|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 20| 37| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2767|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 20| 81| 0.00| 99.69| 0.00| 0.00|| 0.01| 99.99| 3362|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 21| 38| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2765|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 21| 82| 0.00| 99.69| 0.00| 0.00|| 0.01| 99.99| 3071|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 24| 39| 0.00| 99.67| 0.00| 0.00|| 0.01| 99.99| 2974|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 24| 83| 0.00| 99.65| 0.00| 0.00|| 0.01| 99.99| 2894|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 25| 40| 0.00| 99.67| 0.00| 0.00|| 0.01| 99.99| 3177|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 25| 84| 0.00| 99.65| 0.00| 0.00|| 0.01| 99.99| 3032|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 26| 41| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 3189|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 26| 85| 0.00| 99.66| 0.00| 0.00|| 0.01| 99.99| 3207|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 27| 42| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 3076|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 27| 86| 0.00| 99.66| 0.00| 0.00|| 0.01| 99.99| 3112|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 28| 43| 0.00| 99.66| 0.00| 0.00|| 0.01| 99.99| 3048|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 28| 87| 0.00| 99.65| 0.00| 0.00|| 0.01| 99.99| 2764|| 0.00| 0.00| 0.00| 0.00| 99.99

Reproducibility
---------------
100%

System Configuration
--------------------
Standard config, low latency worker node

Branch/Pull Time/Commit
-----------------------
20190729T233000Z

Last Pass
---------

Timestamp/Logs
--------------
see output inline

Test Activity
-------------

Wendy Mitchell (wmitchellwr) wrote :

HW wcp 92-98

tags: added: stx.regression
Numan Waheed (nwaheed) on 2019-07-31
tags: added: stx.retestneeded
Brent Rowsell (brent-rowsell) wrote :

Where are the logs ? Please provide a collect from the worker node the VM was launched on.

Changed in starlingx:
status: New → Incomplete
Wendy Mitchell (wmitchellwr) wrote :
Wendy Mitchell (wmitchellwr) wrote :
Wendy Mitchell (wmitchellwr) wrote :
Wendy Mitchell (wmitchellwr) wrote :
Wendy Mitchell (wmitchellwr) wrote :

Logs attached

Brent Rowsell (brent-rowsell) wrote :

Logs show the libvirt hook was invoked but it does not seem to be working

Changed in starlingx:
status: Incomplete → Triaged
importance: Undecided → High
Brent Rowsell (brent-rowsell) wrote :

The libvirt qemu hook, /etc/libvirt/hooks/qemu should invoke script /usr/bin/set-cpu-wakeup-latency.sh to set the latency.

tags: added: stx2.0
tags: added: stx.2.0
removed: stx2.0
Frank Miller (sensfan22) wrote :

Setting priority to medium as this is a performance issue and not a functional issue.

Changed in starlingx:
importance: High → Medium
assignee: nobody → Jim Gauld (jgauld)
Ghada Khalil (gkhalil) wrote :

As per agreement with the community, moving all unresolved medium priority bugs from stx.2.0 to stx.3.0

tags: added: stx.3.0
removed: stx.2.0
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers