StarlingX

dedicated instance on low latency worker node not appearing in C1 state

Bug #1838524 reported by Wendy Mitchell
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Won't Fix
Low
Jim Gauld

Bug Description

Brief Description
-----------------
Instance launched with dedicated cpu policy on worker with Lowlatency subfunction unexpectedly does not transition into the C1 idle-state

Severity
--------
Standard

Steps to Reproduce
------------------
1. On the worker node, run the followng to confirm C6 idle-stats are ~100%
$sudo cpupower monitor
2. Launch an instance with eg. 4 vcpu and hw:cpu_policy=dedicated
3. The instance lands on the worker node under test (eg. compute-0) which has lowlatency subfunction
4. Run the following command to confirm to get a report of the idle power state statistics for the dedicated cpus for the new instance.
$sudo virsh list
$sudo virsh dumpxml
$sudo cpupower monitor

Expected Behavior
------------------
Expect in step 4, that for the dedicated cpus (in this case 13, 19,57, 63) the C1 stat should have appeared to be roughly 100%

Actual Behavior
---------------
C6 state still appeared roughly 100% instead of C1 for the dedicated cpus under test.

compute-0:~$ sudo virsh list
 Id Name State
----------------------------------
 2 instance-00000007 paused

compute-0:~$ sudo virsh dumpxml instance-00000007
<domain type='kvm' id='2'>
  <name>instance-00000007</name>
  <uuid>44a4e835-d64b-4381-bcf5-ff2a9fb3f16b</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="19.0.2"/>
      <nova:name>fdafds</nova:name>
      <nova:creationTime>2019-07-31 12:09:55</nova:creationTime>
      <nova:flavor name="only4vcpu">
        <nova:memory>1024</nova:memory>
        <nova:disk>10</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="08df4dc23d0e40eb812297da8d043614">admin</nova:user>
        <nova:project uuid="847a07d6f6964049bca9f5984b011760">tenant1</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="9d1a2657-92f5-42d5-9a5a-ace5e87ae5b6"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='19'/>
    <vcpupin vcpu='1' cpuset='63'/>
    <vcpupin vcpu='2' cpuset='57'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <emulatorpin cpuset='13,19,57,63'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>19.0.2</entry>
      <entry name='serial'>44a4e835-d64b-4381-bcf5-ff2a9fb3f16b</entry>
      <entry name='uuid'>44a4e835-d64b-4381-bcf5-ff2a9fb3f16b</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client</model>
    <vendor>Intel</vendor>
    <topology sockets='2' cores='1' threads='2'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='topoext'/>
    <feature policy='disable' name='mpx'/>
    <feature policy='disable' name='xsavec'/>
    <feature policy='disable' name='xgetbv1'/>
    <numa>
      <cell id='0' cpus='0-3' memory='1048576' unit='KiB'/>
    </numa>
  </cpu>

compute-0:~$ sudo cpupower monitor
Password:
Sorry, try again.
Password:
              |Nehalem || Mperf || Idle_Stats
PKG |CORE|CPU | C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1-B | C1E- | C3-B | C6-B
   0| 0| 0| 20.13| 13.33| 0.00| 0.00|| 20.15| 79.85| 3370|| 0.00| 2.46| 3.76| 44.34| 29.24
   0| 0| 44| 20.13| 13.33| 0.00| 0.00|| 18.92| 81.08| 3387|| 0.00| 2.08| 4.75| 11.85| 62.34
   0| 1| 1| 0.00| 0.00| 0.00| 0.00|| 96.90| 3.10| 3452|| 0.00| 0.00| 0.00| 0.00| 0.00
   0| 1| 45| 0.00| 0.00| 0.00| 0.00|| 0.00|100.00| 3384|| 0.00|100.00| 0.00| 0.00| 0.00
   0| 2| 2| 0.00| 0.00| 0.00| 0.00|| 96.87| 3.13| 3452|| 0.00| 0.00| 0.00| 0.00| 0.00
   0| 2| 46| 0.00| 0.00| 0.00| 0.00|| 0.00|100.00| 3400|| 0.00|100.00| 0.00| 0.00| 0.00
   0| 3| 3| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 3215|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 3| 47| 0.00| 99.66| 0.00| 0.00|| 0.01| 99.99| 3191|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 4| 4| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 3101|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 4| 48| 0.00| 99.69| 0.00| 0.00|| 0.01| 99.99| 3102|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 5| 5| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 3023|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 5| 49| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 3017|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 8| 6| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2954|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 8| 50| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2950|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 9| 7| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2876|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 9| 51| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2878|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 10| 8| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2802|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 10| 52| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2800|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 11| 9| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2878|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 11| 53| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2796|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 12| 10| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2798|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 12| 54| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2796|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 16| 11| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2798|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 16| 55| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2799|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 17| 12| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2799|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 17| 56| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2797|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 18| 13| 0.03| 99.55| 0.00| 0.00|| 0.03| 99.97| 3225|| 0.00| 0.00| 0.00| 0.08| 99.89
   0| 18| 57| 0.03| 99.52| 0.00| 0.00|| 0.01| 99.99| 2856|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 19| 14| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2800|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 19| 58| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2795|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 20| 15| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2799|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 20| 59| 0.00| 99.69| 0.00| 0.00|| 0.01| 99.99| 2797|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 21| 16| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2797|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 21| 60| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2797|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 24| 17| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2820|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 24| 61| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2816|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 25| 18| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2842|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 25| 62| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2837|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 26| 19| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2855|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 26| 63| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2864|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 27| 20| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2868|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 27| 64| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2884|| 0.00| 0.00| 0.00| 0.00|100.00
   0| 28| 21| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2891|| 0.00| 0.00| 0.00| 0.00| 99.99
   0| 28| 65| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 2903|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 0| 22| 0.00| 99.73| 0.00| 0.00|| 0.02| 99.98| 1790|| 0.00| 0.00| 0.00| 0.00| 99.98
   1| 0| 66| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1650|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 1| 23| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 1944|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 1| 67| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1600|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 2| 24| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2328|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 2| 68| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1913|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 3| 25| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2455|| 0.00| 0.00| 0.00| 0.00| 99.98
   1| 3| 69| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1819|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 4| 26| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2854|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 4| 70| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 1716|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 5| 27| 0.00| 99.75| 0.00| 0.00|| 0.01| 99.99| 2548|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 5| 71| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2318|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 8| 28| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2485|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 8| 72| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2637|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 9| 29| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2675|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 9| 73| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2934|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 10| 30| 0.00| 99.72| 0.00| 0.00|| 0.01| 99.99| 2858|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 10| 74| 0.00| 99.72| 0.00| 0.00|| 0.01| 99.99| 2768|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 11| 31| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2970|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 11| 75| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 3009|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 12| 32| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2821|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 12| 76| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 3243|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 16| 33| 0.00| 99.72| 0.00| 0.00|| 0.01| 99.99| 2932|| 0.00| 0.00| 0.00| 0.00| 99.98
   1| 16| 77| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 3273|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 17| 34| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 2673|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 17| 78| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 3122|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 18| 35| 0.00| 99.74| 0.00| 0.00|| 0.01| 99.99| 2937|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 18| 79| 0.00| 99.73| 0.00| 0.00|| 0.01| 99.99| 3280|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 19| 36| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2885|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 19| 80| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 3163|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 20| 37| 0.00| 99.71| 0.00| 0.00|| 0.01| 99.99| 2767|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 20| 81| 0.00| 99.69| 0.00| 0.00|| 0.01| 99.99| 3362|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 21| 38| 0.00| 99.70| 0.00| 0.00|| 0.01| 99.99| 2765|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 21| 82| 0.00| 99.69| 0.00| 0.00|| 0.01| 99.99| 3071|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 24| 39| 0.00| 99.67| 0.00| 0.00|| 0.01| 99.99| 2974|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 24| 83| 0.00| 99.65| 0.00| 0.00|| 0.01| 99.99| 2894|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 25| 40| 0.00| 99.67| 0.00| 0.00|| 0.01| 99.99| 3177|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 25| 84| 0.00| 99.65| 0.00| 0.00|| 0.01| 99.99| 3032|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 26| 41| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 3189|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 26| 85| 0.00| 99.66| 0.00| 0.00|| 0.01| 99.99| 3207|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 27| 42| 0.00| 99.68| 0.00| 0.00|| 0.01| 99.99| 3076|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 27| 86| 0.00| 99.66| 0.00| 0.00|| 0.01| 99.99| 3112|| 0.00| 0.00| 0.00| 0.00|100.00
   1| 28| 43| 0.00| 99.66| 0.00| 0.00|| 0.01| 99.99| 3048|| 0.00| 0.00| 0.00| 0.00| 99.99
   1| 28| 87| 0.00| 99.65| 0.00| 0.00|| 0.01| 99.99| 2764|| 0.00| 0.00| 0.00| 0.00| 99.99

Reproducibility
---------------
100%

System Configuration
--------------------
Standard config, low latency worker node

Branch/Pull Time/Commit
-----------------------
20190729T233000Z

Last Pass
---------

Timestamp/Logs
--------------
see output inline

Test Activity
-------------

Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

HW wcp 92-98

tags: added: stx.regression
Numan Waheed (nwaheed)
tags: added: stx.retestneeded
Revision history for this message
Brent Rowsell (brent-rowsell) wrote :

Where are the logs ? Please provide a collect from the worker node the VM was launched on.

Changed in starlingx:
status: New → Incomplete
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

Logs attached

Revision history for this message
Brent Rowsell (brent-rowsell) wrote :

Logs show the libvirt hook was invoked but it does not seem to be working

Changed in starlingx:
status: Incomplete → Triaged
importance: Undecided → High
Revision history for this message
Brent Rowsell (brent-rowsell) wrote :

The libvirt qemu hook, /etc/libvirt/hooks/qemu should invoke script /usr/bin/set-cpu-wakeup-latency.sh to set the latency.

Brent Rowsell (brent-rowsell)
tags: added: stx2.0
tags: added: stx.2.0
removed: stx2.0
Revision history for this message
Frank Miller (sensfan22) wrote :

Setting priority to medium as this is a performance issue and not a functional issue.

Changed in starlingx:
importance: High → Medium
assignee: nobody → Jim Gauld (jgauld)
Revision history for this message
Ghada Khalil (gkhalil) wrote :

As per agreement with the community, moving all unresolved medium priority bugs from stx.2.0 to stx.3.0

tags: added: stx.3.0
removed: stx.2.0
Revision history for this message
Ghada Khalil (gkhalil) wrote :

As per agreement with the community, marking unresolved medium priority bugs (>= 100 days AND not recently reproduced) from stx.3.0 to Low priority / no target release

tags: removed: stx.3.0
Changed in starlingx:
importance: Medium → Low
tags: added: stx.config stx.distro.openstack
Revision history for this message
Ramaswamy Subramanian (rsubrama) wrote :

No progress on this bug for more than 2 years. Candidate for closure.

If there is no update, this issue is targeted to be closed as 'Won't Fix' in 2 weeks.

Revision history for this message
Ramaswamy Subramanian (rsubrama) wrote :

Changing the status to 'Won't Fix' as there is no activity.

Changed in starlingx:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.