Max jitter for 24 hours cyclictest in guest is larger than expected
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
High
|
Alexander Kozyrev |
Bug Description
Brief Description
-----------------
Max jitter for 24 hours cyclictest in guest is larger than expected
Severity
--------
Major
Steps to Reproduce
------------------
1. create flavor with "hw:cpu_
nova flavor-show f.extra
+------
| Property | Value |
+------
| OS-FLV-
| OS-FLV-
| disk | 8 |
| extra_specs | {"hw:cpu_policy": "dedicated", "hw:mem_page_size": "1048576", "hw:cpu_sockets": "1", "hw:cpu_model": "Passthrough", "hw:cpu_
| id | 712f30a9-
| name | f.extra |
| os-flavor-
| ram | 8192 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 16 |
+------
2. Start the VM with above flavor.
3. Update guest cmdline
rw root=LABEL=
4. Insert the ptp module on guest to sync up time with host clock
$ modprobe ptp_kvm
You can use command “lsmod | grep ptp_kvm” to check the ptp_kvm module is already installed.
You can also create a ptp_kvm.conf file to make this module installed after system boot up.
$ echo ptp_kvm > /etc/modules-
Run below command to make guest time always sync up with host time
$ echo "refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0" >> /etc/chrony.conf
$ systemctl restart chronyd
$ chronyc sources | grep PHC0
5. run cyclictest for 24 hours in the guest.
./cyclictest -S -p99 -n -m -d0 -A ffff
# /dev/cpu_
policy: fifo: loadavg: 0.00 0.01 0.05 1/300 5743
T: 0 ( 3749) P:99 I:1000 C:153698899 Min: 3 Act: 4 Avg: 4 Max: 349
T: 1 ( 3750) P:99 I:1000 C:153698900 Min: 4 Act: 7 Avg: 6 Max: 104
T: 2 ( 3751) P:99 I:1000 C:153698886 Min: 4 Act: 7 Avg: 6 Max: 85
T: 3 ( 3752) P:99 I:1000 C:153698887 Min: 4 Act: 7 Avg: 6 Max: 100
T: 4 ( 3753) P:99 I:1000 C:153698882 Min: 4 Act: 6 Avg: 6 Max: 93
T: 5 ( 3754) P:99 I:1000 C:153698879 Min: 4 Act: 7 Avg: 6 Max: 94
T: 6 ( 3755) P:99 I:1000 C:153698876 Min: 5 Act: 7 Avg: 6 Max: 94
T: 7 ( 3756) P:99 I:1000 C:153698873 Min: 4 Act: 7 Avg: 6 Max: 99
T: 8 ( 3757) P:99 I:1000 C:153698869 Min: 5 Act: 7 Avg: 7 Max: 99
T: 9 ( 3758) P:99 I:1000 C:153698866 Min: 4 Act: 7 Avg: 6 Max: 103
T:10 ( 3759) P:99 I:1000 C:153698863 Min: 5 Act: 7 Avg: 7 Max: 107
T:11 ( 3760) P:99 I:1000 C:153698860 Min: 4 Act: 7 Avg: 6 Max: 112
T:12 ( 3761) P:99 I:1000 C:153698858 Min: 4 Act: 6 Avg: 6 Max: 102
T:13 ( 3762) P:99 I:1000 C:153698858 Min: 4 Act: 7 Avg: 6 Max: 101
T:14 ( 3763) P:99 I:1000 C:153698858 Min: 4 Act: 7 Avg: 6 Max: 100
T:15 ( 3764) P:99 I:1000 C:153698858 Min: 4 Act: 7 Avg: 6 Max: 103
Expected Behavior
------------------
Expected max jitter should be less than 20us
Actual Behavior
----------------
max jitter is greater than 100us
Reproducibility
---------------
Reproducible
System Configuration
-------
Hardware/Software Configuration for compute node
Server Model Skylake-SP (Wolf Pass) Comments
CPU model Model name: Intel® Xeon® Gold 6148 CPU @ 2.40GHz
Socket number 2
Cores per socket 20
NUMA nodes 2
NUMA node0 CPUs 0-19
NUMA node1 CPUs 20-39
Memory DDR4 2666MHz 16Gx6
Pre-condition: disable SMI and set Maximum performance profile on BIOS
VM Guest OS: 3.10.0-
Branch/Pull Time/Commit
-------
any starlingX build
Changed in starlingx: | |
assignee: | nobody → Alex Kozyrev (akozyrev) |
status: | New → Triaged |
importance: | Undecided → High |
tags: | added: stx.2019.03 stx.config |
Changed in starlingx: | |
status: | Triaged → In Progress |
Changed in starlingx: | |
status: | Fix Released → In Progress |
tags: |
added: stx.2019.05 removed: stx.2019.03 |
tags: |
added: stx.2.0 removed: stx.2019.05 |
Reviewed: https:/ /review. openstack. org/618308 /git.openstack. org/cgit/ openstack/ stx-config/ commit/ ?id=c3564823d41 46fb0cabebb8d36 8b6f25218f3572
Committed: https:/
Submitter: Zuul
Branch: master
commit c3564823d4146fb 0cabebb8d368b6f 25218f3572
Author: Alex Kozyrev <email address hidden>
Date: Thu Nov 15 14:04:24 2018 -0500
Disable QEMU memory balloon usage statistics.
Set mem_stats_ period_ seconds in nova.conf to 0.
It will disable QEMU memory balloon usage statistics.
There are 2 reasons of doing that in StarlingX:
1. StarlingX doesn’t support memory overcommit and adding
QEMU memory balloon device doesn’t make any difference.
2. QEMU memory balloon usage statistics interrupts a VM run
and causes unacceptable jitter in cyclictest once in a while.
Closes-bug: 1803615 36688f2deb61730 ab1d548b8b1
Change-Id: Iaea19626017557
Signed-off-by: Alex Kozyrev <email address hidden>