[18.04 FEAT] Add kvm_stat from kernel tree

[bugproxy]

Improves RAS characterics of systems running KVM by providing kvm_stat script:
https://github.com/torvalds/linux/tree/master/tools/kvm/kvm_stat

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-11-23 09:23 EDT-------
kvm_stat is a python script located in the Linux kernel source tree and provides useful KVM module statistics that can help debug performance and functional issues of KVM systems.

[Christian Ehrhardt  (paelzer)]

Isn't that covered in the way everybody does it via "perf" [1] ?

Can report the same and is fairly known by everyone right?
Here an example of combined generic and s390 events of start/stopping a guest.

sudo perf stat -e 'kvm:*' -a sleep 1h
^Csleep: Interrupt

 Performance counter stats for 'system wide':

                 0 kvm:kvm_s390_skey_related_inst
                 0 kvm:kvm_s390_major_guest_pfault
                 0 kvm:kvm_s390_pfault_init
                 0 kvm:kvm_s390_pfault_done
            36.984 kvm:kvm_s390_sie_enter
                 1 kvm:kvm_s390_sie_fault
            36.984 kvm:kvm_s390_sie_exit
            16.747 kvm:kvm_s390_intercept_instruction
                 0 kvm:kvm_s390_intercept_prog
                 0 kvm:kvm_s390_intercept_validity
                 0 kvm:kvm_s390_handle_sigp
                 0 kvm:kvm_s390_handle_sigp_pei
            15.542 kvm:kvm_s390_handle_diag
                 1 kvm:kvm_s390_handle_lctl
                 0 kvm:kvm_s390_handle_stctl
                11 kvm:kvm_s390_handle_prefix
                 3 kvm:kvm_s390_handle_stap
                 3 kvm:kvm_s390_handle_stfl
                 8 kvm:kvm_s390_handle_stsi
                 0 kvm:kvm_s390_handle_operexc
                 0 kvm:kvm_s390_handle_sthyi
               973 kvm:kvm_userspace_exit
                76 kvm:kvm_vcpu_wakeup
               320 kvm:kvm_set_irq
                 0 kvm:kvm_ack_irq
                 0 kvm:kvm_mmio
                 0 kvm:kvm_fpu
                 0 kvm:kvm_age_page
                 0 kvm:kvm_try_async_get_page
                 0 kvm:kvm_async_pf_doublefault
                 0 kvm:kvm_async_pf_not_present
                 0 kvm:kvm_async_pf_ready
                 0 kvm:kvm_async_pf_completed
                30 kvm:kvm_halt_poll_n...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-11-23 10:10 EDT-------
Yes it was believed that perf events will eventually replace the kvm_stat. The KVM community has now a different opinion. The kvm_stat counter in debugfs have value and they can provide insights that the perf events will not have.

e.g. see QEMU commit

commit b763adf1a6b271f17f15ea31fae93d03e980d911
Author: Paolo Bonzini <email address hidden>
AuthorDate: Wed May 21 12:42:26 2014 +0200
Commit: Paolo Bonzini <email address hidden>
CommitDate: Wed May 21 17:56:37 2014 +0200

kvm_stat: allow choosing between tracepoints and old stats

The old stats contain information not available in the tracepoints.
By default, keep the old behavior, but allow choosing which set of stats
to present, or even both.

Inspired by a patch from Marcelo Tosatti.

Signed-off-by: Paolo Bonzini <email address hidden>

FWIW, there have been a lot of updates to kvm_stat in the last 2 years.

[Christian Ehrhardt  (paelzer)]

Thanks Christian, I think now things are in shape to convince the kernel Team to do so.

The linux-tools-common or something like it will still be the right package IMHO.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-26 09:41 EDT-------
We would like getting the kvm_stat version as it will appear in kernel 4.17?

Specifically, we'd be interested in the following commits (already queued for 4.17 in the kvm maintainer's tree at https://git.kernel.org/pub/scm/virt/kvm/kvm.git/log/?h=queue):

faa312a543283c717342cd332b5b9247bd305dce tools/kvm_stat: simplify the sortkey function
006f1548ac13d67d21865416a0f4e8062df1a85f tools/kvm_stat: use a namedtuple for storing the values
0eb578009a1d530a11846d7c4733a5db04730884 tools/kvm_stat: use a more pythonic way to iterate over dictionaries
369d5a85bb782ecf63c5bae9686c7e6104eea991 tools/kvm_stat: avoid 'is' for equality checks
3df33a0f34a3883b6696bff8cc8fcda3c7444a62 tools/kvm_stat: fix crash when filtering out all non-child trace events
1cd8bfb1ed9962be6d80d5020508922aa93653ac tools/kvm_stat: print error on invalid regex
1fd6a708c8438403dee17eb411cf81ffba13cf43 tools/kvm_stat: fix debugfs handling
c0e8c21eae616ed8703c1b4b01046a1578ee875c tools/kvm_stat: mark private methods as such
516f1190a1e0cce12128a6446e6438745c8de62a tools/kvm_stat: eliminate extra guest/pid selection dialog
18e8f4100ef14f924514fbd91eb67bd5fa5396b7 tools/kvm_stat: separate drilldown and fields filtering
df72ecfc790fa01de1c41f836ff51d12f9c40318 tools/kvm_stat: group child events indented after parent
6789af030a462708f937137e05eb12ea009fb348 tools/kvm_stat: print 'Total' line for multiple events only

[Dimitri John Ledkov (xnox)]

"as it will appear in kernel 4.17?"

Note, that 18.04 is currently targetting v4.15, thus above mentioned commits, will need to be assessed by the kernel team for suitability to be cherry-picked into v4.15 kernel. At least to me, the commits mentioned, look mostly harmless.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-02-27 07:36 EDT-------
The patches are now in Linus' tree for 4.16(!), here are the respective git commit IDs:

faa312a543283c717342cd332b5b9247bd305dce tools/kvm_stat: simplify the sortkey function
006f1548ac13d67d21865416a0f4e8062df1a85f tools/kvm_stat: use a namedtuple for storing the values
0eb578009a1d530a11846d7c4733a5db04730884 tools/kvm_stat: use a more pythonic way to iterate over dictionaries
369d5a85bb782ecf63c5bae9686c7e6104eea991 tools/kvm_stat: avoid 'is' for equality checks
3df33a0f34a3883b6696bff8cc8fcda3c7444a62 tools/kvm_stat: fix crash when filtering out all non-child trace events
1cd8bfb1ed9962be6d80d5020508922aa93653ac tools/kvm_stat: print error on invalid regex
1fd6a708c8438403dee17eb411cf81ffba13cf43 tools/kvm_stat: fix debugfs handling
c0e8c21eae616ed8703c1b4b01046a1578ee875c tools/kvm_stat: mark private methods as such
516f1190a1e0cce12128a6446e6438745c8de62a tools/kvm_stat: eliminate extra guest/pid selection dialog
18e8f4100ef14f924514fbd91eb67bd5fa5396b7 tools/kvm_stat: separate drilldown and fields filtering
df72ecfc790fa01de1c41f836ff51d12f9c40318 tools/kvm_stat: group child events indented after parent
6789af030a462708f937137e05eb12ea009fb348 tools/kvm_stat: print 'Total' line for multiple events only

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-05 05:04 EDT-------
Canonical. any update on this LP ? Many thx in advance

[Frank Heimes (fheimes)]

That's currently in discussion ...

[Frank Heimes (fheimes)]

Please make sure that the updates/patches to that script do not break python3 compatibility
and that the script stays kernel independent.
Potentially new python scripts to be shipped should be python3 (compatible).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-06 14:43 EDT-------
Hi Frank,
We have actually contributed some some patches which improve python3 support, so that's definitely on our minds. These patches certainly don't make things worse. I did a quick test, and kvm_stat as is after applying above patches would run with python3. Whether there's a conscious decision to support python3 or not is ultimately up to the community, though.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-26 01:05 EDT-------
Frank,
Check the following patch series - it fixes at least one python3 glitch:
https://www.spinics.net/lists/kvm/msg166583.html

[Dimitri John Ledkov (xnox)]

I believe the best way forward is to postpone this request to 18.10, which will be based on the 4.16 (or later) kernel.
Then it will be included in the tools packages, and arrive in 18.04 via -hwe kernel, verbantim.
In the meantime, it is quite trivial to get that single script onto systems, by downloading it from kernel.org browsing.
There are very little incentives to maintain backports onto v4.15 of all of these recent patches, of which there probably still more to come.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-26 10:09 EDT-------
Uh, postponing to 18.10 wouldn't be so great: 18.04 is an LTS release, and we imagine that this tool will come in very handy when debugging performance-related KVM issues in the field. And, mind you, some of our customers shut down their environments for security reasons - so adding a script to run as root is not so easy! So not having kvm_stat in 18.04 is a big deal for us.
We could drop the commits mentioned in comment #17 - that was just courtesy on our part, since a previous comment mentioned that python 3 support is a big deal for Ubuntu.
Or maybe even drop the additional patches requested in comment #12 entirely.
To us, anything is better than not having kvm_stat readily available at customers at all.

[Joseph Salisbury (jsalisbury)]

This bug is still being investigated for the Bionic release.

[Seth Forshee (sforshee)]

Pull request sent to kernel-team list.

https://lists.ubuntu.com/archives/kernel-team/2018-April/091477.html

I've split this out into a new package for a couple of reasons, primarily to avoid adding a python dependency to our existing tools packages. It would be in a package named linux-tools-host which would be for utilities specifically for use on VM hosts, currently only kvm_stat.

[Frank Heimes (fheimes)]

kvm_stats now available for testing via package linux-tools-host from bionic-proposed:

$ apt-cache policy linux-tools-host
linux-tools-host:
  Installed: 4.15.0-17.18
  Candidate: 4.15.0-17.18
  Version table:
 *** 4.15.0-17.18 500
        500 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/main s390x Packages
        100 /var/lib/dpkg/status

$ which kvm_stat
/usr/bin/kvm_stat

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-04-18 13:09 EDT-------
Hi Frank,
is there a place I cab grab the binary from? Couldn't find it anywhere at http://ports.ubuntu.com/ubuntu-ports. Or do I need to access it via a Binonic install...?!

[Frank Heimes (fheimes)]

Hi, with the status Fix Committed it became available via the bionic-proposed archive 'pocket'.
So you have to enable proposed (for now):
$ sudo add-apt-repository "deb http://us.ports.ubuntu.com/ubuntu-ports/ $(lsb_release -sc)-proposed restricted main multiverse universe"
Do a quick check if it worked:
$ grep -R --exclude=*{.save,.*~} "^[^#].*-proposed"
Update your package index:
$ sudo apt update
And you will be able to access and install it as part of the 'linux-tools-host' package:
$ apt-cache policy linux-tools-host
linux-tools-host:
  Installed: 4.15.0-17.18
  Candidate: 4.15.0-17.18
  Version table:
 *** 4.15.0-17.18 500
        500 http://ports.ubuntu.com/ubuntu-ports bionic-proposed/main s390x Packages
        100 /var/lib/dpkg/status
Now just install it with:
$ sudo apt install linux-tools-host
It's located here:
$ which kvm_stat
/usr/bin/kvm_stat

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-04-19 02:58 EDT-------
Uh, doesn't work for whatever reason on my 17.10 :/
Could you attach the kvm_stat file to this bugzilla..?

[Frank Heimes (fheimes)]

Since all Ubuntu packages are hosted on Launchpad you can find it here:
https://launchpad.net/ubuntu/bionic/+package/linux-tools-host
ddl of current / initial version:
http://launchpadlibrarian.net/365926070/linux-tools-host_4.15.0-17.18_all.deb

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-04-19 04:30 EDT-------
Ah, that was easy enough :)
Tested that version, alll good!

[Frank Heimes (fheimes)]

Ok, and thx for having a look at it!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-19.20

---------------
linux (4.15.0-19.20) bionic; urgency=medium

  * linux: 4.15.0-19.20 -proposed tracker (LP: #1766021)

  * Kernel 4.15.0-15 breaks Dell PowerEdge 12th Gen servers (LP: #1765232)
    - Revert "blk-mq: simplify queue mapping & schedule with each possisble CPU"
    - Revert "genirq/affinity: assign vectors to all possible CPUs"

linux (4.15.0-18.19) bionic; urgency=medium

  * linux: 4.15.0-18.19 -proposed tracker (LP: #1765490)

  * [regression] Ubuntu 18.04:[4.15.0-17-generic #18] KVM Guest Kernel:
    meltdown: rfi/fallback displacement flush not enabled bydefault (kvm)
    (LP: #1765429)
    - powerpc/pseries: Fix clearing of security feature flags

  * signing: only install a signed kernel (LP: #1764794)
    - [Packaging] update to Debian like control scripts
    - [Packaging] switch to triggers for postinst.d postrm.d handling
    - [Packaging] signing -- switch to raw-signing tarballs
    - [Packaging] signing -- switch to linux-image as signed when available
    - [Config] signing -- enable Opal signing for ppc64el
    - [Packaging] printenv -- add signing options

  * [18.04 FEAT] Sign POWER host/NV kernels (LP: #1696154)
    - [Packaging] signing -- add support for signing Opal kernel binaries

  * Please cherrypick s390 unwind fix (LP: #1765083)
    - s390/compat: fix setup_frame32

  * Ubuntu 18.04 installer does not detect any IPR based HDD/RAID array [S822L]
    [ipr] (LP: #1751813)
    - d-i: move ipr to storage-core-modules on ppc64el

  * drivers/gpu/drm/bridge/adv7511/adv7511.ko missing (LP: #1764816)
    - SAUCE: (no-up) rename the adv7511 drm driver to adv7511_drm

  * Miscellaneous Ubuntu changes
    - [Packaging] Add linux-oem to rebuild test blacklist.

linux (4.15.0-17.18) bionic; urgency=medium

  * linux: 4.15.0-17.18 -proposed tracker (LP: #1764498)

  * Eventual OOM with profile reloads (LP: #1750594)
    - SAUCE: apparmor: fix memory leak when duplicate profile load

linux (4.15.0-16.17) bionic; urgency=medium

  * linux: 4.15.0-16.17 -proposed tracker (LP: #1763785)

  * [18.04] [bug] CFL-S(CNP)/CNL GPIO testing failed (LP: #1757346)
    - [Config]: Set CONFIG_PINCTRL_CANNONLAKE=y

  * [Ubuntu 18.04] USB Type-C test failed on GLK (LP: #1758797)
    - SAUCE: usb: typec: ucsi: Increase command completion timeout value

  * Fix trying to "push" an already active pool VP (LP: #1763386)
    - SAUCE: powerpc/xive: Fix trying to "push" an already active pool VP

  * hisi_sas: Revert and replace SAUCE patches w/ upstream (LP: #1762824)
    - Revert "UBUNTU: SAUCE: scsi: hisi_sas: export device table of v3 hw to
      userspace"
    - Revert "UBUNTU: SAUCE: scsi: hisi_sas: config for hip08 ES"
    - scsi: hisi_sas: modify some register config for hip08
    - scsi: hisi_sas: add v3 hw MODULE_DEVICE_TABLE()

  * Realtek card reader - RTS5243 [VEN_10EC&DEV_5260] (LP: #1737673)
    - misc: rtsx: Move Realtek Card Reader Driver to misc
    - updateconfigs for Realtek Card Reader Driver
    - misc: rtsx: Add support for RTS5260
    - misc: rtsx: Fix symbol clashes

  * Mellanox [mlx5] [bionic] UBSAN: Undefined behaviour in
    ./include/linux/net_dim.h (LP: #1...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-04-24 03:39 EDT-------
IBM bugzilla status-> closed; Fix Released by Canonical

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Frank Heimes (fheimes)]

Ticket is already Fix Released - hence verification is no longer needed - adjusting the tags ...