[P9][LTCTest][Opal][FW910] cpupower monitor shows multiple stop Idle_Stats

[bugproxy]

== Comment: #0 - PAVAMAN SUBRAMANIYAM <email address hidden> - 2017-06-29 02:30:22 ==
---Problem Description---
cpupower monitor shows multiple stop Idle_Stats

---uname output---
Linux zz376p1 4.10.0-26-generic #30~16.04.1-Ubuntu SMP Tue Jun 27 09:38:48 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = P9

---Debugger---
A debugger is not configured

---Steps to Reproduce---
Install a P9 8375-42A Hardware with Ubuntu 16.04.3 OS.
Then execute the cpupower monitor command to fetch all the Idle_Stats values.

root@zz376p1:~# cpupower monitor
              |Idle_Stats
PKG |CORE|CPU | snoo | stop | stop
   0| 8| 0| 0.00| 0.00| 2.79
   0| 8| 1| 0.00| 0.00| 70.68
   0| 8| 2| 0.00| 0.00| 99.87
   0| 8| 3| 0.00| 0.00| 67.28
   0| 12| 4| 0.00| 0.00| 5.17
   0| 12| 5| 0.00| 0.00| 12.50
   0| 12| 6| 0.00| 0.00| 99.74
   0| 12| 7| 0.00| 0.00| 0.00
   8|2048| 8| 0.00| 0.00| 22.14
   8|2048| 9| 0.00| 0.00| 102.3
   8|2048| 10| 0.00| 0.00| 0.00
   8|2048| 11| 0.00| 0.00| 99.97
   8|2052| 12| 0.00| 0.00| 99.70
   8|2052| 13| 0.00| 0.00| 23.86
   8|2052| 14| 0.00| 0.00| 113.1
   8|2052| 15| 0.00| 0.00| 0.00

As can be seen it shows 2 columns for stop.

root@zz376p1:~# uname -a
Linux zz376p1 4.10.0-26-generic #30~16.04.1-Ubuntu SMP Tue Jun 27 09:38:48 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

root@zz376p1:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

root@zz376p1:~# cat /proc/cpuinfo | tail
processor : 15
cpu : POWER9 (raw), altivec supported
clock : 2600.000000MHz
revision : 1.0 (pvr 004e 0100)

timebase : 512000000
platform : PowerNV
model : 8375-42A
machine : PowerNV 8375-42A
firmware : OPAL

Userspace tool common name: /usr/bin/cpupower

The userspace tool has the following bit modes: 64-bit

Userspace rpm: linux-tools-common

Userspace tool obtained from project website: na

*Additional Instructions for <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.
-Attach ltrace and strace of userspace application.

.
There are 3 idle_stats (3 states) in this system.

root@zz376p1:/sys/devices/system/cpu/cpu0/cpuidle/state2# cpupower monitor -l
Monitor "Idle_Stats" (3 states) - Might overflow after 4294967295 s
snoo [T] -> snooze
stop [T] -> stop0_lite
stop [T] -> stop1_lite

root@zz376p1:/sys/devices/system/cpu/cpu0/cpuidle/state2# cpupower idle-info
CPUidle driver: powernv_idle
CPUidle governor: menu
analyzing CPU 0:

Number of idle states: 3
Available idle states: snooze stop0_lite stop1_lite
snooze:
Flags/Description: snooze
Latency: 0
Usage: 224
Duration: 423
stop0_lite:
Flags/Description: stop0_lite
Latency: 0
Usage: 1685
Duration: 530522
stop1_lite:
Flags/Description: stop1_lite
Latency: 4
Usage: 12693
Duration: 5405898106

root@zz376p1:/sys/devices/system/cpu/cpu0/cpuidle/state2# cpupower monitor -l
Monitor "Idle_Stats" (3 states) - Might overflow after 4294967295 s
snoo [T] -> snooze
stop [T] -> stop0_lite
stop [T] -> stop1_lite
root@zz376p1:/sys/devices/system/cpu/cpu0/cpuidle/state2# cpupower idle-info
CPUidle driver: powernv_idle
CPUidle governor: menu
analyzing CPU 0:

Number of idle states: 3
Available idle states: snooze stop0_lite stop1_lite
snooze:
Flags/Description: snooze
Latency: 0
Usage: 272
Duration: 905
stop0_lite:
Flags/Description: stop0_lite
Latency: 0
Usage: 2141
Duration: 536399
stop1_lite:
Flags/Description: stop1_lite
Latency: 4
Usage: 15396
Duration: 6625668881

cpu monitor will print the results of all the 3 available idle_stats .

The first stop -> stop0_lite stats
and
The second stop -> stop1_lite stats.

cpupower monitor header prints the header list by getting the name of the idle stats.

name (name description)
snoo [T] -> snooze
stop [T] -> stop0_lite
stop [T] -> stop1_lite

We can change the header to be more meaningful so user won't interpret it as 2 columns for stop.

May be we need to change the code to print the header to use name description instead of name.
.
I will work on the patch and get it resolve.

== Comment: #8 - PAVAMAN SUBRAMANIYAM <email address hidden> - 2017-06-30 03:42:01 ==
I have executed the command again with the fixed cpupower utility.

Submitted patch upstream and awaiting response.

https://lkml.org/lkml/2017/7/31/177

[Joseph Salisbury (jsalisbury)]

Can you post a comment in this bug report once the patch gets accepted upstream?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-09-28 04:19 EDT-------
Sure

[Andrew Cloke (andrew-cloke)]

Marking as "incomplete" until patch lands upstream and it becomes actionable.

[Frank Heimes (fheimes)]

@Pavaman: Was the patch in between upstream accepted?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-05-15 05:09 EDT-------
(In reply to comment #31)
> Seeteena, Canonical has asked for an update of this bug. Could you provide
> an update on the patch status (even if that status is "Awaiting patch
> response from the upstream maintainer.. still waiting.")? Thanks.

Still waiting for the response.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-08 14:33 EDT-------
Seeteena,

It looks like a patch to fix this was applied, right?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/tools/power/cpupower/utils/idle_monitor/cpupower-monitor.c?id=f9652d5cae04eb5e85303c087f5842d320499c65

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-16 04:51 EDT-------
Yes. Applied.
.

[Andrew Cloke (andrew-cloke)]

The patch described in comment #6 has landed upstream in 4.18, so the next step would be to backport to 4.15 Ubuntu kernel so that it was available in the 16.04.5 HWE kernel.

[Joseph Salisbury (jsalisbury)]

I built a test kernel with commit f9652d5cae0. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1719545

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
* If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
* If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

[Andrew Cloke (andrew-cloke)]

Marking incomplete while awaiting test results from IBM for Joe's test kernel.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-01-24 05:35 EDT-------
Can you place the test kernel again in the below folder to verify the fixes:
http://kernel.ubuntu.com/~jsalisbury/lp1719545

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-01-30 06:00 EDT-------
We don't have access to the above link. please provide a valid link

------- Comment From <email address hidden> 2019-01-30 06:02 EDT-------
says

The requested URL /~jsalisbury/lp1719545 was not found on this server.

[Khaled El Mously (kmously)]

A xenial-hwe kernel with this requested fix is now available for testing at:

https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-0830ceed0-KfV4/

(Note those are ppc64el binaries)

[Khaled El Mously (kmously)]

Assuming the fix in comment #13 is OK, it should be available in the Bionic general kernel (and its derivatives, include xenial-hwe) in the March 4th SRU release.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-02-12 04:28 EDT-------
I have downloaded the .deb packages and installed on the machine.

root@ltc-zz10:~# wget https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-0830ceed0-KfV4/linux-modules-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
--2019-02-12 03:07:39-- https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-0830ceed0-KfV4/linux-modules-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
Resolving kernel.ubuntu.com (kernel.ubuntu.com)... 91.189.94.216
Connecting to kernel.ubuntu.com (kernel.ubuntu.com)|91.189.94.216|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11400636 (11M) [application/x-debian-package]
Saving to: ?linux-modules-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb?

linux-modules-4.15.0-45-generic_4.15.0- 100%[============================================================================>] 10.87M 7.15MB/s in 1.5s

2019-02-12 03:07:42 (7.15 MB/s) - ?linux-modules-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb? saved [11400636/11400636]

root@ltc-zz10:~# wget https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-0830ceed0-KfV4/linux-modules-extra-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
--2019-02-12 03:07:55-- https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-0830ceed0-KfV4/linux-modules-extra-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
Resolving kernel.ubuntu.com (kernel.ubuntu.com)... 91.189.94.216
Connecting to kernel.ubuntu.com (kernel.ubuntu.com)|91.189.94.216|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30522156 (29M) [application/x-debian-package]
Saving to: ?linux-modules-extra-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb?

linux-modules-extra-4.15.0-45-generic_4 100%[============================================================================>] 29.11M 9.83MB/s in 3.0s

2019-02-12 03:07:59 (9.83 MB/s) - ?linux-modules-extra-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb? saved [30522156/30522156]

root@ltc-zz10:~# wget https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-0830ceed0-KfV4/linux-image-unsigned-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
--2019-02-12 03:08:07-- https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-0830ceed0-KfV4/linux-image-unsigned-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
Resolving kernel.ubuntu.com (kernel.ubuntu.com)... 91.189.94.216
Connecting to kernel.ubuntu.com (kernel.ubuntu.com)|91.189.94.216|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6010702 (5.7M) [application/x-debian-package]
Saving to: ?linux-image-unsigned-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb?

linux-image-unsigned-4.15.0-45-generic_ 100%[============================================================================>] 5.73M 3.79MB/s in 1.5s

2019-02-12 03:08:10 (3.79 MB/s) - ?linux-image-unsigned-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb? saved [6010702/6010702]

root@ltc-zz10:~# dpkg -i linux-modules-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
(Reading database ... 129185 files and directories currently installed.)
Preparing to unpack linux-modules-4.15.0-45-ge...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-02-12 04:30 EDT-------
Then executed the test scenario again as follows:

root@ltc-zz10:~# uname -a
Linux ltc-zz10 4.15.0-45-generic #48~16.04.1-Ubuntu SMP Tue Jan 29 18:03:36 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

root@ltc-zz10:~# dpkg --list | grep linux-image
ii linux-image-4.15.0-43-generic 4.15.0-43.46~16.04.1 ppc64el Signed kernel image generic
rc linux-image-4.15.0-45-generic 4.15.0-45.48~16.04.1 ppc64el Signed kernel image generic
ii linux-image-unsigned-4.15.0-45-generic 4.15.0-45.48~16.04.1 ppc64el Linux kernel image for version 4.15.0 on PowerPC 64el SMP

root@ltc-zz10:~# cpupower monitor
|Idle_Stats
PKG |CORE|CPU | snoo | stop | stop | stop | stop | stop | stop
0| 16| 0| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 16| 1| 0.00| 0.00| 0.00| 0.00| 4.49| 0.00| 105.7
0| 16| 2| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 16| 3| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 20| 4| 0.00| 0.00| 0.00| 0.00| 4.43| 0.00| 81.55
0| 20| 5| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 20| 6| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 20| 7| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 32| 8| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 67.30
0| 32| 9| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 32| 10| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 32| 11| 0.00| 0.00| 0.00| 0.00| 47.16| 0.00| 54.36
0| 36| 12| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 36| 13| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 36| 14| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 36| 15| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 40| 16| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 99.25
0| 40| 17| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 40| 18| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 40| 19| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 44| 20| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 44| 21| 0.00| 0.00| 0.00| 0.01| 45.84| 0.00| 53.01
0| 44| 22| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 44| 23| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 48| 24| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 48| 25| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 48| 26| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 48| 27| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 52| 28| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 51.25
0| 52| 29| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 52| 30| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 52| 31| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 56| 32| 0.00| 0.00| 0.00| 0.00| 1.98| 0.00| 0.00
0| 56| 33| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 56| 34| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 56| 35| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 60| 36| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 60| 37| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00
0| 60| 38| 0.00| 0.00| 0.00| 0.00| 2.04|...

[Khaled El Mously (kmously)]

@PAVAMAN: Is there a reason that you installed the modules and modules-extra packages using 'dpkg -i' and then tried to install the kernel using 'apt-get install'? It doesn't look like you installed the right kernel there.
I think the third command should have been 'dpkg -i linux-image-unsigned-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb'

Please try to install the kernel that way and retry the test. Thanks

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-02-15 01:11 EDT-------
I have installed all the packages as mentioned.

root@ltc-zz10:~# dpkg -i linux-modules-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
(Reading database ... 129194 files and directories currently installed.)
Preparing to unpack linux-modules-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb ...
Unpacking linux-modules-4.15.0-45-generic (4.15.0-45.48~16.04.1) over (4.15.0-45.48~16.04.1) ...
Setting up linux-modules-4.15.0-45-generic (4.15.0-45.48~16.04.1) ...
Processing triggers for linux-image-unsigned-4.15.0-45-generic (4.15.0-45.48~16.04.1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.15.0-45-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
/etc/kernel/postinst.d/x-grub-legacy-ec2:
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Updating /boot/grub/menu.lst ... done

/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found linux image: /boot/vmlinux-4.15.0-45-generic
Found initrd image: /boot/initrd.img-4.15.0-45-generic
Found linux image: /boot/vmlinux-4.15.0-43-generic
Found initrd image: /boot/initrd.img-4.15.0-43-generic
done

root@ltc-zz10:~# dpkg -i linux-modules-extra-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
(Reading database ... 129185 files and directories currently installed.)
Preparing to unpack linux-modules-extra-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb ...
Unpacking linux-modules-extra-4.15.0-45-generic (4.15.0-45.48~16.04.1) over (4.15.0-45.48~16.04.1) ...
Setting up linux-modules-extra-4.15.0-45-generic (4.15.0-45.48~16.04.1) ...
Processing triggers for linux-image-unsigned-4.15.0-45-generic (4.15.0-45.48~16.04.1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.15.0-45-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
/etc/kernel/postinst.d/x-grub-legacy-ec2:
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Updating /boot/grub/menu.lst ... done

/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found linux image: /boot/vmlinux-4.15.0-45-generic
Found initrd image: /boot/initrd.img-4.15.0-45-generic
Found linux image: /boot/vmlinux-4.15.0-43-generic
Found initrd image: /boot/initrd.img-4.15.0-43-generic
done

root@ltc-zz10:~# dpkg -i linux-image-unsigned-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb
(Reading database ... 129175 files and directories currently installed.)
Preparing to unpack linux-image-unsigned-4.15.0-45-generic_4.15.0-45.48~16.04.1_ppc64el.deb ...
Unpacking linux-image-unsigned-4.15.0-45-generic (4.15.0-45.48~16.04.1) over (4.15.0-45.48~16.04.1) ...
Setting up linux-image-unsigned-4.15.0-45-generic (4.15.0-45.48~16.04.1) ...
Processing triggers for lin...

[Manoj Iyer (manjo)]

<email address hidden>, thanks for posting your test results to the bug. Your report is missing any confirmation that the bug is in fact fixed by the test kernel. Could you please provide some additional comments here stating if this kernel fixes the issue for you or not ?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-06 23:35 EDT-------
The kernel does not fix the original issue and the issue is still occurring.

[Khaled El Mously (kmously)]

@Pavaman, thanks for testing.

The patch that was submitted upstream by IBM to fix this bug ( f9652d5cae04eb5e85303c087f5842d320499c65 ) was applied (cleanly) to bionic to generate that kernel. It's not clear to me why that doesn't fix your problem since it was identified as the only required fix for this issue.

Could you please try this kernel: https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/

and could you please also show the output of uname -a as well ?

Thanks

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-08 01:45 EDT-------
I have downloaded the .deb packages and installed on the machine.

root@ltc-zz10:~# mkdir kernel-kmously
root@ltc-zz10:~# cd kernel-kmously/
root@ltc-zz10:~/kernel-kmously# wget https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/linux-modules-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb
--2019-03-08 00:27:57-- https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/linux-modules-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb
Resolving kernel.ubuntu.com (kernel.ubuntu.com)... 91.189.94.216
Connecting to kernel.ubuntu.com (kernel.ubuntu.com)|91.189.94.216|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11780892 (11M) [application/x-debian-package]
Saving to: ?linux-modules-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb?

linux-modules-4.15.0-46-generic_4.15.0-46.49~k1_ppc 100%[==================================================================================================================>] 11.23M 7.25MB/s in 1.5s

2019-03-08 00:27:59 (7.25 MB/s) - ?linux-modules-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb? saved [11780892/11780892]

root@ltc-zz10:~/kernel-kmously# wget https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/linux-modules-extra-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb
--2019-03-08 00:28:09-- https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/linux-modules-extra-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb
Resolving kernel.ubuntu.com (kernel.ubuntu.com)... 91.189.94.216
Connecting to kernel.ubuntu.com (kernel.ubuntu.com)|91.189.94.216|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 31457736 (30M) [application/x-debian-package]
Saving to: ?linux-modules-extra-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb?

linux-modules-extra-4.15.0-46-generic_4.15.0-46.49~ 100%[==================================================================================================================>] 30.00M 7.60MB/s in 4.9s

2019-03-08 00:28:14 (6.13 MB/s) - ?linux-modules-extra-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb? saved [31457736/31457736]

root@ltc-zz10:~/kernel-kmously# wget https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/linux-image-unsigned-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb
--2019-03-08 00:28:36-- https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/linux-image-unsigned-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb
Resolving kernel.ubuntu.com (kernel.ubuntu.com)... 91.189.94.216
Connecting to kernel.ubuntu.com (kernel.ubuntu.com)|91.189.94.216|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6249220 (6.0M) [application/x-debian-package]
Saving to: ?linux-image-unsigned-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb?

linux-image-unsigned-4.15.0-46-generic_4.15.0-46.49 100%[==================================================================================================================>] 5.96M 4.35MB/s in 1.4s

2019-03-08 00:28:38 (4.35 MB/s) - ?linux-image-unsigned-4.15.0-46-generic_4.15.0-46.49~k1_ppc64el.deb? saved [6249220/6249220]

root@ltc-zz1...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-15 00:32 EDT-------
Canonical , Any update on the previous test we did..

[Khaled El Mously (kmously)]

@<email address hidden>: Thanks for that test. Unfortunately I'm not quite sure why the patch isn't fixing the problem. This was the patch that IBM submitted upstream and reported that it fixes the problem. The patch applied cleanly to bionic. I'm still investigating this further and may require your help in testing it again if I can't reproduce the issue myself. Will update again shortly.
Thanks.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-26 03:53 EDT-------
Thanks

[Khaled El Mously (kmously)]

@<email address hidden>: Hello. Now I have a theory as to what the problem might be.

The upstream fix is actually a change only to the 'cpupower' tool, not the kernel itself. This tool is not part of the linux-image or linux-modules .deb files. It is built as part of the linux-tools .deb, which for some reason, wasn't included in the test build that I prepared for you.

I've updated the above link by adding the file: linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb

This .deb includes the updated cpupower binary. The full path after installation should be: /usr/lib/linux-tools-4.15.0-46/cpupower

Please install that .deb and use the included 'cpupower' binary when testing.
Thank you.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-27 03:58 EDT-------
I have downloaded the linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb package from the link:
https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/

Then installed the package and executed the test scenario again.

root@ltc-zz10:~# cd kernel-kmously/
root@ltc-zz10:~/kernel-kmously# wget https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb
--2019-03-27 01:27:08-- https://kernel.ubuntu.com/~kmously/1719545/kernel-kmously-438fd5a-TfNM/linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb
Resolving kernel.ubuntu.com (kernel.ubuntu.com)... 91.189.94.216
Connecting to kernel.ubuntu.com (kernel.ubuntu.com)|91.189.94.216|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4246392 (4.0M) [application/x-debian-package]
Saving to: ?linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb?

linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb 100%[==================================================================================================================>] 4.05M 3.69MB/s in 1.1s

2019-03-27 01:27:10 (3.69 MB/s) - ?linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb? saved [4246392/4246392]

root@ltc-zz10:~/kernel-kmously# ls -l linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb
-rw-r--r-- 1 root root 4246392 Mar 27 00:05 linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb

root@ltc-zz10:~/kernel-kmously# dpkg -i linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb
(Reading database ... 163245 files and directories currently installed.)
Preparing to unpack linux-tools-4.15.0-46_4.15.0-46.49~k1_ppc64el.deb ...
Unpacking linux-tools-4.15.0-46 (4.15.0-46.49~k1) ...
Setting up linux-tools-4.15.0-46 (4.15.0-46.49~k1) ...

root@ltc-zz10:~/kernel-kmously# ls -l /usr/lib/linux-tools-4.15.0-46/cpupower
-rwxr-xr-x 1 root root 233344 Mar 26 01:07 /usr/lib/linux-tools-4.15.0-46/cpupower
root@ltc-zz10:~/kernel-kmously# which cpupower
/usr/bin/cpupower
root@ltc-zz10:~/kernel-kmously# dpkg -S /usr/lib/linux-tools-4.15.0-46/cpupower
linux-tools-4.15.0-46: /usr/lib/linux-tools-4.15.0-46/cpupower
root@ltc-zz10:~/kernel-kmously# dpkg --list | grep linux-tools
ii linux-tools-4.15.0-46 4.15.0-46.49~k1 ppc64el Linux kernel version specific tools for version 4.15.0-46
ii linux-tools-common 4.4.0-143.169 all Linux kernel version specific tools for version 4.4.0

root@ltc-zz10:~/kernel-kmously# /usr/lib/linux-tools-4.15.0-46/cpupower idle-info
CPUidle driver: powernv_idle
CPUidle governor: menu
analyzing CPU 0:

Number of idle states: 7
Available idle states: snooze stop0_lite stop0 stop1 stop2 stop4 stop5
snooze:
Flags/Description: snooze
Latency: 0
Usage: 4323
Duration: 86520
stop0_lite:
Flags/Description: stop0_lite
Latency: 1
Usage: 3273
Duration: 10611854
stop0:
Flags/Description: stop0
Latency: 2
Usage: 5022
Duration: 5492700
stop1:
Flags/Description: stop1
Latency: 5
Usage: 3622
Duration: 286540
stop2:
Flags/Description: stop2
Latency: 10
Usage: 468308
Duration: 34729622421
stop4:
Flags/Description...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-48.51

---------------
linux (4.15.0-48.51) bionic; urgency=medium

  * linux: 4.15.0-48.51 -proposed tracker (LP: #1822820)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
    - [Packaging] resync retpoline extraction

  * 3b080b2564287be91605bfd1d5ee985696e61d3c in ubuntu_btrfs_kernel_fixes
    triggers system hang on i386 (LP: #1812845)
    - btrfs: raid56: properly unmap parity page in finish_parity_scrub()

  * [P9][LTCTest][Opal][FW910] cpupower monitor shows multiple stop Idle_Stats
    (LP: #1719545)
    - cpupower : Fix header name to read idle state name

  * [amdgpu] screen corruption when using touchpad (LP: #1818617)
    - drm/amdgpu/gmc: steal the appropriate amount of vram for fw hand-over (v3)
    - drm/amdgpu: Free VGA stolen memory as soon as possible.

  * [SRU][B/C/OEM]IOMMU: add kernel dma protection (LP: #1820153)
    - ACPICA: AML parser: attempt to continue loading table after error
    - ACPI / property: Allow multiple property compatible _DSD entries
    - PCI / ACPI: Identify untrusted PCI devices
    - iommu/vt-d: Force IOMMU on for platform opt in hint
    - iommu/vt-d: Do not enable ATS for untrusted devices
    - thunderbolt: Export IOMMU based DMA protection support to userspace
    - iommu/vt-d: Disable ATS support on untrusted devices

  * Add basic support to NVLink2 passthrough (LP: #1819989)
    - powerpc/powernv/npu: Do not try invalidating 32bit table when 64bit table is
      enabled
    - powerpc/powernv: call OPAL_QUIESCE before OPAL_SIGNAL_SYSTEM_RESET
    - powerpc/powernv: Export opal_check_token symbol
    - powerpc/powernv: Make possible for user to force a full ipl cec reboot
    - powerpc/powernv/idoa: Remove unnecessary pcidev from pci_dn
    - powerpc/powernv: Move npu struct from pnv_phb to pci_controller
    - powerpc/powernv/npu: Move OPAL calls away from context manipulation
    - powerpc/pseries/iommu: Use memory@ nodes in max RAM address calculation
    - powerpc/pseries/npu: Enable platform support
    - powerpc/pseries: Remove IOMMU API support for non-LPAR systems
    - powerpc/powernv/npu: Check mmio_atsd array bounds when populating
    - powerpc/powernv/npu: Fault user page into the hypervisor's pagetable

  * Huawei Hi1822 NIC has poor performance (LP: #1820187)
    - net-next: hinic: fix a problem in free_tx_poll()
    - hinic: remove ndo_poll_controller
    - net-next/hinic: add checksum offload and TSO support
    - hinic: Fix l4_type parameter in hinic_task_set_tunnel_l4
    - net-next/hinic:replace multiply and division operators
    - net-next/hinic:add rx checksum offload for HiNIC
    - net-next/hinic:fix a bug in set mac address
    - net-next/hinic: fix a bug in rx data flow
    - net: hinic: fix null pointer dereference on pointer hwdev
    - hinic: optmize rx refill buffer mechanism
    - net-next/hinic:add shutdown callback
    - net-next/hinic: replace disable_irq_nosync/enable_irq

  * [CONFIG] please enable highdpi font FONT_TER16x32 (LP: #1819881)
    - Fonts: New Terminus large console font
    - [Config]: enable highdpi Terminus 16x32 font support

  * [19.04 FEAT] qeth: Enhanced link...