perf broken on hwe kernel 5.0.0-23

[Philippe Rigault]

After enabling the hwe stack on ubuntu 18.04, perf no longer reports events

$ perf --version
perf version 5.0.15

$ uname -a
Linux ripper 5.0.0-23-generic #24~18.04.1-Ubuntu SMP Mon Jul 29 16:12:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Results: perf does not report events
Output of `perf top -p <PID>` looks like this:

Samples: 0 of event 'cycles', 4000 Hz, Event count (approx.): 0 lost: 0/0 drop: 0/0
Overhead

Expected results: events reported

Additional info:
 - HWE stack was updated the standard way:
   sudo apt install --install-recommends linux-generic-hwe-18.04

 - perf worked fine on linux kernel 4.15.0-55-generic (non-hwe)

 - the bug reproduces on the following architectures
     AMD Threadripper 1950X
     AMD Opteron 6376
     Intel Core i7-8650U

[Sebastien Boisvert (sebhtml)]

With Linux and perf 4.15.0-55-generic (non-HWE), it works as reported in Ubuntu 18.04.3 LTS:

Versions
--------

$ uname -r
4.15.0-55-generic

$ perf --version
perf version 4.15.18

In dpkg:

ii linux-image-4.15.0-55-generic 4.15.0-55.60 amd64 Signed kernel image generic

ii linux-tools-common 4.15.0-55.60 all Linux kernel version specific tools for version 4.15.0

ii linux-tools-4.15.0-55-generic 4.15.0-55.60 amd64 Linux kernel version specific tools for version 4.15.0-55

A minimal series of steps necessary to make it happen
-----------------------------------------------------

1. In a terminal tab., run:
    $ top

2. In another tab, run
    $ perf top -p `pidof top`

3. Look at the output of "perf top".

What you expected to happen
---------------------------

I expect to see "cycles" events being reported for the process "top" (started at step 1.)

What actually happened
----------------------

I see this, so it works:

Samples: 136 of event 'cycles:ppp', Event count (approx.): 54426558
Overhead Shared Object Symbol
   8.48% libc-2.27.so [.] _IO_vfscanf
   6.49% [kernel] [k] do_syscall_64
   4.06% [kernel] [k] pid_revalidate
   3.50% [kernel] [k] syscall_return_via_sysret
   3.05% [kernel] [k] vsnprintf
   2.90% [kernel] [k] get_pid_task
   2.88% [kernel] [k] seq_put_decimal_ull
   2.75% [kernel] [k] __task_pid_nr_ns
   2.74% [kernel] [k] seq_put_decimal_ll
   2.71% libc-2.27.so [.] __GI_____strtoull_l_internal
   2.41% [kernel] [k] format_decode
   2.38% [kernel] [k] memcpy_erms
   2.18% libc-2.27.so [.] vfprintf
   1.93% [kernel] [k] pid_nr_ns
   1.79% [kernel] [k] __d_lookup_rcu
   1.63% [kernel] [k] rcu_all_qs
   1.58% [kernel] [k] filldir
   1.39% [kernel] [k] render_sigset_t

System calls
------------

perf_event_open System calls:

$ strace perf top -p `pidof top` 2>&1 | grep ^perf_event_open
perf_event_open({type=PERF_TYPE_HARDWARE, size=PERF_ATTR_SIZE_VER5, config=PERF_COUNT_HW_CPU_CYCLES, ...}, 0, -1, -1, 0) = 3
perf_event_open({type=PERF_TYPE_SOFTWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=PERF_COUNT_SW_CPU_CLOCK, ...}, -1, 6, -1, PERF_FLAG_FD_CLOEXEC) = 3
perf_event_open({type=PERF_TYPE_HARDWARE, size=PERF_ATTR_SIZE_VER5, config=PERF_COUNT_HW_CPU_CYCLES, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = 3
perf_event_open({type=PERF_TYPE_HARDWARE, size=PERF_ATTR_SIZE_VER5, config=PERF_COUNT_HW_CPU_CYCLES, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = 3
perf_event_open({type=PERF_TYPE_HARDWARE, size=PERF_ATTR_SIZE_VER5, config=PERF_COUNT_HW_CPU_CYCLES, ...}, 12089, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3

[Sebastien Boisvert (sebhtml)]

With Linux and perf 5.0.0-23-generic (HWE or Hardware Enablement), it is broken.

Versions
--------

$ uname -r
5.0.0-23-generic
$ perf --version
perf version 5.0.15

$ dpkg -l|grep 5.0.0-23|awk '{print $1,$2,$3,$4}'|column -t
ii linux-headers-5.0.0-23 5.0.0-23.24~18.04.1 all
ii linux-headers-5.0.0-23-generic 5.0.0-23.24~18.04.1 amd64
ii linux-hwe-tools-5.0.0-23 5.0.0-23.24~18.04.1 amd64
ii linux-image-5.0.0-23-generic 5.0.0-23.24~18.04.1 amd64
ii linux-modules-5.0.0-23-generic 5.0.0-23.24~18.04.1 amd64
ii linux-modules-extra-5.0.0-23-generic 5.0.0-23.24~18.04.1 amd64
ii linux-tools-5.0.0-23-generic 5.0.0-23.24~18.04.1 amd64

A minimal series of steps necessary to make it happen
-----------------------------------------------------

1. In a terminal tab., run:
    $ top

2. In another tab, run
    $ perf top -p `pidof top`

3. Look at the output of "perf top".

What you expected to happen
---------------------------

I expect to see "cycles" events being reported for the process "top" (started at step 1.)

See section "What actually happened" in comment #1 above.

What actually happened
----------------------

The output of "perf top -p `pidof top`" (step 2) is:

Samples: 0 of event 'cycles:ppp', 4000 Hz, Event count (approx.): 0 lost: 0/0 drop: 0/0
Overhead

It is as if there are no events being sampled.

System calls
------------

perf_event_open System calls:

$ strace perf top -p `pidof top` 2>&1 | grep ^perf_event_open
perf_event_open({type=PERF_TYPE_HARDWARE, size=PERF_ATTR_SIZE_VER5, config=PERF_COUNT_HW_CPU_CYCLES, ...}, 0, -1, -1, 0) = 3
perf_event_open({type=PERF_TYPE_SOFTWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=PERF_COUNT_SW_CPU_CLOCK, ...}, -1, 2, -1, PERF_FLAG_FD_CLOEXEC) = 3
perf_event_open({type=PERF_TYPE_HARDWARE, size=PERF_ATTR_SIZE_VER5, config=PERF_COUNT_HW_CPU_CYCLES, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = 3
perf_event_open({type=PERF_TYPE_HARDWARE, size=PERF_ATTR_SIZE_VER5, config=PERF_COUNT_HW_CPU_CYCLES, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = 3
perf_event_open({type=PERF_TYPE_HARDWARE, size=PERF_ATTR_SIZE_VER5, config=PERF_COUNT_HW_CPU_CYCLES, ...}, 4035, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3

Additional note
---------------

Using the full path to perf 5.0.0-23 also produce 0 events.

$ /usr/lib/linux-tools/5.0.0-23-generic/perf top -p `pidof top`

[Sebastien Boisvert (sebhtml)]

After some investigation, I found this:

If I use the perf executable from the package linux-tools-4.15.0-55-generic
along with the Linux kernel 5.0.0-23-generic from the package
linux-image-5.0.0-23-generic , it works.

$ /usr/lib/linux-tools/4.15.0-55-generic/perf top -p `pidof top`

Samples: 134 of event 'cycles:ppp', Event count (approx.): 26218815
Overhead Shared Object Symbol
   6.69% libc-2.27.so [.] _IO_vfscanf
   4.01% [kernel] [k] entry_SYSCALL_64
   3.71% [kernel] [k] do_syscall_64
   3.36% [kernel] [k] num_to_str
   3.25% libc-2.27.so [.] __GI_____strtoll_l_internal
   2.54% [kernel] [k] syscall_return_via_sysret
   2.35% [kernel] [k] __d_lookup
   2.24% [kernel] [k] get_mem_cgroup_from_mm
   2.22% libc-2.27.so [.] _IO_default_xsputn
   1.84% libc-2.27.so [.] _IO_sputbackc
   1.80% [kernel] [k] kmem_cache_free
   1.77% [kernel] [k] bitmap_string.isra.2
   1.72% [kernel] [k] render_sigset_t
   1.65% [kernel] [k] __alloc_file
   1.62% [kernel] [k] do_task_stat
   1.57% [kernel] [k] number

So, the problem is really the executable /usr/lib/linux-hwe-tools-5.0.0-23/perf
which is in the package:

$ apt-file search /usr/lib/linux-hwe-tools-5.0.0-23/perf
linux-hwe-tools-5.0.0-23: /usr/lib/linux-hwe-tools-5.0.0-23/perf

$ dpkg -l|grep linux-hwe-tools-5.0.0-23|awk '{print $1, $2, $3, $4}'|column -t
ii linux-hwe-tools-5.0.0-23 5.0.0-23.24~18.04.1 amd64

According to https://packages.ubuntu.com/bionic-updates/linux-hwe-cloud-tools-5.0.0-23 ,
the source package for linux-hwe-tools is linux-hwe.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Philippe Rigault (prigault)]

The problem is fixed with perf in linux-hwe-tools-5.0.0-25