Linux perf regression in 4.4.0-42 and newer

[Chris Phlipot]

There is a regression in the 4.4.0-42 kernel update that is still present in the latest 4.4.0 update. I have verified that this appears in Ubuntu 16.04 and Ubuntu 14.04 (with ubuntu-xenial kernel installed) amd64. I also suspect this happens in default Ubuntu 16.10 kernel as well, but i have not tested.

4.4.0-38 did not have this issue.

in 4.4.0-42 and newer, perf records incorrect timestamps which can be verified by running this command:

$ perf record -a -e sched:sched_-k CLOCK_MONOTONIC switch stress -c 10000 -t 10
stress: info: [25560] dispatching hogs: 10000 cpu, 0 io, 0 vm, 0 hdd
stress: info: [25560] successful run completed in 6s
[ perf record: Woken up 16 times to write data ]
Warning:
78487 out of order events recorded.

Note that the warning about out of order events is a symptom of this problem and is not expected to happen. This does not happen(correct behavior) if i run 4.4.0-38 and older, or if i run a fully patched upstream 4.4.35 kernel instead.

This regression has already been fixed upstream but has not been included in ubuntu-xenial kernel yet.

Upstream fix is here:
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=78c7b55b362e868e529ab6579134708fcf5539dd

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1645449

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Chris Phlipot (cphlipot0)]

I have already diagnosed the problem (see above), and I don't have access to the system.

Changing to confirmed.

[Joseph Salisbury (jsalisbury)]

It appears the commit you mention has landed in upstream stable 4.4.25 as commit:

78c7b55 timekeeping: Fix __ktime_get_fast_ns() regression

The upstream stable commits up to version 4.4.30 have landed in Xenial -proposed.

Would it be possible for you to test this latest kernel and post back if it resolves this bug?
See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed.

Thank you in advance!

[Chris (teefs28846)]

The recent 4.4.0-53 update appears to fix this issue for me.