[Potential Regression] System crashes when running ftrace test in ubuntu_kernel_selftests

[Po-Hsu Lin]

[Impact]
ftrace test in ubuntu_kernel_selftests causes a system crash + reboot when testing "Register/unregister many kprobe events".

[Fix]
The issue has been discussed in the following LKML thread:
https://lkml.org/lkml/2019/6/5/274

and the following 3 upstream commits have been identified to resolve this issue:

d2a68c4effd8 x86/ftrace: Do not call function graph from dynamic trampolines
3c0dab44e227 x86/ftrace: Set trampoline pages as executable
7298e24f9042 x86/kprobes: Set instruction page as executable

From these commits, the Disco kernel currently in -proposed (5.0.0-26.27) is missing only the following commit, which has been already committed as part of LP: #1839887 (Disco update: upstream stable patchset 2019-08-12) but not yet released:

7298e24f9042 x86/kprobes: Set instruction page as executable

[Regression potential]
The commit touches the x86/kprobes code so there's a chance of regression there which I would flag as medium. However, it has been applied upstream for v5.2-rc1 and there is no follow-up commits marked as fix for it.

----------------------------------------------
This issue is 100% reproducible. It looks like this has something to do with the Disco kernel in proposed (5.0.0-1014.14).

Test combinations:
kernel in updates + source code master branch = OK
kernel in proposed + source code master-next branch = NOT OK
kernel in proposed + source code master branch = NOT OK

The system will crash and reboot itself when testing:
"Register/unregister many kprobe events"

[33] Kprobe event with comm arguments [PASS]
[34] Kprobe event string type argument [PASS]
[35] Kprobe event symbol argument [PASS]
[36] Kprobe event argument syntax [PASS]
[37] Kprobes event arguments with types [PASS]
[38] Kprobe event auto/manual naming [PASS]
[39] Kprobe dynamic event with function tracer [PASS]
[40] Kretprobe dynamic event with arguments [PASS]
[41] Kretprobe dynamic event with maxactive [PASS]
[42] Register/unregister many kprobe events
packet_write_wait: Connection to 35.233.208.253 port 22: Broken pipe

Tried with use "tail -f /var/log/syslog" and "dmesg -w" to see what happened, but it will just disconnected and reboot itself.

ProblemType: Bug
DistroRelease: Ubuntu 19.04
Package: linux-image-5.0.0-1014-gcp 5.0.0-1014.14
ProcVersionSignature: Ubuntu 5.0.0-1014.14-gcp 5.0.21
Uname: Linux 5.0.0-1014-gcp x86_64
ApportVersion: 2.20.10-0ubuntu27.1
Architecture: amd64
Date: Tue Aug 20 08:13:13 2019
SourcePackage: linux-signed-gcp
UpgradeStatus: No upgrade log present (probably fresh install)

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1840750

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Kleber Sacilotto de Souza (kleber-souza)]

I was able to get some kernel messages from a qemu console:

[ 208.672687] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 208.673786] BUG: unable to handle kernel paging request at ffffffffc054f694
[ 208.674779] #PF error: [PROT] [INSTR]
[ 208.675571] PGD 45a12067 P4D 45a12067 PUD 45a14067 PMD 79808067 PTE 80000000722f8061
[ 208.679132] Oops: 0011 [#1] SMP PTI
[ 208.680526] CPU: 0 PID: 3836 Comm: ftracetest Not tainted 5.0.0-26-generic #27-Ubuntu
[ 208.683102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 208.686150] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 208.686150] BUG: unable to handle kernel paging request at ffffffffc054feae
[ 208.686151] #PF error: [PROT] [INSTR]
[ 208.686151] PGD 45a12067 P4D 45a12067 PUD 45a14067 PMD 79808067 PTE 80000000722f8061
[ 208.686152] Oops: 0011 [#2] SMP PTI
[ 208.686153] CPU: 0 PID: 3836 Comm: ftracetest Not tainted 5.0.0-26-generic #27-Ubuntu
[ 208.686153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 208.686154] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 208.686154] BUG: unable to handle kernel paging request at ffffffffc054feae
[ 208.686154] #PF error: [PROT] [INSTR]
[ 208.686154] PGD 45a12067 P4D 45a12067 PUD 45a14067 PMD 79808067 PTE 80000000722f8061
[ 208.686156] Oops: 0011 [#3] SMP PTI
[ 208.686156] CPU: 0 PID: 3836 Comm: ftracetest Not tainted 5.0.0-26-generic #27-Ubuntu
[ 208.686156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 208.686157] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 208.686168] BUG: unable to handle kernel paging request at ffffffffc054feae
[ 208.686169] #PF error: [PROT] [INSTR]
[ 208.686169] PGD 45a12067 P4D 45a12067 PUD 45a14067 PMD 79808067 PTE 80000000722f8061
[ 208.686170] Oops: 0011 [#4] SMP PTI
[ 208.686170] CPU: 0 PID: 3836 Comm: ftracetest Not tainted 5.0.0-26-generic #27-Ubuntu
[ 208.686170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 208.686171] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 208.686171] BUG: unable to handle kernel paging request at ffffffffc054feae
[ 208.686171] #PF error: [PROT] [INSTR]
[ 208.686171] PGD 45a12067 P4D 45a12067 PUD 45a14067 PMD 79808067 PTE 80000000722f8061
[ 208.686172] Oops: 0011 [#5] SMP PTI
[ 208.686173] CPU: 0 PID: 3836 Comm: ftracetest Not tainted 5.0.0-26-generic #27-Ubuntu
[ 208.686173] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 208.686174] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 208.686174] BUG: unable to handle kernel paging request at ffffffffc054feae
[ 208.686174] #PF error: [PROT] [INSTR]
[ 208.686174] PGD 45a12067 P4D 45a12067 PUD 45a14067 PMD 79808067 PTE 80000000722f8061
[ 208.686176] Oops: 0011 [#6] SMP PTI
[ 208.686176] CPU: 0 PID: 3836 Comm: ftracetest Not tainted 5.0.0-26-generic #27-Ubuntu
[ 208.686176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[...

[Po-Hsu Lin (cypressyew)]

Upstream discussion
https://lkml.org/lkml/2019/6/9/781

[Kleber Sacilotto de Souza (kleber-souza)]

Running tests with the mainline builds the conclusion is that the bug was introduced with stable v5.0.20. From the thread on comment #4, we are missing the following fix:

7298e24f9042 x86/kprobes: Set instruction page as executable

I'm building a test kernel to verify it.

[Kleber Sacilotto de Souza (kleber-souza)]

Eoan already has the fixing commit, so the development series shouldn't be affected.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-disco' to 'verification-done-disco'. If the problem still exists, change the tag 'verification-needed-disco' to 'verification-failed-disco'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Kleber Sacilotto de Souza (kleber-souza)]

ubuntu_kernel_selftests.ftrace completed successfully with kernel 5.0.0-27.28.

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco/disco/amd64/l/linux/20190821_124751_453e6@/log.gz

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.0.0-27.28

---------------
linux (5.0.0-27.28) disco; urgency=medium

  * disco/linux: 5.0.0-27.28 -proposed tracker (LP: #1840816)

  * [Potential Regression] System crashes when running ftrace test in
    ubuntu_kernel_selftests (LP: #1840750)
    - x86/kprobes: Set instruction page as executable

linux (5.0.0-26.27) disco; urgency=medium

  * disco/linux: 5.0.0-26.27 -proposed tracker (LP: #1839972)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  * alsa/hdmi: add icelake hdmi audio support for a Dell machine (LP: #1836916)
    - ALSA: hda: hdmi - add Icelake support
    - ALSA: hda/hdmi - Remove duplicated define
    - ALSA: hda/hdmi - Fix i915 reverse port/pin mapping

  * input/mouse: alps trackpoint-only device doesn't work (LP: #1836752)
    - Input: alps - don't handle ALPS cs19 trackpoint-only device
    - Input: alps - fix a mismatch between a condition check and its comment

  * [18.04 FEAT] Enhanced hardware support (LP: #1836857)
    - s390: report new CPU capabilities
    - s390: add alignment hints to vector load and store

  * System does not auto detect disconnection of external monitor (LP: #1835001)
    - drm/i915: Add support for retrying hotplug
    - drm/i915: Enable hotplug retry

  * [18.04 FEAT] Enhanced CPU-MF hardware counters - kernel part (LP: #1836860)
    - s390/cpum_cf: Add support for CPU-MF SVN 6
    - s390/cpumf: Add extended counter set definitions for model 8561 and 8562

  * EeePC 1005px laptop backlight is off after system boot up (LP: #1837117)
    - platform/x86: asus-wmi: Only Tell EC the OS will handle display hotkeys from
      asus_nb_wmi

  * br_netfilter: namespace sysctl operations (LP: #1836910)
    - netfilter: bridge: port sysctls to use brnf_net
    - netfilter: bridge: namespace bridge netfilter sysctls
    - netfilter: bridge: prevent UAF in brnf_exit_net()

  * ideapad_laptop disables WiFi/BT radios on Lenovo Y540 (LP: #1837136)
    - platform/x86: ideapad-laptop: Remove no_hw_rfkill_list

  * shiftfs: allow overlayfs (LP: #1838677)
    - SAUCE: shiftfs: enable overlayfs on shiftfs

  * bcache: bch_allocator_thread(): hung task timeout (LP: #1784665)
    - bcache: never writeback a discard operation
    - bcache: improve bcache_reboot()
    - SAUCE: bcache: fix deadlock in bcache_allocator

  * Regressions in CMA allocation rework (LP: #1839395)
    - dma-contiguous: do not overwrite align in dma_alloc_contiguous()
    - dma-contiguous: page-align the size in dma_free_contiguous()

  * CVE-2019-3900
    - vhost: introduce vhost_exceeds_weight()
    - vhost_net: fix possible infinite loop
    - vhost: vsock: add weight support
    - vhost: scsi: add weight support

  * Disco update: 5.0.21 upstream stable release (LP: #1837518)
    - bonding/802.3ad: fix slave link initialization transition states
    - cxgb4: offload VLAN flows regardless of VLAN ethtype
    - inet: switch IP ID generator to siphash
    - ipv4/igmp: fix another memory leak in igmpv3_del_delrec()
    - ipv4/igmp: fix build error if !CONFIG_IP_MULTICAST
    - ipv6: Consider sk_bound_dev_if when binding a raw socket to an address
    - ipv6: Fix redi...