not able to unwind the stack from within __kernel_clock_gettime in the Linux vDSO

[bugproxy]

== SRU Justification ==
IBM is requesting this commit in Bionic and Cosmic. They report that they
are not able to unwind the stack from within __kernel_clock_gettime in the
Linux vDSO on Summit. This affects both DDT and MAP (via GDB and libunwind).
The issue is more serious than may first appear as the function appears to
be called somewhat often by the CUDA runtime, and can defer to a syscall
making it relatively time consuming (making it more likely to be encountered).

This commit is currently still in linux-next.

== Fix ==
56d20861c027 ("powerpc/vdso: Correct call frame information") linux-next

== Regression Potential ==
Low. Limited to powerpc.

Original Bug Description

We're not able to unwind the stack from within __kernel_clock_gettime in the Linux vDSO on Summit. This affects both DDT and MAP (via GDB and libunwind). The issue is more serious than may first appear as the function appears to be called somewhat often by the CUDA runtime, and can defer to a syscall making it relatively time consuming (making it more likely to be encountered).

To reproduce:

Compile $CUDA_DIR/samples/0_Simple/matrixMul (attached is a small patch to modify the Makefile to compile outside of the samples directory)

Run the following GDB commands:
user@deb3qwsp1:/usr/local/cuda-10.0/samples/0_Simple/matrixMul$ gdb ./matrixMul
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./matrixMul...(no debugging symbols found)...done.
(gdb) break main
Breakpoint 1 at 0x8284
(gdb) run
Starting program: /usr/local/cuda-10.0/samples/0_Simple/matrixMul/matrixMul
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Breakpoint 1, 0x0000000100008284 in main ()
(gdb) break *(__kernel_clock_gettime+144)
Breakpoint 2 at 0x7ffff7f805e4: file /build/linux-ZIBxfV/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S, line 127.
(gdb) continue
Continuing.
[Matrix Multiply Using CUDA] - Starting...

Breakpoint 2, __kernel_clock_gettime () at /build/linux-ZIBxfV/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S:127
127 /build/linux-ZIBxfV/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S: No such file or directory.
(gdb) bt
#0 __kernel_clock_gettime () at /build/linux-ZIBxfV/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S:127
#1 0x00007ffff7b8f530 in ?? () from /lib/powerpc64le-linux-gnu/libc.so.6
#2 0x00007ffff6b81118 in ?? () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#3 0x00007ffff6a69c70 in ?? () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#4 0x00007ffff6bf0ba0 in cuInit () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#5 0x000000010003ca50 in cudart::__loadDriverInternalUtil() ()
#6 0x00007ffff7f05274 in __pthread_once_slow (
    once_control=0x1000c00f0 <cudart::globalState::loadDriver()::loadDriverControl>,
    init_routine=0x10003c950 <cudart::__loadDriverInternalUtil()>) at pthread_once.c:116
#7 0x000000010008ea88 in cudart::cuosOnce(int*, void (*)()) ()
#8 0x00000001000410a8 in cudart::globalState::initializeDriver() ()
#9 0x000000010005ec90 in cudaGetDeviceCount ()
#10 0x0000000100009930 in gpuGetMaxGflopsDeviceId() ()
#11 0x0000000100009bf4 in findCudaDevice(int, char const**) ()
#12 0x000000010000836c in main ()
(gdb) step
128 in /build/linux-ZIBxfV/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S
(gdb) bt
#0 __kernel_clock_gettime () at /build/linux-ZIBxfV/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S:128
#1 0x0000000000000000 in ?? ()
(gdb)

Note: __kernel_clock_gettime+144 is currently the point in the function at which the syscall made, and is liable to change if updated. It corresponds to the "sc" instruction here: https://gitlab.com/TeeFirefly/linux-kernel/blob/7408b38cfdf9b0c6c3bda97402c75bd27ef69a85/arch/powerpc/kernel/vdso64/gettimeofday.S#L127 and can be rediscovered if needed by disassembling the function.

Note that a backtrace can be collected before entering the syscall, but not during. The inability to unwind also prevents GDB from being able to "finish" (step out of) the function:

(gdb) finish
Run till exit from #0 __kernel_clock_gettime ()
    at /build/linux-ZIBxfV/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S:128
Warning:
Cannot insert breakpoint 0.
Cannot access memory at address 0x0

Command aborted.
(gdb)

The cause of the issue is a lack of Call Frame Information (CFI) in the syscall code path, and so a potential fix here could be to save the link register and add the corresponding CFI directive for the syscall code path (as is done for the alternative code path).[Less]

This is now upstream accepted in the powerpc tree as git commit
https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?h=next&id=56d20861c027498b5a1112b4f9f05b56d906fdda
("powerpc/vdso: Correct call frame information")

[Joseph Salisbury (jsalisbury)]

I built a test kernel with linux-next commit 56d20861c027. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1797963

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-10-15 18:04 EDT-------
Test kernel looks good:
user@deb3qwsp2:/usr/local/cuda-10.0/samples/0_Simple/matrixMul$ cat /proc/version
Linux version 4.15.0-36-generic (jsalisbury@kathleen) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #40~lp1797963 SMP Mon Oct 15 19:50:41 UTC 2018
user@deb3qwsp2:/usr/local/cuda-10.0/samples/0_Simple/matrixMul$ gdb ./matrixMul
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./matrixMul...(no debugging symbols found)...done.
(gdb) break main
Breakpoint 1 at 0x8284
(gdb) run
Starting program: /usr/local/cuda-10.0/samples/0_Simple/matrixMul/matrixMul
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Breakpoint 1, 0x0000000100008284 in main ()
(gdb) break *(__kernel_clock_gettime+144)
Breakpoint 2 at 0x7ffff7f805e4: file /home/jsalisbury/bugs/lp1797963/ubuntu-bionic/arch/powerpc/kernel/vdso64/gettimeofday.S, line 128.
(gdb) continue
Continuing.
[Matrix Multiply Using CUDA] - Starting...

Breakpoint 2, __kernel_clock_gettime ()
at /home/jsalisbury/bugs/lp1797963/ubuntu-bionic/arch/powerpc/kernel/vdso64/gettimeofday.S:128
128 /home/jsalisbury/bugs/lp1797963/ubuntu-bionic/arch/powerpc/kernel/vdso64/gettimeofday.S: No such file or directory.
(gdb) bt
#0 __kernel_clock_gettime () at /home/jsalisbury/bugs/lp1797963/ubuntu-bionic/arch/powerpc/kernel/vdso64/gettimeofday.S:128
#1 0x00007ffff7b8f574 in __GI___clock_gettime (clock_id=<optimized out>, tp=<optimized out>)
at ../sysdeps/unix/clock_gettime.c:115
#2 0x00007ffff6bcab24 in ?? () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#3 0x00007ffff6b81118 in ?? () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#4 0x00007ffff6a69c70 in ?? () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#5 0x00007ffff6bf0ba0 in cuInit () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#6 0x000000010003ca50 in cudart::__loadDriverInternalUtil() ()
#7 0x00007ffff7f05274 in __pthread_once_slow (
once_control=0x1000c00f0 <cudart::globalState::loadDriver()::loadDriverControl>,
init_routine=0x10003c950 <cudart::__loadDriverInternalUtil()>) at pthread_once.c:116
#8 0x000000010008ea88 in cudart::cuosOnce(int*, void (*)()) ()
#9 0x00000001000410a8 in cudart::globalState::initializeDriver() ()
#10 0x000000010005ec90 in cudaGetDeviceCount ()
#11 0x0000000100009930 in gpuGetMaxGflopsDeviceId() ()
#12 0x0000000100009bf4 in findCudaDevice(int, char const**) ()
#13 0x0000...

[Joseph Salisbury (jsalisbury)]

SRU request submitted:
https://lists.ubuntu.com/archives/kernel-team/2018-October/096068.html

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-cosmic' to 'verification-done-cosmic'. If the problem still exists, change the tag 'verification-needed-cosmic' to 'verification-failed-cosmic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Mike Ranweiler (mranweil)]

Looks good on bionic:

user@deb3qwsp1:/usr/local/cuda-10.0/samples/0_Simple/matrixMul$ cat /proc/version
Linux version 4.15.0-39-generic (buildd@bos02-ppc64el-016) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #42-Ubuntu SMP Tue Oct 23 15:41:45 UTC 2018
user@deb3qwsp1:/usr/local/cuda-10.0/samples/0_Simple/matrixMul$ gdb ./matrixMul
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./matrixMul...(no debugging symbols found)...done.
(gdb) break main
Breakpoint 1 at 0x8284
(gdb) run
Starting program: /usr/local/cuda-10.0/samples/0_Simple/matrixMul/matrixMul
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Breakpoint 1, 0x0000000100008284 in main ()
(gdb) break *(__kernel_clock_gettime+144)
Breakpoint 2 at 0x7ffff7f805e4: file /build/linux-DTG0T7/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S, line 128.
(gdb) continue
Continuing.
[Matrix Multiply Using CUDA] - Starting...

Breakpoint 2, __kernel_clock_gettime () at /build/linux-DTG0T7/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S:128
128 /build/linux-DTG0T7/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S: No such file or directory.
(gdb) bt
#0 __kernel_clock_gettime () at /build/linux-DTG0T7/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday.S:128
#1 0x00007ffff7b8f574 in __GI___clock_gettime (clock_id=<optimized out>, tp=<optimized out>) at ../sysdeps/unix/clock_gettime.c:115
#2 0x00007ffff6bcab24 in ?? () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#3 0x00007ffff6b81118 in ?? () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#4 0x00007ffff6a69c70 in ?? () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#5 0x00007ffff6bf0ba0 in cuInit () from /usr/lib/powerpc64le-linux-gnu/libcuda.so.1
#6 0x000000010003ca50 in cudart::__loadDriverInternalUtil() ()
#7 0x00007ffff7f05274 in __pthread_once_slow (once_control=0x1000c00f0 <cudart::globalState::loadDriver()::loadDriverControl>,
    init_routine=0x10003c950 <cudart::__loadDriverInternalUtil()>) at pthread_once.c:116
#8 0x000000010008ea88 in cudart::cuosOnce(int*, void (*)()) ()
#9 0x00000001000410a8 in cudart::globalState::initializeDriver() ()
#10 0x000000010005ec90 in cudaGetDeviceCount ()
#11 0x0000000100009930 in gpuGetMaxGflopsDeviceId() ()
#12 0x0000000100009bf4 in findCudaDevice(int, char const**) ()
#13 0x000000010000836c in main ()
(gdb) step
129 in /build/linux-DTG0T7/linux-4.15.0/arch/powerpc/kernel/vdso64/gettimeofday...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-39.42

---------------
linux (4.15.0-39.42) bionic; urgency=medium

  * linux: 4.15.0-39.42 -proposed tracker (LP: #1799411)

  * Linux: insufficient shootdown for paging-structure caches (LP: #1798897)
    - mm: move tlb_table_flush to tlb_flush_mmu_free
    - mm/tlb: Remove tlb_remove_table() non-concurrent condition
    - mm/tlb, x86/mm: Support invalidating TLB caches for RCU_TABLE_FREE
    - [Config] CONFIG_HAVE_RCU_TABLE_INVALIDATE=y

  * Ubuntu18.04: GPU total memory is reduced (LP: #1792102)
    - Revert "powerpc/powernv: Increase memory block size to 1GB on radix"

  * arm64: snapdragon: reduce boot noise (LP: #1797154)
    - [Config] arm64: snapdragon: DRM_MSM=m
    - [Config] arm64: snapdragon: SND*=m
    - [Config] arm64: snapdragon: disable ARM_SDE_INTERFACE
    - [Config] arm64: snapdragon: disable DRM_I2C_ADV7511_CEC
    - [Config] arm64: snapdragon: disable VIDEO_ADV7511, VIDEO_COBALT

  * [Bionic] CPPC bug fixes (LP: #1796949)
    - ACPI / CPPC: Update all pr_(debug/err) messages to log the susbspace id
    - cpufreq: CPPC: Don't set transition_latency
    - ACPI / CPPC: Fix invalid PCC channel status errors

  * regression in 'ip --family bridge neigh' since linux v4.12 (LP: #1796748)
    - rtnetlink: fix rtnl_fdb_dump() for ndmsg header

  * screen displays abnormally on the lenovo M715 with the AMD GPU (Radeon Vega
    8 Mobile, rev ca, 1002:15dd) (LP: #1796786)
    - drm/amd/display: Fix takover from VGA mode
    - drm/amd/display: early return if not in vga mode in disable_vga
    - drm/amd/display: Refine disable VGA

  * arm64: snapdragon: WARNING: CPU: 0 PID: 1 arch/arm64/kernel/setup.c:271
    reserve_memblock_reserved_regions (LP: #1797139)
    - SAUCE: arm64: Fix /proc/iomem for reserved but not memory regions

  * The front MIC can't work on the Lenovo M715 (LP: #1797292)
    - ALSA: hda/realtek - Fix the problem of the front MIC on the Lenovo M715

  * Keyboard backlight sysfs sometimes is missing on Dell laptops (LP: #1797304)
    - platform/x86: dell-smbios: Correct some style warnings
    - platform/x86: dell-smbios: Rename dell-smbios source to dell-smbios-base
    - platform/x86: dell-smbios: Link all dell-smbios-* modules together
    - [Config] CONFIG_DELL_SMBIOS_SMM=y, CONFIG_DELL_SMBIOS_WMI=y

  * rpi3b+: ethernet not working (LP: #1797406)
    - lan78xx: Don't reset the interface on open

  * 87cdf3148b11 was never backported to 4.15 (LP: #1795653)
    - xfrm: Verify MAC header exists before overwriting eth_hdr(skb)->h_proto

  * [Ubuntu18.04][Power9][DD2.2]package installation segfaults inside debian
    chroot env in P9 KVM guest with HTM enabled (kvm) (LP: #1792501)
    - KVM: PPC: Book3S HV: Fix guest r11 corruption with POWER9 TM workarounds

  * Provide mode where all vCPUs on a core must be the same VM (LP: #1792957)
    - KVM: PPC: Book3S HV: Provide mode where all vCPUs on a core must be the same
      VM

  * fscache: bad refcounting in fscache_op_complete leads to OOPS (LP: #1797314)
    - SAUCE: fscache: Fix race in decrementing refcount of op->npages

  * CVE-2018-9363
    - Bluetooth: hidp: buffer overflow in hidp_process_report

  * CVE-20...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.18.0-11.12

---------------
linux (4.18.0-11.12) cosmic; urgency=medium

  * linux: 4.18.0-11.12 -proposed tracker (LP: #1799445)

  * arm64: snapdragon: WARNING: CPU: 0 PID: 1 arch/arm64/kernel/setup.c:271
    reserve_memblock_reserved_regions (LP: #1797139)
    - SAUCE: arm64: Fix /proc/iomem for reserved but not memory regions

  * arm64: snapdragon: WARNING: CPU: 0 PID: 1 at drivers/irqchip/irq-gic.c:1016
    gic_irq_domain_translate (LP: #1797143)
    - SAUCE: arm64: dts: msm8916: camms: fix gic_irq_domain_translate warnings

  * The front MIC can't work on the Lenovo M715 (LP: #1797292)
    - ALSA: hda/realtek - Fix the problem of the front MIC on the Lenovo M715

  * Provide mode where all vCPUs on a core must be the same VM (LP: #1792957)
    - KVM: PPC: Book3S HV: Provide mode where all vCPUs on a core must be the same
      VM

  * fscache: bad refcounting in fscache_op_complete leads to OOPS (LP: #1797314)
    - SAUCE: fscache: Fix race in decrementing refcount of op->npages

  * hns3: autoneg settings get lost on down/up (LP: #1797654)
    - net: hns3: Fix for information of phydev lost problem when down/up

  * not able to unwind the stack from within __kernel_clock_gettime in the Linux
    vDSO (LP: #1797963)
    - powerpc/vdso: Correct call frame information

  * Signal 7 error when running GPFS tracing in cluster (LP: #1792195)
    - powerpc/mm/books3s: Add new pte bit to mark pte temporarily invalid.
    - powerpc/mm/radix: Only need the Nest MMU workaround for R -> RW transition

  * Support Edge Gateway's WIFI LED (LP: #1798330)
    - SAUCE: mwifiex: Switch WiFi LED state according to the device status

  * Support Edge Gateway's Bluetooth LED (LP: #1798332)
    - SAUCE: Bluetooth: Support for LED on Edge Gateways

  * kvm doesn't work on 36 physical bits systems (LP: #1798427)
    - KVM: x86: fix L1TF's MMIO GFN calculation

  * CVE-2018-15471
    - xen-netback: fix input validation in xenvif_set_hash_mapping()

  * regression in 'ip --family bridge neigh' since linux v4.12 (LP: #1796748)
    - rtnetlink: fix rtnl_fdb_dump() for ndmsg header

 -- Stefan Bader <email address hidden> Tue, 23 Oct 2018 18:59:15 +0200