[UBUNTU 20.04] s390x/pci: s390_pci_mmio_write/read fail when MIO instructions are available

Bug #1874055 reported by bugproxy on 2020-04-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
High
Skipper Bug Screeners
linux (Ubuntu)
Undecided
Canonical Kernel Team
Focal
Undecided
Unassigned

Bug Description

SRU Justification:
==================

[Impact]

* Code that is using s390_pci_mmio_write/read system calls on a z15 (that comes with enhanced PCI load/store instructions), fails with "Unable to handle kernel pointer dereference in virtual kernel address space".

* This issue happens if enablement for z15 PCI enhancements is in place and where customers run workloads which access PCI adapters from user space, like RoCE/RDMA.

* To solve this, the system call implementation needs to be improved to execute the enhanced PCI load/store instructions on behalf of the user space application,
  making use of the mappings into its virtual address space.

[Fix]

* f058599e22d59e594e5aae1dc10560568d8f4a8b f058599e22d5 "s390/pci: Fix s390_mmio_read/write with MIO"

[Test Case]

* Setting up a z15 with at least one PCI card (like RoCE) using an operating system that includes support and enablement for z15 (line 20.04).

* Install the rdma tools: sudo apt install ibverbs-providers ibverbs-utils

* Verify you have some RDMA devices (requires ConnectX adapter)
  $ ibv_devices
  device node GUID
  ------ ----------------
  mlx5_0 98039b0300c682b4

* Verify MIO instructions are enabled for the device
  $ cat /sys/bus/pci/devices/0000\:00\:00.0/mio_enabled
  1

* Try to run an RDMA application from user space, e.g. ibv_rc_pingpong
  server side:
  ibv_rc_pingpong -d mlx5_0 -g 0 &
  client side:
  ibv_rc_pingpong -d mlx5_0 -g 0 localhost

* Verify whether the kernel crashes or not.

* Verification needs to be done by IBM on z15 hardware.

[Regression Potential]

* There is some regression potential with having code changes in the zPCI sub-system (zPCI is limited to s390x)

* It could be that zPCI hardware get harmed, but zPCI hardware is not as wide-spread on s390x than ccw hardware components.

* Only z15 hardware is affected - no other s390x hardware that is supported by Ubuntu.

* However, the zPCI system is s390x only and the patch was accepted upstream with v5.7-rc7 and Linus commented: "And none of the fixes look like there's anything particularly scary going on. Most of it is very small, and the slightly larger patches aren't huge either and are well-contained (the two slightly larger patches are to s390 and rxrpc - and even those patches aren't really all _that_ big)"
__________

One of the PCI enhancements on Z15 are the enhanced PCI load/store instructions which can be executed directly from user space code. When these instructions are available and preexisting user space code still uses the old s390_pci_mmio_write/read system calls, the system calls fail with an "Unable to handle kernel pointer dereference in virtual kernel address space" in the kernel. This issue affects distributions which have the enablement for Z15 PCI enhancements and where customers run workloads which accesses PCI adapters from user space, e.g. RDMA applications. To solve this, the system call implementation needs to be enhanced to provide to execute enhanced PCI load/store instructions on behalf of the user space application making use of the mappings into its virtual address space

CVE References

bugproxy (bugproxy) on 2020-04-21
tags: added: architecture-s39064 bugnameltc-185396 severity-high targetmilestone-inin2004
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
summary: - s390x/pci: s390_pci_mmio_write/read fail when MIO instructions are
- available
+ [UBUNTU 20.04] s390x/pci: s390_pci_mmio_write/read fail when MIO
+ instructions are available

------- Comment From <email address hidden> 2020-04-21 09:41 EDT-------
One of the PCI enhancements on Z15 are the enhanced PCI load/store instructions which can be executed directly from user space code. When these instructions are available and preexisting user space code still uses the old s390_pci_mmio_write/read system calls, the system calls fail with an "Unable to handle kernel pointer dereference in virtual kernel address space" in the kernel.

This issue affects distributions which have the enablement for Z15 PCI enhancements and where customers run workloads which accesses PCI adapters from user space, e.g. RDMA applications.

To solve this, the system call implementation needs to be enhanced to provide to execute enhanced PCI load/store instructions on behalf of the user space application making use of the mappings into its virtual address space

Frank Heimes (fheimes) on 2020-04-21
description: updated
Frank Heimes (fheimes) wrote :

That should be okay to SRU, but was this fix already brought upstream?
And if so would you please share the commit (or backport) that does apply cleanly to the focal master-next tree?

Changed in ubuntu-z-systems:
status: New → Incomplete
importance: Undecided → High
Frank Heimes (fheimes) on 2020-05-13
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
Changed in linux (Ubuntu):
status: New → Incomplete
assignee: Skipper Bug Screeners (skipper-screen-team) → Canonical Kernel Team (canonical-kernel-team)
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-05-18 05:20 EDT-------
My patch for this issue is now available publicly on the fixes branch of the public s390 repository on kernel.org.

https://git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git/commit/?h=fixes&id=f058599e22d59e594e5aae1dc10560568d8f4a8b

Frank Heimes (fheimes) wrote :

It's now also in linux-next, tagged with 'next-20200515'.
Pretty big patch for an SRU, but can cherry-picked cleanly from focal master-next.
I'll start to prepare the SRU soon - changing status to Triaged ...

Changed in linux (Ubuntu):
status: Incomplete → Triaged
Changed in ubuntu-z-systems:
status: Incomplete → Triaged
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-05-18 07:18 EDT-------
I agree, it's pretty big but this was the least impact approach we could think
of and it fixes a kernel crash that can be triggered from user space simply by invoking the s390_pci_read/write_mmio syscalls when running with MIO.

The code mostly mirrors existing PCI code with the addition of the secondary address space access and is otherwise self contained so it should be extremely unlikely to impact anything apart from these two syscalls.

With the new PCI instructions that can be used in userspace, these syscalls will not be needed in the future but we of course still need them for applications that have not been updated to use the new instructions.

Frank Heimes (fheimes) wrote :

Hi Niklas, please can you have a look at the SRU Justification that I've now added to the bug description, with special focus on Test Case and Regression Potential.
Please let me know if is inaccurate or missing or in case additional information should be added (this is mandatory information needed for the SRU).
Thx

description: updated
Frank Heimes (fheimes) on 2020-05-19
description: updated
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-05-26 04:04 EDT-------
(In reply to comment #12)
> Hi Niklas, please can you have a look at the SRU Justification that I've now
> added to the bug description, with special focus on Test Case and Regression
> Potential.
> Please let me know if is inaccurate or missing or in case additional
> information should be added (this is mandatory information needed for the
> SRU).
> Thx

Sounds good, maybe to add that this can be triggered with shipping user space tools (including in the Ubuntu repos) e.g.:

1. install the rdma tools:
sudo apt-get install ibverbs-providers ibverbs-utils

2. verify you have some RDMA devices (requires ConnectX adapter)
pcidev@T224LP06:~$ ibv_devices
device node GUID
------ ----------------
mlx5_0 98039b0300c682b4

3. verify MIO instructions are enabled for the device
pcidev@T224LP06:~$ cat /sys/bus/pci/devices/0000\:00\:00.0/mio_enabled
1

4. try to run an RDMA application from user space, e.g. ibv_rc_pingpong

server process:
ibv_rc_pingpong -d mlx5_0 -g 0 &

client process:
ibv_rc_pingpong -d mlx5_0 -g 0 localhost

5. verify that the kernel crashes

[92406.190525] Unable to handle kernel pointer dereference in virtual kernel add
ress space
[92406.190529] Failing address: ed00000000090000 TEID: ed00000000090403
[92406.190529] Fault in home space mode while using kernel ASCE.
[92406.190531] AS:0000000c1c98c007 R3:0000000ff3bd0007 S:0000000ff3bd6000 P:0000
00000009013d

Also this patch made it into v5.7-rc7 and Linus himself commented:

"And none of the fixes look like there's anything particularly scary
going on. Most of it is very small, and the slightly larger patches
aren't huge either and are well-contained (the two slightly larger
patches are to s390 and rxrpc - and even those patches aren't really
all _that_ big)"

(this patch obviously being the bigger s390 change)

Frank Heimes (fheimes) wrote :

Hi Niklas, thanks - I'll adjust the justification slightly and will reference your comment - especially regarding the test and the now upstream acceptance.

Frank Heimes (fheimes) on 2020-05-26
description: updated
Frank Heimes (fheimes) on 2020-05-28
description: updated
Frank Heimes (fheimes) wrote :

Kernel SRU request submitted:
https://lists.ubuntu.com/archives/kernel-team/2020-May/thread.html#110497
Updating status to 'In Progress'.

Changed in linux (Ubuntu):
status: Triaged → In Progress
Changed in ubuntu-z-systems:
status: Triaged → In Progress
Changed in linux (Ubuntu Focal):
status: New → In Progress
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Frank Heimes (fheimes) on 2020-06-05
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-06-12 03:37 EDT-------
Ok I just verified that this is now working as expected on
5.4.0-38-generic from Focal Proposed. Thank you!

Andrew Cloke (andrew-cloke) wrote :

Thanks Niklas! Adjusting tags accordingly.

tags: added: verification-done-focal
removed: verification-needed-focal

All autopkgtests for the newly accepted linux-oracle-5.4 (5.4.0-1019.19~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

zfs-linux/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-oracle-5.4

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Launchpad Janitor (janitor) wrote :
Download full text (30.0 KiB)

This bug was fixed in the package linux - 5.4.0-40.44

---------------
linux (5.4.0-40.44) focal; urgency=medium

  * linux-oem-5.6-tools-common and -tools-host should be dropped (LP: #1881120)
    - [Packaging] Add Conflicts/Replaces to remove linux-oem-5.6-tools-common and
      -tools-host

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  * Slow send speed with Intel I219-V on Ubuntu 18.04.1 (LP: #1802691)
    - e1000e: Disable TSO for buffer overrun workaround

  * CVE-2020-0543
    - UBUNTU/SAUCE: x86/speculation/srbds: do not try to turn mitigation off when
      not supported

  * Realtek 8723DE [10ec:d723] subsystem [10ec:d738] disconnects unsolicitedly
    when Bluetooth is paired: Reason: 23=IEEE8021X_FAILED (LP: #1878147)
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: Move driver IQK to set channel before
      association for 11N chip"
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: fix rate for a while after being
      connected"
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: No retry and report for auth and assoc"
    - SAUCE: Revert "UBUNTU: SAUCE: rtw88: 8723d: Add coex support"
    - rtw88: add a debugfs entry to dump coex's info
    - rtw88: add a debugfs entry to enable/disable coex mechanism
    - rtw88: 8723d: Add coex support
    - SAUCE: rtw88: coex: 8723d: set antanna control owner
    - SAUCE: rtw88: coex: 8723d: handle BT inquiry cases
    - SAUCE: rtw88: fix EAPOL 4-way failure by finish IQK earlier

  * CPU stress test fails with focal kernel (LP: #1867900)
    - [Config] Disable hisi_sec2 temporarily

  * Enforce all config annotations (LP: #1879327)
    - [Config]: do not enforce CONFIG_VERSION_SIGNATURE
    - [Config]: prepare to enforce all
    - [Config]: enforce all config options

  * Focal update: v5.4.44 upstream stable release (LP: #1881927)
    - ax25: fix setsockopt(SO_BINDTODEVICE)
    - dpaa_eth: fix usage as DSA master, try 3
    - net: don't return invalid table id error when we fall back to PF_UNSPEC
    - net: dsa: mt7530: fix roaming from DSA user ports
    - net: ethernet: ti: cpsw: fix ASSERT_RTNL() warning during suspend
    - __netif_receive_skb_core: pass skb by reference
    - net: inet_csk: Fix so_reuseport bind-address cache in tb->fast*
    - net: ipip: fix wrong address family in init error path
    - net/mlx5: Add command entry handling completion
    - net: mvpp2: fix RX hashing for non-10G ports
    - net: nlmsg_cancel() if put fails for nhmsg
    - net: qrtr: Fix passing invalid reference to qrtr_local_enqueue()
    - net: revert "net: get rid of an signed integer overflow in
      ip_idents_reserve()"
    - net sched: fix reporting the first-time use timestamp
    - net/tls: fix race condition causing kernel panic
    - nexthop: Fix attribute checking for groups
    - r8152: support additional Microsoft Surface Ethernet Adapter variant
    - sctp: Don't add the shutdown timer if its already been added
    - sctp: Start shutdown on association restart if in SHUTDOWN-SENT state and
      socket is closed
    - tipc: block BH before using dst_cache
    - net/mlx5e: kTLS, Destroy key object after destroying the TIS
    - net/mlx5e: Fix inner tirs handling
    - net/m...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released

------- Comment From <email address hidden> 2020-07-01 07:04 EDT-------
IBM Bugzilla-> closed, Fix Released with focal

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers