[Ubuntu 20.10] zPCI DMA tables and bitmap leak on hard unplug (PCI Event 0x0304)

[bugproxy]

SRU Justification:
==================

[Impact]

* Since zpci_dma_exit_device() is never called on a zPCI device there is a potential leaking in DMA tables and bitmaps.

* This is because commit "s390/pci: adapt events for zbus" removed the zpci_disable_device() call for a zPCI event with PEC 0x0304 (means on hot unplug).

* It is only not called on hot unplug with event type PEC 0x0304 - this is the one where Linux is informed the device is
gone instead of being asked to deconfigure it.

* It should also always leak them with that event type on an enabled device.

[Fix]

* afdf9550e54627fcf4dd609bdc1153059378cdf5 afdf9550e546 "s390/pci: fix leak of DMA tables on hard unplug"

[Test Case]

* Have an IBM Z LPAR, that has PCIe devices (like RoCE adapters) assigned and Ubuntu Server 20.04 installed.

* Disable and re-enable one (or more) of the assigned PCIe cards (using hotplug) - on LPAR this can be triggered using the 'Reassign I/O Path' function at the HMC/SE.

* Monitor DMA tables and bitmaps for any kind of leaking.

* Since these tables are vmalloc-ed memory, it's sufficient to monitor via /proc/meminfo and see that reassigning back and forth of a device will have the memory usage grow continuously.

* The test and verification needs to be conducted by IBM.

[Regression Potential]

* There regression risk can be considered as moderate, because:

* only a call of zpci_disable_device(zdev) got reintroduced (and some comment lines).

* Since __zpci_event_availability gets modified, the zPCI event handling could be scrud up,

* which could cause issues regarding the availability of zPCI devices

* and in worst case make zPCI devices unusable.

* But only one switch case of the function is modified and all cases break, so only PEC 0x0304 should be affected.

* And the code changes themselves are minimal, and the zPCI code is limited to the s390x architecture.

* On top test kernels were built and shared for further testing.

[Other]

* Since this commit needs to land in groovy too, but groovy is still in development (hence the SRU process does not apply for groovy yet), I've sent a separate Patch request for groovy.

__________

Commit "s390/pci: adapt events for zbus" removed the zpci_disable_device() call for a zPCI event with PEC 0x0304 (hot unplug) because the device is already deconfigured by the platform.

This however skips the Linux side of the disable in particular it leads
to leaking the DMA tables and bitmaps because zpci_dma_exit_device() is
never called on the device.

This has been fixed in the following commit (currently in linux-next)

afdf9550e54627fcf4dd609bdc1153059378cdf5 s390/pci: fix leak of DMA tables on hard unplug

The commit re-introduces the zpci_disable_device() call as it was before the zbus introduction, for good measure I also added a comment to zpci_disable_device()
to call out the fact that it may be called with the device disabled
already.

As the commit was introduced with the multi-function support
this of course should go into both 20.10 and 20.04.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-09-18 06:41 EDT-------
Note I just checked and this cherry-picks cleanly on focal/master-next.

[Frank Heimes (fheimes)]

Commit afdf9550e546 landed upstream in linux-next and is currently tagged with 'next-20200915' only - probably planned to end up in 5.10.

[Frank Heimes (fheimes)]

Patch request submitted for groovy:
https://lists.ubuntu.com/archives/kernel-team/2020-September/thread.html#113599
changing status to 'In Progress' for groovy.

[Frank Heimes (fheimes)]

Patched kernel packages (based on focal master-next and groovy master-next) are available here for further testing: https://people.canonical.com/~fheimes/lp189284/

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-09-21 05:55 EDT-------
SRU Justification:

[Impact]

* As zpci_dma_exit_device() is never called on a zPCI device that
is removed via PCI event which only informs Linxu of device removal
instead of requesting deconfiguration, the vmalloced memory for
DMA tables and vma tracking leaks in this case.

* This is because commit "s390/pci: adapt events for zbus" removed the zpci_disable_device() call for a zPCI event with PEC 0x0304 since
the device is then removed via zpci_release_device() however this
did not free the dma tables because the device already appeared
to be in Standby state.

[Fix]

* afdf9550e54627fcf4dd609bdc1153059378cdf5 afdf9550e546 "s390/pci: fix leak of DMA tables on hard unplug"

[Test Case]

* Have an IBM Z LPAR, that has PCIe devices (like RoCE adapters) assigned and Ubuntu Server 20.04 installed.

* Disable and re-enable one (or more) of the assigned PCIe cards using the Reassign I/O Path functionality of the HMC/SE,

* Monitor /proc/meminfo which shows the vmalloc memory usage not
go back to the value before attaching the device.

* The test and verification will be conducted by IBM.

[Regression Potential]

* There regression risk can be considered as moderate, because:

* Only a call of zpci_disable_device(zdev) got reintroduced (and some comment lines). This call was done in the same place and with the same
functionality prior to commit "s390/pci: adapt events for zbus".

* Since __zpci_event_availability gets modified, the zPCI event handling could be broken. It is however only modified for the single PEC 0x0304 case
and all cases execute independently.

* Nevertheless this could cause issues regarding the availability of zPCI devices

* In the worst case zPCI devices could become unusable.

* The code changes themselves are minimal, and the zPCI code is limited to the s390x architecture.

* On top test kernels were built and shared for further testing.

[Other]

* Since this commit needs to land in groovy too, but groovy is still in development (hence the SRU process does not apply for groovy yet, a separate Patch request for groovy was made.

[Frank Heimes (fheimes)]

Patch request submitted for focal:
https://lists.ubuntu.com/archives/kernel-team/2020-September/thread.html#113609
changing status to 'In Progress' for focal.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.8.0-20.21

---------------
linux (5.8.0-20.21) groovy; urgency=medium

  * groovy/linux: 5.8.0-20.21 -proposed tracker (LP: #1896668)

  * Lenovo ThinkBook 14-IML Touchpad not showing up in /proc/bus/input/devices
    (LP: #1853277)
    - i2c: core: Call i2c_acpi_install_space_handler() before
      i2c_acpi_register_devices()

  * Enable LTR for endpoints behind VMD (LP: #1896598)
    - SAUCE: PCI/ASPM: Enable LTR for endpoints behind VMD

  * Remove duplicated code in ip_defrag.sh of kselftests/net (LP: #1894062)
    - Revert "UBUNTU: SAUCE: selftests: net: ip_defrag: modprobe missing
      nf_defrag_ipv6 support"

  * [SRU] [Focal/OEM-5.6/Groovy]Fix AMD usb host controller lost after stress S3
    (LP: #1893914)
    - SAUCE: xhci: workaround for S3 issue on AMD SNPS 3.0 xHC

  * debian/rules editconfigs does not work on s390x to change s390x only configs
    (LP: #1863116)
    - [Packaging] kernelconfig -- only update/edit configurations on architectures
      we have compiler support

  * [Ubuntu 20.10] zPCI DMA tables and bitmap leak on hard unplug (PCI Event
    0x0304) (LP: #1896216)
    - s390/pci: fix leak of DMA tables on hard unplug

  * md: improve IO accounting (LP: #1891151)
    - md: improve io stats accounting

  * Groovy update: v5.8.10 upstream stable release (LP: #1896078)
    - ARM: OMAP2+: Fix an IS_ERR() vs NULL check in _get_pwrdm()
    - ARM: dts: logicpd-torpedo-baseboard: Fix broken audio
    - ARM: dts: logicpd-som-lv-baseboard: Fix broken audio
    - ARM: dts: logicpd-som-lv-baseboard: Fix missing video
    - regulator: push allocation in regulator_ena_gpio_request() out of lock
    - regulator: remove superfluous lock in regulator_resolve_coupling()
    - ARM: dts: socfpga: fix register entry for timer3 on Arria10
    - ARM: dts: omap5: Fix DSI base address and clocks
    - ARM: dts: ls1021a: fix QuadSPI-memory reg range
    - ARM: dts: imx7ulp: Correct gpio ranges
    - arm64: dts: imx: Add missing imx8mm-beacon-kit.dtb to build
    - ARM: dts: imx7d-zii-rmu2: fix rgmii phy-mode for ksz9031 phy
    - RDMA/rtrs-srv: Replace device_register with device_initialize and device_add
    - RDMA/rxe: Fix memleak in rxe_mem_init_user
    - RDMA/rxe: Drop pointless checks in rxe_init_ports
    - RDMA/rxe: Fix panic when calling kmem_cache_create()
    - RDMA/bnxt_re: Do not report transparent vlan from QP1
    - RDMA/bnxt_re: Fix the qp table indexing
    - RDMA/bnxt_re: Static NQ depth allocation
    - RDMA/bnxt_re: Fix driver crash on unaligned PSN entry address
    - RDMA/bnxt_re: Remove the qp from list only if the qp destroy succeeds
    - drm/sun4i: add missing put_device() call in sun8i_r40_tcon_tv_set_mux()
    - arm64: dts: imx8mq: Fix TMU interrupt property
    - drm/sun4i: Fix dsi dcs long write function
    - scsi: qla2xxx: Fix regression on sparc64
    - scsi: libsas: Set data_dir as DMA_NONE if libata marks qc as NODATA
    - drm/virtio: fix unblank
    - RDMA/core: Fix unsafe linked list traversal after failing to allocate CQ
    - RDMA/core: Fix reported speed and width
    - scsi: megaraid_sas: Don't call disable_irq from process IRQ poll
    - scsi: mpt3sas: Don'...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-11-17 09:58 EDT-------
(In reply to comment #16)
> This bug is awaiting verification that the kernel in -proposed solves the
> problem. Please test the kernel and update this bug with the results. If the
> problem is solved, change the tag 'verification-needed-focal' to
> 'verification-done-focal'. If the problem still exists, change the tag
> 'verification-needed-focal' to 'verification-failed-focal'.
>
> If verification is not done by 5 working days from today, this fix will be
> dropped from the source code, and this bug will be closed.
>
> See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
> enable and use -proposed. Thank you!

I've tested the Kernel and also checked the code in focal master-next
everything looks good thanks!

[Frank Heimes (fheimes)]

Thanks for the verification, Niklas - adjusting tag accordingly ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.4.0-56.62

---------------
linux (5.4.0-56.62) focal; urgency=medium

  * focal/linux: 5.4.0-56.62 -proposed tracker (LP: #1905300)

  * CVE-2020-4788
    - selftests/powerpc: rfi_flush: disable entry flush if present
    - powerpc/64s: flush L1D on kernel entry
    - powerpc/64s: flush L1D after user accesses
    - selftests/powerpc: entry flush test

linux (5.4.0-55.61) focal; urgency=medium

  * focal/linux: 5.4.0-55.61 -proposed tracker (LP: #1903175)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * CVE-2020-14351
    - perf/core: Fix race in the perf_mmap_close() function

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout
    - dm raid: fix discard limits for raid1 and raid10
    - dm raid: remove unnecessary discard limits for raid10

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: drop unnecessary offset_in_page in extent buffer helpers
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Ethernet no link lights after reboot (Intel i225-v 2.5G) (LP: #1902578)
    - igc: Add PHY power management control

  * Undetected Data corruption in MPI workloads that use VSX for reductions on
    POWER9 DD2.1 systems (LP: #1902694)
    - powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
    - selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
      workaround

  * [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
    - s390: nvme ipl
    - s390: nvme reipl
    - s390/ipl: support NVMe IPL kernel parameters

  * uvcvideo: add mapping for HEVC payloads (LP: #1895803)
    - media: uvcvideo: Add mapping for HEVC payloads

  * Focal update: v5.4.73 upstream stable release (LP: #1902115)
    - ibmveth: Switch order of ibmveth_helper calls.
    - ibmveth: Identify ingress large send packets.
    - ipv4: Restore flowi4_oif update before call to xfrm_lookup_route
    - mlx4: handle non-napi callers to napi_poll
    - net: fec: Fix phy_device lookup for phy_reset_after_clk_enable()
    - net: fec: Fix PHY init after phy_reset_after_clk_enable()
    - net: fix pos incrementment in ipv6_route_seq_next
    - net/smc: fix valid DMBE buffer sizes
    - net...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-01 05:27 EDT-------
IBM Bugzilla status->closed, Fix Released by all requested distros