[Ubuntu 20.04] Problem leading IUCV service down (on s390x)

[bugproxy]

SRU Justification:
==================

[Impact]

* Problems occur in IBM z/VM's IUCV (Inter User Communication Vehicle) environments and its communication.

* Errors like "usercopy: Kernel memory overwrite attempt detected to SLUB object 'dma-kmalloc-1 k' (offset 0, size 11)!" pop up and cause failures.

* This is because IUCV uses kmalloc() with __GFP_DMA because of memory address restrictions.

* The solution is to mark dma-kmalloc caches as usercopy caches.

[Fix]

* 49f2d2419d60a103752e5fbaf158cf8d07c0d884 49f2d2419d60 "usercopy: mark dma-kmalloc caches as usercopy caches"

* Due to changes in the context of the upstream patch,
  a cherry-pick was not possible and the following backport was created:
  https://bugs.launchpad.net/bugs/1913442/+attachment/5457885/+files/commit_49f2d2419d60_backport.patch

[Test Case]

* Setup Ubuntu Server 20.04 on s390x system as IBM z/VM guest aka virtual machine.

* Setup IUCV on z/VM: Setting up the (IUCV TCPIP) service machine:
https://www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.linux.z.ljdd/ljdd_t_iucv_tcpservice.html

* Set up a Linux IUCV instance: https://www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.linux.z.ljdd/ljdd_t_iucv_scen1_guest.html

* Set up an IUCV direct: https://www.ibm.com/support/knowledgecenter/linuxonibm/com.ibm.linux.z.ljdd/ljdd_c_iucv_connect.html

* Make use of IUCV, for example using ssh on a direct connection.

* Verify if the connections is stable and watch out for messages starting with "usercopy".

[Regression Potential]

* Problems could occur in case the create_kmalloc_cache call is done wrong,
  for example with wrong index, wrong size or just wrong comma separations.

* Wrong size or index will probably lead to similar instability problems that exist today.

* Problems in the syntax (commas etc.) will be detected at compile time.

* But it's just a single line modification in function create_kmalloc_caches of /mm/slab_common.c,

* so the change is very limited and quite traceable.

* And it was in depth discussed here:
  https://lore.kernel<email address hidden>/

* a reviewed by a lot of kernel engineers (see provenance)

* and it was already upstream accepted with kernel 5.8.

[Other]

* Since the commit is upstream accepted with 5.8, so it's already in impish and hirsute (and groovy).

* Hence this kernel SRU submission is for Focal only and covers only the above single but common code commit/patch.
__________

When I deployed a Ubuntu20.04 instance with kernel version of 5.4.0-58-generic under z/VM, I saw below messages from kernel and the iucvserv program malfunctioned. Hence it caused some devices like network device configuration failure and deployment failure.

Dec 14 22:02:26 ub2004img iucvserv: Receive OPNCLD4 0.0.0.1 pwd sent from IUCV client.
Dec 14 22:02:26 ub2004img iucvserv: /etc/iucv_authorized_userid exists, check authorization.
Dec 14 22:02:26 ub2004img iucvserv: senduserid=OPNCLD4, authuserid=OPNCLD4, len=7
Dec 14 22:02:26 ub2004img iucvserv: Current version is 0.0.0.1, upgraded version is 0.0.0.1
Dec 14 22:02:26 ub2004img iucvserv: Will execute the linux command pwd 2>&1; echo iucvcmdrc=$? sent from IUCV client.
Dec 14 22:02:26 ub2004img iucvserv: result length=14, send message length=14,#012 /#012iucvcmdrc=0
Dec 14 22:02:26 ub2004img kernel: [63084.184649] ------------[ cut here ]------------
Dec 14 22:02:26 ub2004img kernel: [63084.184654] Bad or missing usercopy whitelist? Kernel memory exposure attempt detected from SLUB object 'dma-kmalloc-1k' (offset 0, size 20)!
Dec 14 22:02:26 ub2004img kernel: [63084.184680] WARNING: CPU: 1 PID: 697 at mm/usercopy.c:75 usercopy_warn+0xa0/0xd0
Dec 14 22:02:26 ub2004img kernel: [63084.184681] Modules linked in: tcp_diag udp_diag raw_diag inet_diag unix_diag xt_CT iptable_raw ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack nf_conntrack nf_defr
ag_ipv6 nf_defrag_ipv4 iptable_filter bpfilter af_iucv nls_utf8 isofs dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmur vfio_ccw vfio_mdev mdev s390_trng vfio_iommu_type1 vfio sch_fq_codel drm drm
_panel_orientation_quirks i2c_core ip_tables x_tables btrfs zstd_compress zlib_deflate raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear
 pkey zcrypt crc32_vx_s390 ghash_s390 prng aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 sha512_s390 sha256_s390 sha1_s390 sha_common dasd_fba_mod dasd_mod qeth_l2 qeth qdio ccwgroup
Dec 14 22:02:26 ub2004img kernel: [63084.184718] CPU: 1 PID: 697 Comm: iucvserv Not tainted 5.4.0-58-generic #64-Ubuntu
Dec 14 22:02:26 ub2004img kernel: [63084.184718] Hardware name: IBM 8561 LT1 400 (z/VM 7.1.0)
Dec 14 22:02:26 ub2004img kernel: [63084.184719] Krnl PSW : 0704c00180000000 00000000b3c37a60 (usercopy_warn+0xa0/0xd0)
Dec 14 22:02:26 ub2004img kernel: [63084.184721] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
Dec 14 22:02:26 ub2004img kernel: [63084.184722] Krnl GPRS: 0000000000000004 0000000000000006 0000000000000081 0000000000000007
Dec 14 22:02:26 ub2004img kernel: [63084.184722] 0000000000000007 00000000f2ecb400 00000000b43fdc6a 000003e000000014
Dec 14 22:02:26 ub2004img kernel: [63084.184723] 0000000000000000 0000000000000014 0000000000000000 00000000b43f01f0
Dec 14 22:02:26 ub2004img kernel: [63084.184723] 00000000aae13300 00000000e9332a00 00000000b3c37a5c 000003e000987a10
Dec 14 22:02:26 ub2004img kernel: [63084.184728] Krnl Code: 00000000b3c37a50: c020003e310f larl %r2,00000000b43fdc6e
Dec 14 22:02:26 ub2004img kernel: [63084.184728] 00000000b3c37a56: c0e5ffedbe85 brasl %r14,00000000b39ef760
Dec 14 22:02:26 ub2004img kernel: [63084.184728] #00000000b3c37a5c: a7f40001 brc 15,00000000b3c37a5e
Dec 14 22:02:26 ub2004img kernel: [63084.184728] >00000000b3c37a60: eb6ff0c00004 lmg %r6,%r15,192(%r15)
Dec 14 22:02:26 ub2004img kernel: [63084.184728] 00000000b3c37a66: 07fe bcr 15,%r14
Dec 14 22:02:26 ub2004img kernel: [63084.184728] 00000000b3c37a68: 47000700 bc 0,1792
Dec 14 22:02:26 ub2004img kernel: [63084.184728] 00000000b3c37a6c: c020003e30fa larl %r2,00000000b43fdc60
Dec 14 22:02:26 ub2004img kernel: [63084.184728] 00000000b3c37a72: a7f4ffd4 brc 15,00000000b3c37a1a
Dec 14 22:02:26 ub2004img kernel: [63084.184735] Call Trace:
Dec 14 22:02:26 ub2004img kernel: [63084.184736] ([<00000000b3c37a5c>] usercopy_warn+0x9c/0xd0)
Dec 14 22:02:26 ub2004img kernel: [63084.184740] [<00000000b3c0fcc8>] __check_heap_object+0xd8/0x150
Dec 14 22:02:26 ub2004img kernel: [63084.184741] [<00000000b3c37bc4>] __check_object_size+0x134/0x200
Dec 14 22:02:26 ub2004img kernel: [63084.184744] [<00000000b4080a7e>] simple_copy_to_iter+0x3e/0x70
Dec 14 22:02:26 ub2004img kernel: [63084.184745] [<00000000b407fe02>] __skb_datagram_iter+0x72/0x280
Dec 14 22:02:26 ub2004img kernel: [63084.184745] [<00000000b40800be>] skb_copy_datagram_iter+0x5e/0xe0
Dec 14 22:02:26 ub2004img kernel: [63084.184747] [<000003ff805014ea>] iucv_sock_recvmsg+0xaa/0x460 [af_iucv]
Dec 14 22:02:26 ub2004img kernel: [63084.184749] [<00000000b406ce36>] __sys_recvfrom+0xb6/0x140
Dec 14 22:02:26 ub2004img kernel: [63084.184750] [<00000000b406e042>] __s390x_sys_socketcall+0x222/0x350
Dec 14 22:02:26 ub2004img kernel: [63084.184753] [<00000000b4250ba2>] system_call+0x2a6/0x2c8
Dec 14 22:02:26 ub2004img kernel: [63084.184753] Last Breaking-Event-Address:
Dec 14 22:02:26 ub2004img kernel: [63084.184754] [<00000000b3c37a5c>] usercopy_warn+0x9c/0xd0
Dec 14 22:02:26 ub2004img kernel: [63084.184754] ---[ end trace b0232fe5536a773d ]---
Dec 14 22:02:26 ub2004img iucvserv: Receive OPNCLD4 0.0.0.1 ls /etc/*-release sent from IUCV client.
Dec 14 22:02:26 ub2004img iucvserv: /etc/iucv_authorized_userid exists, check authorization.
Dec 14 22:02:26 ub2004img iucvserv: senduserid=OPNCLD4, authuserid=OPNCLD4, len=7
Dec 14 22:02:26 ub2004img iucvserv: Current version is 0.0.0.1, upgraded version is 0.0.0.1
Dec 14 22:02:26 ub2004img iucvserv: Will execute the linux command ls /etc/*-release 2>&1; echo iucvcmdrc=$? sent from IUCV client.
Dec 14 22:02:26 ub2004img iucvserv: result length=45, send message length=45,#012 /etc/lsb-release#012/etc/os-release#012iucvcmdrc=0
Dec 14 22:02:26 ub2004img iucvserv: Receive OPNCLD4 0.0.0.1 cat /etc/os-release sent from IUCV client.
Dec 14 22:02:26 ub2004img iucvserv: /etc/iucv_authorized_userid exists, check authorization.
Dec 14 22:02:26 ub2004img iucvserv: senduserid=OPNCLD4, authuserid=OPNCLD4, len=7
Dec 14 22:02:26 ub2004img iucvserv: Current version is 0.0.0.1, upgraded version is 0.0.0.1

But I didn't see such problem with kernel version 5.4.0-40-generic #44-Ubuntu when I did the same operation.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-01-27 09:34 EDT-------
The solution be provided with the following git-commit.
https://github.com/torvalds/linux/commit/49f2d2419d60a103752e5fbaf158cf8d07c0d884

[Frank Heimes (fheimes)]

A kernel test build (binary-debs) from s390x can be found here:
https://people.canonical.com/~fheimes/lp1913442/

Cross-arch. builds (for amd64, arm64, ppc64el and amd64) can be found in this PPA:
https://launchpad.net/~fheimes/+archive/ubuntu/lp1913442

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2021-02-02 07:38 EDT-------

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-02-02 08:14 EDT-------
The latest update from Canonical is the same as the one updated last week.
During testingsted I don't see the kernel warning/error messages in the console log.
But I got a new issue.
Basically, the Nic device cannot be online via znetconfig.
I will attach the console log. That's the only thing we can get.
The guest didn't seem in emergency mode or other shell env.
So there is no way for me to check in the shell mode.
From the z/VM guest side, we saw the Nic was created and attached the vswitch.

[Frank Heimes (fheimes)]

Yes, the two kernel are on the same level - I did a rebuild since the above patch is common code and the kernel needed to be build on all significant architectures - that's what I did based on the PPA.

So looks like the messages/warning are gone, but that there is another issue.
I'm changing the status to Incomplete for the time being ...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-02-09 02:56 EDT-------
In the console log I see the following error message:

5.836326? zvmguestconfigure?619?: Successfully removed device 0.0.1000 (enc1000)
? 9.973839? zvmguestconfigure?652?: Scanning for network devices...
? 10.016611? zvmguestconfigure?664?: /usr/sbin/znetconf: line 562: echo: write error: Invalid argument
? 10.016650? zvmguestconfigure?664?: znetconf: Error: Failed to make 0.0.1000 online
? 10.198654? zvmguestconfigure?681?: inactive
?

"Invalid argument" is errno EINVAL, which is different from the errno of an other problem we are aware of. So this one is something different.

@Canonical: given the backport patch that is shown in the attachment this new problem cannot be related to the patch at all. So something else must have been changed in the kernel. Any ideas, additional fixes that went in, ... ?

[Frank Heimes (fheimes)]

Is it now a znetconf issue or still a kernel issue?
Can you manually enable the device by directly echo-ing to the sysfs entry?
(I guess the device is not in the cio ignore list or so.)
Did you also tried chzdev to enable it? (ideally with '--verbose')
(I was once told that this is the strategic tool to enable ccw devices, and since then I use this all the time).

Well, there are hundreds of patches that regularly find it's way into the focal kernel, since we continuously pick up almost everything that comes as upstream stable update.
These are handled by tickets like these:

...
Focal update: v5.4.89 upstream stable release - LP 1913486
Focal update: v5.4.90 upstream stable release - LP 1913487
Focal update: v5.4.91 upstream stable release - LP 1914654
...

There are of course some commits (just as an example, like: "s390/cio: fix use-after-free in ccw_device_destroy_console" introduced with Focal update: v5.4.86 upstream stable release (LP: #1910822)) that could have an impact.
But it's difficult to figure that out, since between the kernel you've used (5.4.0-58) and the sample build (5.4.0-66) from above, quite a lot came in.

So I went back in history (aka git log) to 5.4.0-58.64 / 6e9188016357
and patched that version with the same backport and did another test build.
You can now find kernel files with version 5.4.0-58 (in addition to 5.4.0-66) here: https://people.canonical.com/~fheimes/lp1913442/
that are worth to retry, I think.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-03-16 07:39 EDT-------
(In reply to comment #24)
> In the console log I see the following error message:
>
> 5.836326? zvmguestconfigure619?: Successfully removed device 0.0.1000
> (enc1000)
> 9.973839? zvmguestconfigure652?: Scanning for network devices...
> 10.016611? zvmguestconfigure664?: /usr/sbin/znetconf: line 562: echo:
> write error: Invalid argument
> 10.016650? zvmguestconfigure664?: znetconf: Error: Failed to make
> 0.0.1000 online
> 10.198654? zvmguestconfigure681?: inactive
>
>
> "Invalid argument" is errno EINVAL, which is different from the errno of an
> other problem we are aware of. So this one is something different.
>
> @Canonical: given the backport patch that is shown in the attachment this
> new problem cannot be related to the patch at all. So something else must
> have been changed in the kernel. Any ideas, additional fixes that went in,
> ... ?

FYI for the problem described in comment #24 a new problem was opened:
https://bugzilla.linux.ibm.com/show_bug.cgi?id=191691

[Dimitri John Ledkov (xnox)]

191691 has not been mirrored to launchpad, thus Ubuntu developers cannot see any of that details.

Note that Ubuntu does not have access to the LTC bugzilla, instead bugproxy mirrors reports to Launchpad as needed. Please check with hws if 191691 should be mirrored across, or not.

[Dimitri John Ledkov (xnox)]

This bug is now marked private on launchpad side. No longer visible to the public.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-03-19 10:59 EDT-------
With applying the provided upstream patch. This issue can be closed.
The refering other BZ is a different problem...

[Frank Heimes (fheimes)]

Ok, so the above patch is still needed for focal.
It's not in focal, yet, in case yes we have to proceed with SRU to focal.
But I want to do that only if _really_ needed, since it affects common code...

[Frank Heimes (fheimes)]

The commit mentioned in comment #3 is upstream with 5.8 (rc-1),
hence it's not only included in Impish, but also in Hirsute (and even in Groovy).
So if this will be SRUed, then it's an SRU to focal only.

[Frank Heimes (fheimes)]

Patch request submitted for focal:
https://lists.ubuntu.com/archives/kernel-team/2021-October/thread.html#124589
changing status to 'In Progress' for focal.

[Harald Freudenberger (freude-5)]

Hello Frank
hm, the output shows absolutely no relationship with AP bus or zcrypt device driver code.
Neither the kernel callstack nor the userspace iucvserv application has any relations to
my patch series.

[Frank Heimes (fheimes)]

Hi Harald, well, I'm a bit confused - probably because I don't see the entire discussion that you may have in bugzilla.
I guess you are talking about the new/add issue that was mentioned in LP comment #7 ?
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1913442/comments/7

Well, since this is obviously not related to the original issue that was reported by this bug,
I proceeded with the patch request (SRU) to focal and just requested to add a backport of this commit:
https://github.com/torvalds/linux/commit/49f2d2419d60a103752e5fbaf158cf8d07c0d884
to focal master next (backport only because of changes in the context).

And I agree with not seeing any relationship to AP and crypt (not sure why this assumption came up)?!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-10-19 09:40 EDT-------
Is there anything else we can do for this IUCV problem? If not then I would like to close the bug. Thank you!

[Frank Heimes (fheimes)]

Hi Karsten, no, nothing needed on top.
It's in status "Fix Committed", hence it's already available from the -proposed pocket of the Ubuntu archive, and will be migrated to -release with the next kernel update (which is a matter of days).

(What I just didn't understood was https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1913442/comments/18
maybe this was an accident ...)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-10-20 06:08 EDT-------
Hi Frank,

thanks for the summary! Closing this bug now...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.4.0-97.110 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Frank Heimes (fheimes)]

I could validate this, didn't noticed any 'usercopy' messages - same behavior like impish, hence updating tags...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.4.0-97.110

---------------
linux (5.4.0-97.110) focal; urgency=medium

  * icmp_redirect from selftests fails on F/kvm (unary operator expected)
    (LP: #1938964)
    - selftests: icmp_redirect: pass xfail=0 to log_test()

  * Focal: CIFS stable updates (LP: #1954926)
    - cifs: use the expiry output of dns_query to schedule next resolution
    - cifs: set a minimum of 120s for next dns resolution
    - cifs: To match file servers, make sure the server hostname matches

  * seccomp_bpf in seccomp from ubuntu_kernel_selftests failed to build on B-5.4
    (LP: #1896420)
    - SAUCE: selftests/seccomp: fix "storage size of 'md' isn't known" build issue
    - SAUCE: selftests/seccomp: Fix s390x regs not defined issue

  * system crash when removing ipmi_msghandler module (LP: #1950666)
    - ipmi: Move remove_work to dedicated workqueue
    - ipmi: msghandler: Make symbol 'remove_work_wq' static

  * zcrypt DD: Toleration for new IBM Z Crypto Hardware - (Backport to Ubuntu
    20.04) (LP: #1954680)
    - s390/AP: support new dynamic AP bus size limit

  * [UBUNTU 20.04] KVM hardware diagnose data improvements for guest kernel -
    kernel part (LP: #1953334)
    - s390/setup: diag 318: refactor struct
    - s390/kvm: diagnose 0x318 sync and reset
    - KVM: s390: remove diag318 reset code
    - KVM: s390: add debug statement for diag 318 CPNC data

  * Updates to ib_peer_memory requested by Nvidia (LP: #1947206)
    - SAUCE: RDMA/core: Updated ib_peer_memory

  * Include Infiniband Peer Memory interface (LP: #1923104)
    - IB: Allow calls to ib_umem_get from kernel ULPs
    - SAUCE: RDMA/core: Introduce peer memory interface

  * Focal update: v5.4.162 upstream stable release (LP: #1954834)
    - arm64: zynqmp: Do not duplicate flash partition label property
    - arm64: zynqmp: Fix serial compatible string
    - ARM: dts: NSP: Fix mpcore, mmc node names
    - scsi: lpfc: Fix list_add() corruption in lpfc_drain_txq()
    - arm64: dts: hisilicon: fix arm,sp805 compatible string
    - RDMA/bnxt_re: Check if the vlan is valid before reporting
    - usb: musb: tusb6010: check return value after calling
      platform_get_resource()
    - usb: typec: tipd: Remove WARN_ON in tps6598x_block_read
    - arm64: dts: qcom: msm8998: Fix CPU/L2 idle state latency and residency
    - arm64: dts: freescale: fix arm,sp805 compatible string
    - ASoC: SOF: Intel: hda-dai: fix potential locking issue
    - clk: imx: imx6ul: Move csi_sel mux to correct base register
    - ASoC: nau8824: Add DMI quirk mechanism for active-high jack-detect
    - scsi: advansys: Fix kernel pointer leak
    - firmware_loader: fix pre-allocated buf built-in firmware use
    - ARM: dts: omap: fix gpmc,mux-add-data type
    - usb: host: ohci-tmio: check return value after calling
      platform_get_resource()
    - ARM: dts: ls1021a: move thermal-zones node out of soc/
    - ARM: dts: ls1021a-tsn: use generic "jedec,spi-nor" compatible for flash
    - ALSA: ISA: not for M68K
    - tty: tty_buffer: Fix the softlockup issue in flush_to_ldisc
    - MIPS: sni: Fix the build
    - scsi: target: Fix ordered tag handling
    - scsi: target: Fix al...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-ibm-5.4/5.4.0-1014.15~18.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Frank Heimes (fheimes)]

This bug was for 20.04 GA kernel plain Ubuntu - hence verification on linux-ibm-5.4/5.4.0-1014 does not apply here.
However, I'm updating the tags to verification-done-bionic, just to unblock the process.