dev test in ubuntu_stress_smoke_tests throws usercopy warning s390x Ubuntu on LPAR

Bug #1800609 reported by Po-Hsu Lin on 2018-10-30
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Stress-ng
High
Colin Ian King
ubuntu-kernel-tests
Undecided
Unassigned
stress-ng (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned

Bug Description

SRU Request [Bionic, Cosmic]

== Justification ==

When running stress-ng with the dev stressor on s390x the FIBMAP ioctl is being incorrectly invoked in a raw block device. FIBMAP is intended for files
and not block devices. This ioctl needs to be removed from the stress test.

== Fix ==

Upstream commit http://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=53e5a98cd58d3b146cf8ffe52540722ff410a5ad

== Testing ==

Run:

sudo stress-ng --dev 1

Without the fix the kernel will throw warning errors. With the fix the test will run correctly without kernel issues.

== Regression Potential ==

Minimal. This affects just the dev stressor in stress-ng and removes functionality that was broken.

-----

The dev test on a s390x node (Ubuntu on LPAR) will hang, kernel bug could be found in the syslog

[54084.082277] ------------[ cut here ]------------
[54084.082279] kernel BUG at mm/usercopy.c:100!
[54084.082294] illegal operation: 0001 ilc:1 [#256] SMP
[54084.082297] Modules linked in: vhost_net vhost tap cuse dccp_ipv4 dccp algif_rng salsa20_generic camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw algif_skcipher tgr192 wp512 rmd320 rmd256 rmd160 rmd128 md4 algif_hash af_alg aufs ip6t_MASQUERADE ip6table_nat ip6table_filter ip6_tables xt_comment xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat xt_tcpudp bridge iptable_filter bpfilter openvswitch nsh nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_conncount nf_nat nf_conntrack 8021q garp mrp stp llc binfmt_misc zfs(PO) zunicode(PO) zavl(PO) icp(PO) dm_service_time zcommon(PO) znvpair(PO) dm_multipath scsi_dh_rdac scsi_dh_emc spl(O) scsi_dh_alua ghash_s390 prng aes_s390 des_s390 des_generic sha512_s390 sha256_s390
[54084.082354] sha1_s390 sha_common genwqe_card crc_itu_t chsc_sch eadm_sch ctcm fsm vfio_ccw vfio_mdev mdev vfio_iommu_type1 vfio sch_fq_codel nfsd ib_iser rdma_cm auth_rpcgss nfs_acl iw_cm lockd grace ib_cm iscsi_tcp libiscsi_tcp sunrpc libiscsi scsi_transport_iscsi ip_tables x_tables btrfs zstd_compress zlib_deflate raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear mlx4_en mlx4_ib qeth_l2 ib_core ptp pps_core crc32_vx_s390 mlx4_core devlink zfcp scsi_transport_fc qeth qdio ccwgroup dasd_eckd_mod dasd_mod
[54084.082418] CPU: 3 PID: 217520 Comm: stress-ng-dev Tainted: P D O 4.18.0-11-generic #12-Ubuntu
[54084.082420] Hardware name: IBM 2964 N63 400 (LPAR)
[54084.082422] Krnl PSW : 00000000ed7ed85f 00000000816ddc5c (usercopy_abort+0xb2/0xb8)
[54084.082433] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[54084.082436] Krnl GPRS: 0000000000000320 0000000000000006 0000000000000056 0000000000000007
[54084.082438] 0000000000000007 000000007fe00000 0000000000a96d3a 0000000000000000
[54084.082441] 000003ff00000000 0000000000000006 0000000000000006 00000002acb4ee00
[54084.082443] 0000000000a96d3a 0000000000914430 0000000000386606 000000038f087a78
[54084.082449] Krnl Code: 00000000003865fa: c02000390dc4 larl %r2,aa8182
                          0000000000386600: c0e5fff2b418 brasl %r14,1dce30
                         #0000000000386606: a7f40001 brc 15,386608
                         >000000000038660a: 0707 bcr 0,%r7
                          000000000038660c: 0707 bcr 0,%r7
                          000000000038660e: 0707 bcr 0,%r7
                          0000000000386610: c00400000000 brcl 0,386610
                          0000000000386616: ebcff0780024 stmg %r12,%r15,120(%r15)
[54084.082484] Call Trace:
[54084.082487] ([<0000000000386606>] usercopy_abort+0xae/0xb8)
[54084.082489] [<0000000000386522>] __check_object_size+0x19a/0x1d0
[54084.082492] [<00000000005c69d8>] sg_scsi_ioctl+0x120/0x3d8
[54084.082495] [<00000000005c75b6>] scsi_cmd_ioctl+0x3ce/0x3d8
[54084.082497] [<00000000005c87d0>] bsg_ioctl+0x118/0x250
[54084.082499] [<00000000003a3c9c>] do_vfs_ioctl+0xac/0x650
[54084.082502] [<00000000003a42cc>] ksys_ioctl+0x8c/0xb8
[54084.082504] [<00000000003a432a>] sys_ioctl+0x32/0x40
[54084.082507] [<00000000008f0dc0>] system_call+0xdc/0x2c8
[54084.082508] Last Breaking-Event-Address:
[54084.082510] [<0000000000386606>] usercopy_abort+0xae/0xb8
[54084.082512]
[54084.082513] ---[ end trace 5009dd6293a23031 ]---

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: linux-image-4.18.0-11-generic 4.18.0-11.12
ProcVersionSignature: Ubuntu 4.18.0-11.12-generic 4.18.12
Uname: Linux 4.18.0-11-generic s390x
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.10-0ubuntu13
Architecture: s390x
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Tue Oct 30 09:05:17 2018
HibernationDevice: RESUME=UUID=46fcf640-03e2-4fd5-b7d6-321d6a11ff8d
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
PciMultimedia:

ProcFB: Error: [Errno 2] No such file or directory: '/proc/fb'
ProcKernelCmdLine: root=UUID=7ca41340-b34a-4b16-87e9-3ceb5b51767d crashkernel=196M BOOT_IMAGE=0
RelatedPackageVersions:
 linux-restricted-modules-4.18.0-11-generic N/A
 linux-backports-modules-4.18.0-11-generic N/A
 linux-firmware 1.175
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)

Po-Hsu Lin (cypressyew) wrote :

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed

Occurs when performing the FIBMAP ioctl on /dev/bsg/rport-1:0-0

Did this issue start happening after an update/upgrade? Was there a
prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer
to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest
v4.19 kernel[0].

If this bug is fixed in the mainline kernel, please add the following
tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag:
'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as
"Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Changed in linux (Ubuntu):
assignee: nobody → Colin Ian King (colin-king)
importance: Undecided → Medium
status: Incomplete → In Progress
Colin Ian King (colin-king) wrote :

Fix committed to stress-ng:

http://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=53e5a98cd58d3b146cf8ffe52540722ff410a5ad

FIBMAP on a block device is plainly wrong, it should only be applied to a file. The FIBMAP ioctl to a scsi block device is the same as the SCSI_IOCTL_SEND_COMMAND ioctl that requires a scsi_ioctl_command struct. As it stands, we are passing an int with stack garbage as this struct which is passing garbage to the ioctl. The deprecated SCSI ioctl creates a scsi_request from the garbage and this causes the kernel to correctly warn with a usercopy_abort that the structure is bogos. Fix this by not calling FIBMAP on a raw SCSI device.

Changed in stress-ng:
status: New → Fix Committed
importance: Undecided → High
assignee: nobody → Colin Ian King (colin-king)
Changed in linux (Ubuntu):
status: In Progress → Invalid
no longer affects: linux (Ubuntu)
summary: - dev test in ubuntu_stress_smoke_tests will hang on s390x Ubuntu on LPAR
+ dev test in ubuntu_stress_smoke_tests throws usercopy warning s390x
+ Ubuntu on LPAR
Po-Hsu Lin (cypressyew) wrote :

Tested on the same node, the system can pass with the dev test now.
Thanks!

Changed in ubuntu-kernel-tests:
status: New → Fix Released
Colin Ian King (colin-king) wrote :

This requires SRU'ing.

Changed in stress-ng:
status: Fix Committed → Fix Released
description: updated
description: updated

Hello Po-Hsu, or anyone else affected,

Accepted stress-ng into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/stress-ng/0.09.42-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in stress-ng (Ubuntu):
status: New → Fix Released
Changed in stress-ng (Ubuntu Cosmic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Timo Aaltonen (tjaalton) wrote :

Hello Po-Hsu, or anyone else affected,

Accepted stress-ng into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/stress-ng/0.09.25-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in stress-ng (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
Colin Ian King (colin-king) wrote :

verified 0.09.42-1ubuntu1 is fixed for Cosmic

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
Colin Ian King (colin-king) wrote :

verified 0.09.25-1ubuntu1 is fixed for Bionic

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic

The verification of the Stable Release Update for stress-ng has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package stress-ng - 0.09.25-1ubuntu1

---------------
stress-ng (0.09.25-1ubuntu1) bionic; urgency=medium

  * stress-dev: remove FIBMAP ioctl on device (LP: #1800609)
    - fixes error of incorrect ioctl on block device
  * stress-revio: ensure fiemap extent struct is zero'd (LP: #1798566)
    - fixes fiemap ioctl failures because of garbage in struct

 -- Colin King <email address hidden> Wed, 14 Nov 2018 18:10:15 +0000

Changed in stress-ng (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package stress-ng - 0.09.42-1ubuntu1

---------------
stress-ng (0.09.42-1ubuntu1) cosmic; urgency=medium

  * stress-affinity: fix set affinity setting across all
    stressors (LP: #1803322)
    - fixes --seq 0 affinity setting stopping all CPUs from being used
  * stress-seccomp: add sanity check to see if SECCOMP_SET_MODE_FILTER
    works (LP: #1801856)
    - fixes false failure errors if filtering is not enabled
  * stress-dev: remove FIBMAP ioctl on device (LP: #1800609)
    - fixes error of incorrect ioctl on block device
  * stress-revio: ensure fiemap extent struct is zero'd (LP: #1798566)
    - fixes fiemap ioctl failures because of garbage in struct

 -- Colin King <email address hidden> Wed, 14 Nov 2018 17:28:22 +0000

Changed in stress-ng (Ubuntu Cosmic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers