revio in ubuntu_stress_smoke_test failed with ARM64 X-hwe-edge

Bug #1798566 reported by Po-Hsu Lin
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Stress-ng
Fix Released
Medium
Colin Ian King
ubuntu-kernel-tests
Fix Released
Undecided
Unassigned
linux-hwe (Ubuntu)
Invalid
Undecided
Unassigned
Bionic
Invalid
Undecided
Unassigned
Cosmic
Invalid
Undecided
Unassigned
stress-ng (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned
Cosmic
Fix Released
Undecided
Unassigned

Bug Description

SRU Request [Bionic, Cosmic]

== Justification ==

When running stress-ng with the revio stressor, the FIEMAP ioctl is failing and causing the stress test to fail because the ioctl is being passed non-zero'd data from the stack.

== Fix ==

Upstream commit http://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=3784579562506d0b5f3d071aea10bb3dbc9f45d6

== Testing ==

On a ARM64 system run:

stress-ng --revio 1

Without the fix the revio stress test will randomly fail depending on the garbage on the stack. With the fix the stress test will run successfully.

== Regression Potential ==

Minimal, this affects just the revio stress test and is a simple memory zero'ing fix of a struct being passed to the FIEMAP ioctl. Regression potential is negligible.

--------------------

On a ThunderX Cavium ARM64 node, with kernel: 4.15.0-36-generic #39~16.04.1-Ubuntu
and the proposed 4.15.0-38-generic #41~16.04.1-Ubuntu

The revio test in ubuntu_stress_smoke_test will fail with:

 revio STARTING
  revio RETURNED 2
  revio FAILED
  stress-ng: debug: [4771] 48 processors online, 48 processors configured
  stress-ng: info: [4771] dispatching hogs: 4 revio
  stress-ng: info: [4771] cache allocate: using built-in defaults as unable to determine cache details
  stress-ng: debug: [4771] cache allocate: default cache size: 2048K
  stress-ng: debug: [4771] starting stressors
  stress-ng: debug: [4772] stress-ng-revio: started [4772] (instance 0)
  stress-ng: debug: [4773] stress-ng-revio: started [4773] (instance 1)
  stress-ng: debug: [4771] 4 stressors spawned
  stress-ng: debug: [4774] stress-ng-revio: started [4774] (instance 2)
  stress-ng: debug: [4775] stress-ng-revio: started [4775] (instance 3)
  stress-ng: fail: [4774] (null): open failed, errno=2 (No such file or directory)
  stress-ng: fail: [4775] (null): open failed, errno=2 (No such file or directory)
  stress-ng: fail: [4774] (null): rmdir './tmp-(null)-0-6' failed, errno=2 (No such file or directory)
  stress-ng: fail: [4775] (null): rmdir './tmp-(null)-0-6' failed, errno=2 (No such file or directory)
  stress-ng: fail: [4773] (null): rmdir './tmp-(null)-0-6' failed, errno=2 (No such file or directory)
  stress-ng: fail: [4772] (null): rmdir './tmp-(null)-0-6' failed, errno=2 (No such file or directory)
  stress-ng: debug: [4771] process [4772] (stress-ng-revio) terminated on signal: 6 (Aborted)
  stress-ng: debug: [4771] process [4772] terminated
  stress-ng: debug: [4771] process [4773] (stress-ng-revio) terminated on signal: 6 (Aborted)
  stress-ng: debug: [4771] process [4773] terminated
  stress-ng: debug: [4771] process [4774] (stress-ng-revio) terminated on signal: 6 (Aborted)
  stress-ng: debug: [4771] process [4774] terminated
  stress-ng: debug: [4771] process [4775] (stress-ng-revio) terminated on signal: 6 (Aborted)
  stress-ng: debug: [4771] process [4775] terminated
  stress-ng: info: [4771] unsuccessful run completed in 0.76s

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.15.0-36-generic 4.15.0-36.39~16.04.1
ProcVersionSignature: User Name 4.15.0-36.39~16.04.1-generic 4.15.18
Uname: Linux 4.15.0-36-generic aarch64
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: arm64
Date: Thu Oct 18 09:08:59 2018
SourcePackage: linux-hwe
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

This issue can be reproduced on a moonshot ARM64 node as well.

Revision history for this message
Colin Ian King (colin-king) wrote :

The fiemap data structure fields should all be zero'd before calling the fiemap ioctl. Looks like in ARM platforms the garbage data on the stack was causing stack clobbering causing the args->name field to be damaged because of the incorrect data being passed over to the ioctl.

Fixed: http://kernel.ubuntu.com/git/cking/stress-ng.git/commit/?id=3784579562506d0b5f3d071aea10bb3dbc9f45d6

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Tested on an ARM64 moonshot node (4.15.0-36-generic #39~16.04.1-Ubuntu),

11:21:44 INFO | revio STARTING
11:21:44 INFO | revio RETURNED 0
11:21:44 INFO | revio PASSED

Test passed with the fix.
Thanks

Changed in stress-ng:
importance: Undecided → Medium
status: New → Fix Committed
assignee: nobody → Colin Ian King (colin-king)
Po-Hsu Lin (cypressyew)
Changed in ubuntu-kernel-tests:
status: New → Fix Released
Changed in linux-hwe (Ubuntu):
status: New → Invalid
Po-Hsu Lin (cypressyew)
Changed in stress-ng:
status: Fix Committed → Fix Released
description: updated
description: updated
Timo Aaltonen (tjaalton)
Changed in linux-hwe (Ubuntu Bionic):
status: New → Invalid
Changed in linux-hwe (Ubuntu Cosmic):
status: New → Invalid
Changed in stress-ng (Ubuntu):
status: New → Fix Released
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Po-Hsu, or anyone else affected,

Accepted stress-ng into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/stress-ng/0.09.42-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in stress-ng (Ubuntu Cosmic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Po-Hsu, or anyone else affected,

Accepted stress-ng into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/stress-ng/0.09.25-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in stress-ng (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Colin Ian King (colin-king) wrote :

verified 0.09.42-1ubuntu1 is fixes on Cosmic

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
Revision history for this message
Colin Ian King (colin-king) wrote :

verified 0.09.25-1ubuntu1 is fixed on Bionic

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for stress-ng has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package stress-ng - 0.09.25-1ubuntu1

---------------
stress-ng (0.09.25-1ubuntu1) bionic; urgency=medium

  * stress-dev: remove FIBMAP ioctl on device (LP: #1800609)
    - fixes error of incorrect ioctl on block device
  * stress-revio: ensure fiemap extent struct is zero'd (LP: #1798566)
    - fixes fiemap ioctl failures because of garbage in struct

 -- Colin King <email address hidden> Wed, 14 Nov 2018 18:10:15 +0000

Changed in stress-ng (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package stress-ng - 0.09.42-1ubuntu1

---------------
stress-ng (0.09.42-1ubuntu1) cosmic; urgency=medium

  * stress-affinity: fix set affinity setting across all
    stressors (LP: #1803322)
    - fixes --seq 0 affinity setting stopping all CPUs from being used
  * stress-seccomp: add sanity check to see if SECCOMP_SET_MODE_FILTER
    works (LP: #1801856)
    - fixes false failure errors if filtering is not enabled
  * stress-dev: remove FIBMAP ioctl on device (LP: #1800609)
    - fixes error of incorrect ioctl on block device
  * stress-revio: ensure fiemap extent struct is zero'd (LP: #1798566)
    - fixes fiemap ioctl failures because of garbage in struct

 -- Colin King <email address hidden> Wed, 14 Nov 2018 17:28:22 +0000

Changed in stress-ng (Ubuntu Cosmic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.