stress-ng-malloc: allocation at 0x5631fc61e150 does not contain correct value

[Michael Reed]

== SRU Jammy ==

== Justification ==

stress-ng --malloc 1 --verify verification incorrectly performs a page checksum on the wrong page and will incorrectly report a test failure when the data is perfectly correct.

== Fix ==

This was fixed with the upstream fix:

commit ddecc607f916f8c9561f40c7ddc7d9e5c7528ff1
Author: Mayuresh Chitale <email address hidden>
Date: Tue Jun 7 12:07:10 2022 +0530

    stress-malloc: Fix for https://github.com/ColinIanKing/stress-ng/issues/201

    After realloc is done the address of the allocated memory is not stashed to the
    buffer like it is done after calloc/malloc. This causes the verify check
    to fail as the buffer contains the stale address.

    Signed-off-by: Mayuresh Chitale <email address hidden>

..this is the Kinetic release and works as expected. The fix is clean application of this patch.

== Test Case ==

Run:
stress-ng --aggressive --verify --timeout 300 --malloc 0

Without the fix, it will spit out errors like:
stress-ng: fail: [2060] stress-ng-malloc: allocation at 0x5590f0d68670 does not contain correct value
stress-ng: fail: [2060] stress-ng-malloc: allocation at 0x5590f0d68670 does not contain correct value
stress-ng: fail: [2059] stress-ng-malloc: allocation at 0x5590ea580830 does not contain correct value

== Regression Risk ==

This is a one line fix to store the correct checksum in the correct page. it only affects the one stress-ng stress case, namely the malloc stressor. The backport has been regression tested on Jammy for a 1hr run w/o seeing any regression. The change is also present in Kinetic and works perfectly. The regression risk is very small, this only change the behavior of the malloc test when using the --verify option. The modification is to memory that is mapped into the address space so there is no segmentation fault regression potential.

---------------------------------------------------------

On 22.04 stress-ng --malloc appears to pass but gives the following error messages:

stress-ng: fail: [2903753] stress-ng-malloc: allocation at 0x5631fc61b530 does not contain correct value
stress-ng: fail: [2903853] stress-ng-malloc: allocation at 0x5631fcb17920 does not contain correct value
...
info: 5 failures reached, aborting stress process
...
stress-ng: fail: [2903814] stress-ng-malloc: allocation at 0x5631fc6a92e0 does not contain correct value
stress-ng: info: [2903729] successful run completed in 10.81s

Steps to recreate:
stress-ng --aggressive --verify --timeout 300 --malloc 0

This appears to be a regression as this error is not seen on 20.04

[Michael Reed (mreed8855)]

Logfile test-memory command from the canonical-certification-server package

[Michael Reed (mreed8855)]

uname -a
Linux 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

sudo apt-cache policy canonical-certification-server
canonical-certification-server:
  Installed: 0.57.0+git202204251415+pkg174~ubuntu22.04.1
  Candidate: 0.57.0+git202204251415+pkg174~ubuntu22.04.1
  Version table:
 *** 0.57.0+git202204251415+pkg174~ubuntu22.04.1 500
        500 https://ppa.launchpadcontent.net/checkbox-dev/ppa/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

$ sudo apt-cache policy stress-ng
stress-ng:
  Installed: 0.13.12-2
  Candidate: 0.13.12-2
  Version table:
 *** 0.13.12-2 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
        100 /var/lib/dpkg/status

[Colin Ian King (colin-king)]

This error may indicate DRAM bit errors, so this may be a H/W memory corruption issue

[Jeff Lane  (bladernr)]

@colin-king so in that case, could this be a legitimate failure then? (and if so, shouldn't Stress--NG fail rather than pass the case?

As this is a gate for server cert, legitimate failures should always produce a failure exist in stress-ng.

@mreed8855 is there a stress-ng log in the submission that can be attached here to see the actual stress-ng output? Your example is seen through the lens of a python wrapper script and so related to my comment for Colin above, I'd like to be able to determine if this issue where it's passing though it appears to have failed is related to bad parsing in the wrapper, or because stress-ng is doing something unexpected.

[Zhanglei Mao (zhanglei-mao)]

One of our partners can reproduce it too on their server with Ubuntu 22.04
  stress-ng --aggressive --verify --timeout 300 --malloc 0

[Jeff Lane  (bladernr)]

Looks like Colin pushed a fix for this just a few hours ago in the upstream, we should work out who to bug to get that pulled into the Ubuntu package now...

https://github.com/ColinIanKing/stress-ng/issues/201

[Michael Reed (mreed8855)]

I have verified that the fix does work.

ubuntu@fleetroc:~/stress-ng$ ./stress-ng --aggressive --verify --timeout 300 --malloc 0
stress-ng: info: [19549] setting to a 300 second (5 mins, 0.00 secs) run per stressor
stress-ng: info: [19549] dispatching hogs: 160 malloc
stress-ng: info: [19549] successful run completed in 300.05s (5 mins, 0.05 secs)

ubuntu@fleetroc:~/stress-ng$ ./stress-ng --version
stress-ng, version 0.14.03 (gcc 11.2, x86_64 Linux 5.15.0-41-generic)

[Jeff Lane  (bladernr)]

@colin-king - how can we go about getting stress-ng updated in Ubuntu with the latest fixes, or at least is there a public PPA we can use so customers can get the latest fixes?

[Colin Ian King (colin-king)]

Package uploaded to -proposed

[Andreas Hasenack (ahasenack)]

Hi Colin, thanks for the upload. This is my first SRU review as a new member of the SRU team.

I noticed a few things:
a) there is no LP bug in the metadata (even though it's in d/changelog), was this source package built on debian by any chance?
b) an update-maintainer run is missing (d/control is still listing you as a maintainer instead of Ubuntu). Newer dpkg has a check for that and refuses a build:

  dpkg-source: error: Version number suggests Ubuntu changes, but Maintainer: does not have Ubuntu
  address

c) not a blocker, but I noticed that stress-ng doesn't follow the Security Team's versioning practices[1]. You are updating the version to 0.13.12-2ubuntu1 instead of 0.13.12-2ubuntu0.1. The upgrade path to the next Ubuntu release still works, so that's fine, and I see this pattern in previous stress-ng updates, but thought I would mention it.

I can fix (a) and (b) with a rebuild here, and upload and accept it myself, but you might prefer doing the rebuild yourself.

1. https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging

[Colin Ian King (colin-king)]

Hi Andreas,

Thanks for message. I've not made a SRU now for about a year, so it was
highly probable I made some mistakes :-)

On 28/10/2022 20:42, Andreas Hasenack wrote:
> Hi Colin, thanks for the upload. This is my first SRU review as a new
> member of the SRU team.
>
> I noticed a few things:
> a) there is no LP bug in the metadata (even though it's in d/changelog), was this source package built on debian by any chance?

Yes indeed. I primarily use Debian now as I normally upload these
packages into Debian and I haven't worked for Canonical for a year now.

> b) an update-maintainer run is missing (d/control is still listing you as a maintainer instead of Ubuntu). Newer dpkg has a check for that and refuses a build:
>
> dpkg-source: error: Version number suggests Ubuntu changes, but Maintainer: does not have Ubuntu
> address
Ooops. I forgot to run update-maintainer. My bad.

>
> c) not a blocker, but I noticed that stress-ng doesn't follow the
> Security Team's versioning practices[1]. You are updating the version to
> 0.13.12-2ubuntu1 instead of 0.13.12-2ubuntu0.1. The upgrade path to the
> next Ubuntu release still works, so that's fine, and I see this pattern
> in previous stress-ng updates, but thought I would mention it.

Good point. I'll follow that practice for subsequent uploads.

>
> I can fix (a) and (b) with a rebuild here, and upload and accept it
> myself, but you might prefer doing the rebuild yourself.

If you can fix these for me that would be most appreciated. :-)

Colin

>
>
> 1. https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging
>
> ** Changed in: stress-ng
> Status: Confirmed => Incomplete
>

[Andreas Hasenack (ahasenack)]

I rebuilt it and uploaded to unapproved, I'll continue my review with that one. The previous upload I rejected due to the reasons in comment #10.

[Andreas Hasenack (ahasenack)]

Hello Michael, or anyone else affected,

Accepted stress-ng into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/stress-ng/0.13.12-2ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Colin Ian King (colin-king)]

Tested on Ubuntu Jammy in a VM, without the fix I can reproduce the issue in several seconds, however with the -proposed 0.13.12-2ubuntu1 version I cannot trigger the issue with a 300 second test run.

Tested and verified as fixed.

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for stress-ng has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package stress-ng - 0.13.12-2ubuntu1

---------------
stress-ng (0.13.12-2ubuntu1) jammy; urgency=medium

  * Fix malloc stressor verification (LP: #1972906)
    - backport of upstream commit ddecc607f916f8c9561f40c7ddc7d9e5c7528ff1

 -- Colin Ian King <email address hidden> Fri, 28 Oct 2022 08:21:41 +0100