[UBUNTU 22.04] Running smartctl on NVME hit segmentation fault

[bugproxy]

[ Impact ]

 * smartmontools crash when it is reading NVME error log on a big-endian system.

[ Test Plan ]

 * Prepare a big-endian system (like an Ubuntu on s390x)
   and attach an NVME drive to it.
   This nvme drive could have error logs, but it's ok if it doesn't.

 * Run `sudo smartctl -i -a /dev/nvme0` against NVME drive to get SMART data
   and hit Segmentation fault.

 * Run this command again with patched smartctl will show SMART data and error
   logs normally

[ Where problems could occur ]

 * The process of reading error log page from NVME device on big-endian system.

[ Other Info ]

 * Ubuntu 'noble' is not affected, since it incl. v7.4 which incl. the fix already.

 * We skip 23.04, since it is too close to end of life (in about 2 weeks from now).

------------------------------------------

== Comment: #0 - SCHAYNE BELLROSE <email address hidden> - 2023-10-31 10:08:10 ==
---Problem Description---
Ran sudo smartctl -i -a /dev/nvme0 against NVME drive to get SMART data but hit Segmentation fault

Contact Information = Schayne <email address hidden>

---Additional Hardware Info---
DPM system and added Storage group contains a NVME drive set for data.

---uname output---
Linux t249sb2 5.15.0-87-generic #97-Ubuntu SMP Mon Oct 2 21:11:23 UTC 2023 s390x s390x s390x GNU/Linux

Machine Type = 3932

---Steps to Reproduce---
 Run the following command
smartctl -i -a /dev/nvme0

*Additional Instructions for Schayne <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-11-06 19:32 EDT-------
--- Comment from Niklas S. ---
This is almost certainly the same problem as described in this upstream issue:
https://github.com/smartmontools/smartmontools/issues/172
"On Big-Endian systems smartctl -a /dev/nvmeXnY may segfault in nvme_read_error_log()"

[Zhaoxuan Zhai (kxuan)]

@fheimes Could you please help me to review this changes?

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "debdiff_smartmontools_jammy_from_7.2-1build2_to_7.2-1ubuntu0.2.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Łukasz Zemczak (sil2100)]

Okay, this looks okay-ish enough to sponsor, so I will upload it to the jammy queue. However, there's a few things:
1) This is only the debdiff for jammy, and I see mantic also doesn't have this change. Should we also upload this to mantic? We have this rule to make sure every change is available in the newer series as well as the older ones.
2) The test case should be made a bit more specific. Like, this only reproduces on a big endian system, right? Can you detail a bit more on where to test it then etc.?

[Łukasz Zemczak (sil2100)]

Sponsored for jammy. Not needed in noble as the change is already included in 7.4. I'd like to see this for mantic as well.

[Zhaoxuan Zhai (kxuan)]

Thank you Łukasz. The test case has been changed.

And I think I can continue work for mantic. This won't be long. :)

[Frank Heimes (fheimes)]

Thanks for the upload @kxuan.
And @sil2100, you've beaten me (I wanted to review, but I'm just back today from the YE break)...

[Zhaoxuan Zhai (kxuan)]

Hi @sil2100, this is the debdiff for mantic. Could you please take a look at this? Thank you!

[Andreas Hasenack (ahasenack)]

Subscribing ubuntu-sponsors, since the SRU cannot proceed without a mantic upload.

[Andreas Hasenack (ahasenack)]

Hello bugproxy, or anyone else affected,

Accepted smartmontools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/smartmontools/7.2-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

This was accepted for jammy so that the verification there can start, but an upload for mantic is still needed before anything can be released to updates.

[Simon Chopin (schopin)]

Mantic debdiff uploaded, now in the queue. Thanks for the patch :)

[Zhaoxuan Zhai (kxuan)]

Thank you @ahasenack.

Here is the verification. Please take a look.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Andreas Hasenack (ahasenack)]

Hello bugproxy, or anyone else affected,

Accepted smartmontools into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/smartmontools/7.3-1ubuntu0.23.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Zhaoxuan Zhai (kxuan)]

Hi @ahasenack, here is the verification result on mantic. Please take a look. Thank you!

[Robie Basak (racb)]

I think this should be also tested on a little endian system please. Otherwise we don't know if we fixed big endian by breaking little endian.

[Zhaoxuan Zhai (kxuan)]

Hi @racb, here are the verifications on amd64. Both of them do not have segmentation fault.

[Andreas Hasenack (ahasenack)]

Hi @kxuan,

thanks for the detailed verification steps. For some reason, however, you chose to download the debs directly from launchpad, instead of enabling the proposed repository, so I had to download them from mantic-proposed and the ones you used for verification and compare the hashes.

They check out, but please in the future just enable proposed, then your final "apt-cache policy" output will also show that the package is indeed coming from proposed, instead of just locally installed :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package smartmontools - 7.3-1ubuntu0.23.10.1

---------------
smartmontools (7.3-1ubuntu0.23.10.1) mantic; urgency=medium

  * Add d/p/lp2042885-fix-segfault-reading-nvme-error-log-on-be.patch
    to fix a smartctl on NVME segmentation fault. (LP: #2042885)

 -- Zhaoxuan Zhai <email address hidden> Sun, 24 Dec 2023 14:31:32 +0800

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for smartmontools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package smartmontools - 7.2-1ubuntu0.1

---------------
smartmontools (7.2-1ubuntu0.1) jammy; urgency=medium

  * Add d/p/lp2042885-fix-segfault-reading-nvme-error-log-on-be.patch
    to fix a smartctl on NVME segmentation fault. (LP: #2042885)

 -- Zhaoxuan Zhai <email address hidden> Sun, 03 Dec 2023 16:51:43 +0800

[Zhaoxuan Zhai (kxuan)]

@ahasenack Thanks for the notice. I will verify it through enable -proposed in the future.