cannot display sensor name when its owner is lun1

[Gabriel B. Sant'Anna]

SRU Justification:

[ Impact ]

ipmitool sel does not correctly display the sensor's name if its owner is set to lun1.
Upstream bug: https://codeberg.org/IPMITool/ipmitool/issues/8

We were asked to track this in order to enable new hardware from a partner.
It was reported in v1.8.19 (used in Noble), but likely affects previous versions as well.

Steps to reproduce, copied verbatim from the upstream report:

1. Using AMI/BMC to generate a sensor error event. The sensor belong to LUN1
GPU1_MEM | 10h | ok | 11.1 | Uncorrectable ECC
GPU2_MEM | 11h | ok | 11.2 | Uncorrectable ECC
GPU3_MEM | 12h | ok | 11.3 | Uncorrectable ECC
GPU4_MEM | 13h | ok | 11.4 | Uncorrectable ECC
GPU5_MEM | 14h | ok | 11.5 | Uncorrectable ECC
GPU6_MEM | 15h | ok | 11.6 | Uncorrectable ECC
GPU7_MEM | 16h | ok | 11.7 | Uncorrectable ECC
GPU8_MEM | 17h | ok | 11.8 | Uncorrectable ECC

2. Run `ipmitool sel elist`

3. Observe the abnormal reply:
c5 | 2023/08/02 | 17时17分24秒 CST | Memory | Uncorrectable ECC | Asserted
c6 | 2023/08/02 | 17时18分29秒 CST | Memory | Uncorrectable ECC | Asserted
c7 | 2023/08/02 | 17时18分29秒 CST | Memory | Uncorrectable ECC | Asserted
c8 | 2023/08/02 | 17时18分30秒 CST | Memory | Uncorrectable ECC | Asserted
c9 | 2023/08/02 | 17时18分30秒 CST | Memory | Uncorrectable ECC | Asserted
ca | 2023/08/02 | 17时19分34秒 CST | Memory | Uncorrectable ECC | Asserted
cb | 2023/08/02 | 17时19分34秒 CST | Memory INTEGRAL_DIMM | Uncorrectable ECC | Asserted
cc | 2023/08/02 | 17时19分34秒 CST | Memory | Uncorrectable ECC | Asserted
SensorName is empty or wrong.(Expexct GPU1_MEM)

A fix was proposed upstream but is yet to be merged: https://codeberg.org/IPMITool/ipmitool/pulls/39

[ Test Plan ]

I have confirmed that this cleanly applies to the latest Plucky ipmitool and prepared a test PPA: https://launchpad.net/~mitchellaugustin/+archive/ubuntu/ipmitool-lun-sauce/

I tested for regressions when running `ipmitool sel elist` on our DGX A100, and did not observe any. (results were the same as with current plucky ipmitool)

I am asking Nvidia to confirm that this works for them, since we do not currently have the hardware to test the new functionality internally. (However, they have already confirmed that this patch, when applied to jammy ipmitool, works as expected.)

[ Fix ]

The change will add checking of SEL Generator ID byte 2 LUN bits [1:0]
in the compare with the SDR LUN field to display the correct SDR string
in the SEL event

[ Where problems could occur ]

If upstream ever does accept a different version of this patch which conflicts with our sauce, we may need to revert and apply their version. However, they have not responded to the upstream merge request in over 7 months [0][1], so since this functionality is still required by our users, and since it only adjusts a part of ipmitool to match the ipmi specification, it seems appropriate for a sauce patch.

The regression risk should be low since this just adds a check for a field that should already be present in any hardware-generated event records according to the IPMI spec[2] (section 32.1 SEL Event Records), and this check is only done/used in a function that is specifically for printing sensor-generated event records.

[0]:
Patch: https://codeberg.org/IPMITool/ipmitool/commit/719105ebb26b74f402437844ac9ea7707ff0ffb0
PR: https://codeberg.org/IPMITool/ipmitool/pulls/39

[1]: https://codeberg.org/IPMITool/ipmitool/issues/8

[2]: https://www.intel.la/content/dam/www/public/us/en/documents/specification-updates/ipmi-intelligent-platform-mgt-interface-spec-2nd-gen-v2-0-spec-update.pdf

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Upstream proposed patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Gabriel B. Sant'Anna (baioc)]

I wasn't able to test the specific case covered by this (sensor with lun1 owner), but in case anyone's interested in trying it out, I've built patched versions for Noble and Jammy in ppa:baioc/ipmitool-sru-sel-sensor-lun1
I'll attach the debdiffs here as well.

[Sergio Durigan Junior (sergiodj)]

Hi Gabriel,

Thanks for the bug report. IMHO, I believe it's best to wait for upstream's comments regarding the proposed patch before going forward with the upload. Also, it seems that one needs specific hardware to reproduce the issue...? Is there any way to reproduce it without such hardware?

Thanks.

[Gabriel B. Sant'Anna (baioc)]

Hi Sergio,

Correct, I haven't been able to reproduce it without specific hardware.
Someone pointed me to <https://github.com/wrouesnel/openipmi/tree/master/lanserv>, which contains an IPMI simulator, but after an afternoon of experiments I'm still not able to generate events with the right lun.

[Mitchell Dzurick (mitchdz)]

Thanks Gabriel. I agree with Sergio, let's wait for comment from upstream.

[Joao Andre Simioni (jasimioni)]

Early tests with the provided PPA for Jammy worked as expected. Below the report:

We've confirmed successful results with the PPA ipmitool. Previously, the SEL events for SPD_FAN_[1..4]_5 plus SPD_FAN_4_[1..4] were not properly reported. E.g. these IDs for these fans were 0x1a0-0x1a7, and only the lower byte was being used.

# ipmitool -V

ipmitool version 1.8.18

ipmitool sel elist log as below:

| 08/07/2024 | 09:41:47 | Fan SPD_FAN_1_5_R | Lower Critical going low | Asserted | Reading 0 < Threshold 2200 RPM

| 08/07/2024 | 09:41:47 | Fan SPD_FAN_1_5_F | Lower Critical going low | Asserted | Reading 0 < Threshold 2464 RPM

| 08/07/2024 | 09:42:14 | Fan SPD_FAN_1_5_R | Lower Critical going low | Deasserted | Reading 4840 < Threshold 2200 RPM

| 08/07/2024 | 09:42:14 | Fan SPD_FAN_1_5_F | Lower Critical going low | Deasserted | Reading 5720 < Threshold 2464 RPM

| 08/07/2024 | 09:42:31 | Fan SPD_FAN_2_5_R | Lower Critical going low | Asserted | Reading 0 < Threshold 2200 RPM

| 08/07/2024 | 09:42:31 | Fan SPD_FAN_2_5_F | Lower Critical going low | Asserted | Reading 0 < Threshold 2464 RPM

| 08/07/2024 | 09:43:00 | Fan SPD_FAN_2_5_R | Lower Critical going low | Deasserted | Reading 3520 < Threshold 2200 RPM

| 08/07/2024 | 09:43:00 | Fan SPD_FAN_2_5_F | Lower Critical going low | Deasserted | Reading 4224 < Threshold 2464 RPM

| 08/07/2024 | 09:43:23 | Fan SPD_FAN_3_5_R | Lower Critical going low | Asserted | Reading 0 < Threshold 2200 RPM

| 08/07/2024 | 09:43:23 | Fan SPD_FAN_3_5_F | Lower Critical going low | Asserted | Reading 0 < Threshold 2464 RPM

| 08/07/2024 | 09:43:58 | Fan SPD_FAN_3_5_R | Lower Critical going low | Deasserted | Reading 3520 < Threshold 2200 RPM

| 08/07/2024 | 09:44:03 | Fan SPD_FAN_3_5_F | Lower Critical going low | Deasserted | Reading 5632 < Threshold 2464 RPM

| 08/07/2024 | 09:44:14 | Fan SPD_FAN_4_1_R | Lower Critical going low | Asserted | Reading 0 < Threshold 2200 RPM

| 08/07/2024 | 09:44:16 | Fan SPD_FAN_4_1_F | Lower Critical going low | Asserted | Reading 0 < Threshold 2464 RPM

| 08/07/2024 | 09:44:37 | Fan SPD_FAN_4_1_R | Lower Critical going low | Deasserted | Reading 3608 < Threshold 2200 RPM

| 08/07/2024 | 09:44:37 | Fan SPD_FAN_4_1_F | Lower Critical going low | Deasserted | Reading 4312 < Threshold 2464 RPM

| 08/07/2024 | 09:44:59 | Fan SPD_FAN_4_2_R | Lower Critical going low | Asserted | Reading 0 < Threshold 2200 RPM

| 08/07/2024 | 09:45:00 | Fan SPD_FAN_4_2_F | Lower Critical going low | Asserted | Reading 0 < Threshold 2464 RPM

| 08/07/2024 | 09:45:34 | Fan SPD_FAN_4_2_R | Lower Critical going low | Deasserted | Reading 3520 < Threshold 2200 RPM

| 08/07/2024 | 09:45:34 | Fan SPD_FAN_4_2_F | Lower Critical going low | Deasserted | Reading 4136 < Threshold 2464 RPM

| 08/07/2024 | 09:45:49 | Fan SPD_FAN_4_3_R | Lower Critical going low | Asserted | Reading 0 < Threshold 2200 RPM

| 08/07/2024 | 09:45:49 | Fan SPD_FAN_4_3_F | Lower Critical going low | Asserted | Reading 0 < Threshold 2464 RPM

| 08/07/2024 | 09:46:09 | Fan SPD_FAN_4_3_R | Lower Critical going low | Deasserted | Reading 3168 < Threshold 2200 RPM

| 08/07/2024 | 09:46:09 | Fan SPD_FAN_4_3_F | Lower Critical going low | D...

[Launchpad Janitor (janitor)]

[Expired for ipmitool (Ubuntu) because there has been no activity for 60 days.]

[Mitchell Augustin (mitchellaugustin)]

MP submitted: https://code.launchpad.net/~mitchellaugustin/ubuntu/+source/ipmitool/+git/ipmitool/+merge/484252

[Launchpad Janitor (janitor)]

This bug was fixed in the package ipmitool - 1.8.19-7.1ubuntu1

---------------
ipmitool (1.8.19-7.1ubuntu1) questing; urgency=medium

  * d/patches: Apply sauce patch not addressed by upstream
    + sel/sdr : Fix SEL cannot display Sensor name (lp: #2076173)

 -- Mitchell Augustin <email address hidden> Wed, 30 Apr 2025 15:43:00 -0500

[Timo Aaltonen (tjaalton)]

Hello Gabriel, or anyone else affected,

Accepted ipmitool into plucky-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ipmitool/1.8.19-7.1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-plucky to verification-done-plucky. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-plucky. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mitchell Augustin (mitchellaugustin)]

Nvidia QA verified that this patch resolves their issue as expected, and I did not observe any `ipmitool sel elist` regressions on my DUT, so I have marked it as verified. Thanks!

[Andreas Hasenack (ahasenack)]

This was inadvertently removed[1] from plucky-proposed during the opening of the questing archive.

I'll try to restore it:

$ ./copy-package --include-binaries -s plucky-proposed --force-same-destination --auto-approve -e 1.8.19-7.1ubuntu0.1 ipmitool
Copy candidates:
 ipmitool 1.8.19-7.1ubuntu0.1 in plucky
Candidate copy target: https://api.launchpad.net/devel/ubuntu/+archive/primary, plucky, proposed
Copy [y|N]? y
1 copy requested.

1. https://launchpad.net/ubuntu/+source/ipmitool/1.8.19-7.1ubuntu0.1/+publishinghistory

[Launchpad Janitor (janitor)]

This bug was fixed in the package ipmitool - 1.8.19-7.1ubuntu0.1

---------------
ipmitool (1.8.19-7.1ubuntu0.1) plucky; urgency=medium

  * d/patches: Apply sauce patch not addressed by upstream
    + sel/sdr : Fix SEL cannot display Sensor name (lp: #2076173)

 -- Mitchell Augustin <email address hidden> Thu, 1 May 2025 12:04:00 -0500

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for ipmitool has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Mitchell Augustin (mitchellaugustin)]

Hi,
I have linked the branches for the remaining series and tagged ubuntu-sponsors to request an upload/sponsorship.
Nvidia has verified that PPAs for each work to resolve the issue (which requires specialized hardware), and I have not observed any regressions in `ipmitool sel elist`.

Oracular PPA: https://launchpad.net/~mitchellaugustin/+archive/ubuntu/ipmitool-patch-oracular
Noble PPA: https://launchpad.net/~mitchellaugustin/+archive/ubuntu/ipmitool-patch-noble-2
Jammy PPA: https://launchpad.net/~mitchellaugustin/+archive/ubuntu/ipmitool-patch-jammy

[Timo Aaltonen (tjaalton)]

Hello Gabriel, or anyone else affected,

Accepted ipmitool into oracular-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ipmitool/1.8.19-7ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-oracular to verification-done-oracular. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-oracular. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Gabriel, or anyone else affected,

Accepted ipmitool into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ipmitool/1.8.19-7ubuntu0.24.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Gabriel, or anyone else affected,

Accepted ipmitool into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ipmitool/1.8.18-11ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ipmitool (1.8.19-7ubuntu1.1) for oracular have finished running.
The following regressions have been reported in tests triggered by the package:

ceilometer/2:23.0.0-0ubuntu2 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/oracular/update_excuses.html#ipmitool

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ipmitool (1.8.19-7ubuntu0.24.04.2) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

ceilometer/2:22.0.0-0ubuntu1.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#ipmitool

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Mitchell Augustin (mitchellaugustin)]

Regarding the amd64/noble and s390x/oracular ceilometer autopkgtest regression reports:
They both look to be failing due to ceilometer-agent-compute here:
136s A dependency job for ceilometer-agent-compute.service failed. See 'journalctl -xe' for details.
136s autopkgtest [22:06:24]: test test-services: -----------------------]
137s test-services FAIL non-zero exit status 1

However, the ceilometer-agent-compute package produced by the ceilometer source package doesn't itself depend on ipmitool (only ceilometer-agent-ipmi depends on ipmitool), so I believe this autopkgtest failure is a false positive, not a real regression that would have been introduced by the ipmitool update.

Additionally:
For s390x, the test-services autopkgtest indicates [0] that s390x is not a relevant architecture for the ipmi agent service, so we should be able to safely ignore that failure.
For amd64, I installed ceilometer-agent-ipmi and the proposed ipmitool package on an amd64 machine and verified that the service started successfully.

Additionally, I have verified that I see the same behavior for `ipmitool sel elist` with the -proposed packages as I do with the current versions in j,n,o, and Nvidia has verified via the builds from the same source packages that the new hardware functionality introduced by this patch works as expected.

With all of this said, I'm marking these as verified. Please let me know if you disagree with my autopkgtest assessments.

[0] https://git.launchpad.net/ubuntu/+source/ceilometer/tree/debian/tests/test-services?h=applied/ubuntu/oracular-devel

[Mitchell Augustin (mitchellaugustin)]

I ran the migration-reference/0 autopkgtest trigger on both impacted tests, and they failed in the same way as well:

https://autopkgtest.ubuntu.com/packages/c/ceilometer/noble/amd64
https://autopkgtest.ubuntu.com/packages/c/ceilometer/oracular/s390x

So I think we can conclude this failure is unrelated to my changes

[Launchpad Janitor (janitor)]

This bug was fixed in the package ipmitool - 1.8.19-7ubuntu1.1

---------------
ipmitool (1.8.19-7ubuntu1.1) oracular; urgency=medium

  * d/patches: Apply sauce patch not addressed by upstream
    + sel/sdr : Fix SEL cannot display Sensor name (lp: #2076173)

 -- Mitchell Augustin <email address hidden> Fri, 2 May 2025 17:45:00 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package ipmitool - 1.8.19-7ubuntu0.24.04.2

---------------
ipmitool (1.8.19-7ubuntu0.24.04.2) noble; urgency=medium

  * d/patches: Apply sauce patch not addressed by upstream
    + sel/sdr : Fix SEL cannot display Sensor name (lp: #2076173)

 -- Mitchell Augustin <email address hidden> Fri, 2 May 2025 17:56:00 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package ipmitool - 1.8.18-11ubuntu2.2

---------------
ipmitool (1.8.18-11ubuntu2.2) jammy; urgency=medium

  * d/patches: Apply sauce patch not addressed by upstream
    + sel/sdr : Fix SEL cannot display Sensor name (lp: #2076173)

 -- Mitchell Augustin <email address hidden> Fri, 2 May 2025 18:05:00 -0500