ipmi-locate crash on arm64

Bug #1699933 reported by Ike Panhc on 2017-06-23
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
freeipmi (Debian)
Confirmed
Unknown
freeipmi (Ubuntu)
Undecided
dann frazier
Xenial
Undecided
Ike Panhc
Yakkety
Undecided
Ike Panhc
Zesty
Undecided
Ike Panhc
Artful
Undecided
dann frazier

Bug Description

[Impact]

 * Read from /dev/mem and scan DMI tables is dangerous, if /dev/mem is not stable to read, it will cause ipmi-locate crash.

[Test Case]

 * `sudo ipmi-locate` shall not crash

[Regression Potential]

 * Reading from DMI tables is sysfs is more stable then reading from /dev/mem and if DMI tables is not available ipmi-locate will use the old way to search in /dev/mem. I believe its low regression risk

----
Similiar to bug 1499838 but not 100% same.

ipmi-locate looks into /dev/mem for DMI tables and crash when DMI tables are not in accessible memory region. Kernel > v4.2 provides /sys/firmware/DMI/tables/DMI which is a much safer place to read.

Ike Panhc (ikepanhc) on 2017-06-23
Changed in freeipmi (Ubuntu):
assignee: nobody → Ike Panhc (ikepanhc)

Thanks Ike for the report, you also assigned yourself so I wanted to ask - are you going ahead and work on a fix that you'll report here?

Ike Panhc (ikepanhc) wrote :

@paelzer,

Yes, I have a fix already and plan to propose to upstream for feedback.

Ike Panhc (ikepanhc) wrote :
Changed in freeipmi (Debian):
status: Unknown → Confirmed
dann frazier (dannf) on 2017-06-28
Changed in freeipmi (Ubuntu Xenial):
assignee: nobody → Ike Panhc (ikepanhc)
Changed in freeipmi (Ubuntu Yakkety):
assignee: nobody → Ike Panhc (ikepanhc)
Changed in freeipmi (Ubuntu Zesty):
assignee: nobody → Ike Panhc (ikepanhc)
Changed in freeipmi (Ubuntu Artful):
assignee: Ike Panhc (ikepanhc) → dann frazier (dannf)
Changed in freeipmi (Ubuntu Xenial):
status: New → Triaged
Changed in freeipmi (Ubuntu Yakkety):
status: New → Triaged
Changed in freeipmi (Ubuntu Zesty):
status: New → Triaged
Changed in freeipmi (Ubuntu Artful):
status: New → In Progress
Ike Panhc (ikepanhc) on 2017-06-29
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package freeipmi - 1.4.11-1.1ubuntu4

---------------
freeipmi (1.4.11-1.1ubuntu4) artful; urgency=medium

  * Use kernel DMI table interface, when available, instead of trolling
    /dev/mem to avoid a crash on ARM. LP: #1699933.

 -- dann frazier <email address hidden> Wed, 28 Jun 2017 13:26:09 -0600

Changed in freeipmi (Ubuntu Artful):
status: In Progress → Fix Released
Ike Panhc (ikepanhc) on 2017-06-30
Changed in freeipmi (Ubuntu Zesty):
status: Triaged → In Progress
Changed in freeipmi (Ubuntu Yakkety):
status: Triaged → In Progress
Changed in freeipmi (Ubuntu Xenial):
status: Triaged → In Progress

Hello Ike, or anyone else affected,

Accepted freeipmi into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.4.11-1.1ubuntu4~0.16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in freeipmi (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Changed in freeipmi (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed-zesty
Łukasz Zemczak (sil2100) wrote :

Hello Ike, or anyone else affected,

Accepted freeipmi into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.4.11-1.1ubuntu4~0.17.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in freeipmi (Ubuntu Yakkety):
status: In Progress → Fix Committed
tags: added: verification-needed-yakkety
Łukasz Zemczak (sil2100) wrote :

Hello Ike, or anyone else affected,

Accepted freeipmi into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.4.11-1.1ubuntu4~0.16.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-yakkety to verification-done-yakkety. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-yakkety. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Ike Panhc (ikepanhc) wrote :

ubuntu@d05-4:~$ sudo ipmi-locate
[68526.721538] pgd = ffff801fcbfbb000
[68526.724948] [ffffa5ce0c36] *pgd=0000001fd272c003, *pud=0000001fc3da0003, *pmd=0000001fd2851003, *pte=0160000039860fc3
Bus error (core dumped)
ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu3~0.16.04 arm64 GNU implementation of the IPMI protocol - tools

ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.16.04 arm64 GNU implementation of the IPMI protocol - tools
ubuntu@d05-4:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... FAILED

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: BT
BMC driver device:
BMC I/O base address: 0xE4
Register spacing: 1

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... FAILED

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ubuntu@d05-4:~$

tags: added: verification-done-xenial
removed: verification-needed-xenial
Ike Panhc (ikepanhc) wrote :

ubuntu@d05-4:~$ sudo ipmi-locate
[70950.778753] pgd = ffff8417cbcae000
[70950.782162] [ffff8b8f0c36] *pgd=00000417bf0e9003, *pud=00000417bf959003, *pmd=00000417bf5fa003, *pte=0160000039860fc3
Bus error (core dumped)
ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu3~0.16.10 arm64 GNU implementation of the IPMI protocol - tools

ubuntu@d05-4:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... FAILED

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: BT
BMC driver device:
BMC I/O base address: 0xE4
Register spacing: 1

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... FAILED

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.16.10 arm64 GNU implementation of the IPMI protocol - tools

tags: added: verification-done-yakkety
removed: verification-needed-yakkety
Ike Panhc (ikepanhc) wrote :

ubuntu@d05-4:~$ sudo ipmi-locate
[71285.659835] pgd = ffff801fbe0b3000
[71285.663245] [ffff93687c36] *pgd=0000001fd16f6003, *pud=0000001fbecab003, *pmd=0000001fbed7f003, *pte=0160000039860fc3
Bus error (core dumped)
ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu3 arm64 GNU implementation of the IPMI protocol - tools
ubuntu@d05-4:~$

ubuntu@d05-4:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... FAILED

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: BT
BMC driver device:
BMC I/O base address: 0xE4
Register spacing: 1

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... FAILED

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.17.04 arm64 GNU implementation of the IPMI protocol - tools

tags: added: verification-done verification-done-zesty
removed: verification-needed verification-needed-zesty
Robie Basak (racb) wrote :

Thank you for your comprehensive verification report.

Has there been any testing performed to verify that ipmi-locate still works correctly now that its data source has been changed? I think we need to do at least some level of testing of this before it's safe to recommend this update to users.

Robie Basak (racb) wrote :

And presumably this testing should take place on more than just arm64 as well?

dann frazier (dannf) wrote :
Download full text (5.6 KiB)

I verified this on an x86 server by testing that the output is the same before and after the update (see below). I also gave a heads-up to the MAAS team to consider adding this to their CI (see: https://bugs.launchpad.net/maas-images/+bug/1702976/comments/11), though I don't know if that occurred. @Andres: was your team able to run any tests with this?

dannf@r815:~$ apt policy freeipmi-tools
freeipmi-tools:
  Installed: 1.4.11-1.1ubuntu3~0.16.04
  Candidate: 1.4.11-1.1ubuntu4~0.16.04
  Version table:
     1.4.11-1.1ubuntu4~0.16.04 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
 *** 1.4.11-1.1ubuntu3~0.16.04 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.11-1ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
dannf@r815:~$ sudo ipmi-locate 2>&1 | tee /tmp/ipmi-locate.xenial
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA8
Register spacing: 4

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... done
IPMI Version: 2.0
IPMI locate driver: SMBIOS
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA8
Register spacing: 4

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

dannf@r815:~$ sudo apt install freeipmi-tools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  freeipmi-common libfreeipmi16 libipmiconsole2 libipmidetect0
Suggested packages:
  freeipmi-ipmidetect freeipmi-bmc-watchdog
The following packages will be upgraded:
  freeipmi-common freeipmi-tools libfreeipmi16 libipmiconsole2 libipmidetect0
5 upgraded, 0 newly installed, 0 to remove and 261 not upgraded.
Need to get 1,735 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libfreeipmi16 amd64 1.4.11-1.1ubuntu4~0.16.04 [827 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libipmiconsole2 amd64 1.4.11-1.1ubuntu4~0.16.04 [85.1 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libipmidetect0 amd64 1.4.11-1.1ubuntu4~0.16.04 [25.3 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 freeipmi-tools amd64 1.4.11-1.1ubuntu4~0.16.04 [623 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 freeipmi-common amd64 1.4.11-1.1ubuntu4~0.16.04 [174 kB]
Fetched 1,735 kB in 0s (21.2 MB/s)
(...

Read more...

Ike Panhc (ikepanhc) wrote :

Same here. On x86 server, freeipmi-tools in -update and -proposed has the same output

xenial-proposed
ikepanhc@keiko:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... done
IPMI Version: 2.0
IPMI locate driver: SMBIOS
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ikepanhc@keiko:~$ sudo dpkg --list | grep freeipmi
ii freeipmi-common 1.4.11-1.1ubuntu4~0.16.04 amd64 GNU implementation of the IPMI protocol - common files
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.16.04 amd64 GNU implementation of the IPMI protocol - tools
ii libfreeipmi16 1.4.11-1.1ubuntu4~0.16.04 amd64 GNU IPMI - libraries

xenial-update
Setting up freeipmi-tools (1.4.11-1.1ubuntu3~0.16.04) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...
ikepanhc@keiko:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... done
IPMI Version: 2.0
IPMI locate driver: SMBIOS
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

Robie Basak (racb) wrote :

FTR, Dann asked me about what I was looking for. I said I wanted testing of some representative sample of hardware in the wild. I'm open to suggestions on what that might mean, but I suggested testing against a couple of different manufacturers' BMCs, for example Dell and HP. Dann will ask the MAAS team if they can help with this.

Robie Basak (racb) wrote :

> Dann

Sorry, I mean dann :)

[ + Andres, for MAAS ]

On Thu, Jul 20, 2017 at 9:40 AM, Robie Basak <email address hidden> wrote:
> FTR, Dann asked me about what I was looking for. I said I wanted testing
> of some representative sample of hardware in the wild. I'm open to
> suggestions on what that might mean, but I suggested testing against a
> couple of different manufacturers' BMCs, for example Dell and HP.

Thanks Robie! The above r815 system is a Dell PowerEdge r815, so we've
got one Dell system covered.
I also found an HP ProLiant m720 in our lab and tested there w/ good
results (attached).

I tested with both 4.4 and 4.10 kernels, since the /sys/firmware/dmi
interface wasn't yet present in 4.4.

> Dann will ask the MAAS team if they can help with this.

Andres: Please let us know if you consider the above testing
sufficient, or if you'd like to do further testing in MAAS CI.

  -dann

Brian Murray (brian-murray) wrote :

With regards to comment #13 and testing that the update still works on non-arm64 systems - I don't see any indication of this testing happening for zesty.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package freeipmi - 1.4.11-1.1ubuntu4~0.16.10

---------------
freeipmi (1.4.11-1.1ubuntu4~0.16.10) yakkety; urgency=medium

  * Backport to yakkety
    Use kernel DMI table interface, when available, instead of trolling
    /dev/mem to avoid a crash on ARM. LP: #1699933.

 -- Ike Panhc <email address hidden> Fri, 30 Jun 2017 11:01:05 +0800

Changed in freeipmi (Ubuntu Yakkety):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for freeipmi has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Ike Panhc (ikepanhc) wrote :

On zesty ipmi-locate works well as expected. Actually in zesty and xenial are same source.

ubuntu@zesty:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... FAILED

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ubuntu@zesty:~$ sudo dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.17.04 amd64 GNU implementation of the IPMI protocol - tools

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package freeipmi - 1.4.11-1.1ubuntu4~0.16.04

---------------
freeipmi (1.4.11-1.1ubuntu4~0.16.04) xenial; urgency=medium

  * Backport to xenial
    Use kernel DMI table interface, when available, instead of trolling
    /dev/mem to avoid a crash on ARM. LP: #1699933.

 -- Ike Panhc <email address hidden> Fri, 30 Jun 2017 11:01:05 +0800

Changed in freeipmi (Ubuntu Xenial):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package freeipmi - 1.4.11-1.1ubuntu4~0.17.04

---------------
freeipmi (1.4.11-1.1ubuntu4~0.17.04) zesty; urgency=medium

  * Backport to zesty
    Use kernel DMI table interface, when available, instead of trolling
    /dev/mem to avoid a crash on ARM. LP: #1699933.

 -- Ike Panhc <email address hidden> Fri, 30 Jun 2017 11:01:05 +0800

Changed in freeipmi (Ubuntu Zesty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.