ipmi-locate crash on arm64

[Ike Panhc]

[Impact]

 * Read from /dev/mem and scan DMI tables is dangerous, if /dev/mem is not stable to read, it will cause ipmi-locate crash.

[Test Case]

 * `sudo ipmi-locate` shall not crash

[Regression Potential]

 * Reading from DMI tables is sysfs is more stable then reading from /dev/mem and if DMI tables is not available ipmi-locate will use the old way to search in /dev/mem. I believe its low regression risk

----
Similiar to bug 1499838 but not 100% same.

ipmi-locate looks into /dev/mem for DMI tables and crash when DMI tables are not in accessible memory region. Kernel > v4.2 provides /sys/firmware/DMI/tables/DMI which is a much safer place to read.

[Christian Ehrhardt  (paelzer)]

Thanks Ike for the report, you also assigned yourself so I wanted to ask - are you going ahead and work on a fix that you'll report here?

[Ike Panhc (ikepanhc)]

@paelzer,

Yes, I have a fix already and plan to propose to upstream for feedback.

[Ike Panhc (ikepanhc)]

http://lists.gnu.org/archive/html/freeipmi-devel/2017-06/msg00000.html

Patch on upstream mailing list

[Ike Panhc (ikepanhc)]

http://lists.gnu.org/archive/html/freeipmi-devel/2017-06/msg00001.html

Patch reviewed and get positive feedback

[Launchpad Janitor (janitor)]

This bug was fixed in the package freeipmi - 1.4.11-1.1ubuntu4

---------------
freeipmi (1.4.11-1.1ubuntu4) artful; urgency=medium

  * Use kernel DMI table interface, when available, instead of trolling
    /dev/mem to avoid a crash on ARM. LP: #1699933.

 -- dann frazier <email address hidden> Wed, 28 Jun 2017 13:26:09 -0600

[Łukasz Zemczak (sil2100)]

Hello Ike, or anyone else affected,

Accepted freeipmi into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.4.11-1.1ubuntu4~0.16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Ike, or anyone else affected,

Accepted freeipmi into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.4.11-1.1ubuntu4~0.17.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Ike, or anyone else affected,

Accepted freeipmi into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/freeipmi/1.4.11-1.1ubuntu4~0.16.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-yakkety to verification-done-yakkety. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-yakkety. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ike Panhc (ikepanhc)]

ubuntu@d05-4:~$ sudo ipmi-locate
[68526.721538] pgd = ffff801fcbfbb000
[68526.724948] [ffffa5ce0c36] *pgd=0000001fd272c003, *pud=0000001fc3da0003, *pmd=0000001fd2851003, *pte=0160000039860fc3
Bus error (core dumped)
ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu3~0.16.04 arm64 GNU implementation of the IPMI protocol - tools

ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.16.04 arm64 GNU implementation of the IPMI protocol - tools
ubuntu@d05-4:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... FAILED

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: BT
BMC driver device:
BMC I/O base address: 0xE4
Register spacing: 1

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... FAILED

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ubuntu@d05-4:~$

[Ike Panhc (ikepanhc)]

ubuntu@d05-4:~$ sudo ipmi-locate
[70950.778753] pgd = ffff8417cbcae000
[70950.782162] [ffff8b8f0c36] *pgd=00000417bf0e9003, *pud=00000417bf959003, *pmd=00000417bf5fa003, *pte=0160000039860fc3
Bus error (core dumped)
ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu3~0.16.10 arm64 GNU implementation of the IPMI protocol - tools

ubuntu@d05-4:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... FAILED

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: BT
BMC driver device:
BMC I/O base address: 0xE4
Register spacing: 1

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... FAILED

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.16.10 arm64 GNU implementation of the IPMI protocol - tools

[Ike Panhc (ikepanhc)]

ubuntu@d05-4:~$ sudo ipmi-locate
[71285.659835] pgd = ffff801fbe0b3000
[71285.663245] [ffff93687c36] *pgd=0000001fd16f6003, *pud=0000001fbecab003, *pmd=0000001fbed7f003, *pte=0160000039860fc3
Bus error (core dumped)
ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu3 arm64 GNU implementation of the IPMI protocol - tools
ubuntu@d05-4:~$

ubuntu@d05-4:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... FAILED

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: BT
BMC driver device:
BMC I/O base address: 0xE4
Register spacing: 1

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... FAILED

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ubuntu@d05-4:~$ dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.17.04 arm64 GNU implementation of the IPMI protocol - tools

[Robie Basak (racb)]

Thank you for your comprehensive verification report.

Has there been any testing performed to verify that ipmi-locate still works correctly now that its data source has been changed? I think we need to do at least some level of testing of this before it's safe to recommend this update to users.

[Robie Basak (racb)]

And presumably this testing should take place on more than just arm64 as well?

[dann frazier (dannf)]

I verified this on an x86 server by testing that the output is the same before and after the update (see below). I also gave a heads-up to the MAAS team to consider adding this to their CI (see: https://bugs.launchpad.net/maas-images/+bug/1702976/comments/11), though I don't know if that occurred. @Andres: was your team able to run any tests with this?

dannf@r815:~$ apt policy freeipmi-tools
freeipmi-tools:
  Installed: 1.4.11-1.1ubuntu3~0.16.04
  Candidate: 1.4.11-1.1ubuntu4~0.16.04
  Version table:
     1.4.11-1.1ubuntu4~0.16.04 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
 *** 1.4.11-1.1ubuntu3~0.16.04 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.11-1ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
dannf@r815:~$ sudo ipmi-locate 2>&1 | tee /tmp/ipmi-locate.xenial
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA8
Register spacing: 4

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... done
IPMI Version: 2.0
IPMI locate driver: SMBIOS
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA8
Register spacing: 4

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

dannf@r815:~$ sudo apt install freeipmi-tools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  freeipmi-common libfreeipmi16 libipmiconsole2 libipmidetect0
Suggested packages:
  freeipmi-ipmidetect freeipmi-bmc-watchdog
The following packages will be upgraded:
  freeipmi-common freeipmi-tools libfreeipmi16 libipmiconsole2 libipmidetect0
5 upgraded, 0 newly installed, 0 to remove and 261 not upgraded.
Need to get 1,735 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libfreeipmi16 amd64 1.4.11-1.1ubuntu4~0.16.04 [827 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libipmiconsole2 amd64 1.4.11-1.1ubuntu4~0.16.04 [85.1 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libipmidetect0 amd64 1.4.11-1.1ubuntu4~0.16.04 [25.3 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 freeipmi-tools amd64 1.4.11-1.1ubuntu4~0.16.04 [623 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 freeipmi-common amd64 1.4.11-1.1ubuntu4~0.16.04 [174 kB]
Fetched 1,735 kB in 0s (21.2 MB/s)
(...

[Ike Panhc (ikepanhc)]

Same here. On x86 server, freeipmi-tools in -update and -proposed has the same output

xenial-proposed
ikepanhc@keiko:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... done
IPMI Version: 2.0
IPMI locate driver: SMBIOS
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ikepanhc@keiko:~$ sudo dpkg --list | grep freeipmi
ii freeipmi-common 1.4.11-1.1ubuntu4~0.16.04 amd64 GNU implementation of the IPMI protocol - common files
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.16.04 amd64 GNU implementation of the IPMI protocol - tools
ii libfreeipmi16 1.4.11-1.1ubuntu4~0.16.04 amd64 GNU IPMI - libraries

xenial-update
Setting up freeipmi-tools (1.4.11-1.1ubuntu3~0.16.04) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...
ikepanhc@keiko:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... done
IPMI Version: 2.0
IPMI locate driver: SMBIOS
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

[Robie Basak (racb)]

FTR, Dann asked me about what I was looking for. I said I wanted testing of some representative sample of hardware in the wild. I'm open to suggestions on what that might mean, but I suggested testing against a couple of different manufacturers' BMCs, for example Dell and HP. Dann will ask the MAAS team if they can help with this.

[Robie Basak (racb)]

> Dann

Sorry, I mean dann :)

[dann frazier (dannf)]

[ + Andres, for MAAS ]

On Thu, Jul 20, 2017 at 9:40 AM, Robie Basak <email address hidden> wrote:
> FTR, Dann asked me about what I was looking for. I said I wanted testing
> of some representative sample of hardware in the wild. I'm open to
> suggestions on what that might mean, but I suggested testing against a
> couple of different manufacturers' BMCs, for example Dell and HP.

Thanks Robie! The above r815 system is a Dell PowerEdge r815, so we've
got one Dell system covered.
I also found an HP ProLiant m720 in our lab and tested there w/ good
results (attached).

I tested with both 4.4 and 4.10 kernels, since the /sys/firmware/dmi
interface wasn't yet present in 4.4.

> Dann will ask the MAAS team if they can help with this.

Andres: Please let us know if you consider the above testing
sufficient, or if you'd like to do further testing in MAAS CI.

  -dann

[Brian Murray (brian-murray)]

With regards to comment #13 and testing that the update still works on non-arm64 systems - I don't see any indication of this testing happening for zesty.

[Launchpad Janitor (janitor)]

This bug was fixed in the package freeipmi - 1.4.11-1.1ubuntu4~0.16.10

---------------
freeipmi (1.4.11-1.1ubuntu4~0.16.10) yakkety; urgency=medium

  * Backport to yakkety
    Use kernel DMI table interface, when available, instead of trolling
    /dev/mem to avoid a crash on ARM. LP: #1699933.

 -- Ike Panhc <email address hidden> Fri, 30 Jun 2017 11:01:05 +0800

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for freeipmi has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Ike Panhc (ikepanhc)]

On zesty ipmi-locate works well as expected. Actually in zesty and xenial are same source.

ubuntu@zesty:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA2
Register spacing: 1

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... FAILED

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

ubuntu@zesty:~$ sudo dpkg --list | grep freeipmi-tools
ii freeipmi-tools 1.4.11-1.1ubuntu4~0.17.04 amd64 GNU implementation of the IPMI protocol - tools

[Launchpad Janitor (janitor)]

This bug was fixed in the package freeipmi - 1.4.11-1.1ubuntu4~0.16.04

---------------
freeipmi (1.4.11-1.1ubuntu4~0.16.04) xenial; urgency=medium

  * Backport to xenial
    Use kernel DMI table interface, when available, instead of trolling
    /dev/mem to avoid a crash on ARM. LP: #1699933.

 -- Ike Panhc <email address hidden> Fri, 30 Jun 2017 11:01:05 +0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package freeipmi - 1.4.11-1.1ubuntu4~0.17.04

---------------
freeipmi (1.4.11-1.1ubuntu4~0.17.04) zesty; urgency=medium

  * Backport to zesty
    Use kernel DMI table interface, when available, instead of trolling
    /dev/mem to avoid a crash on ARM. LP: #1699933.

 -- Ike Panhc <email address hidden> Fri, 30 Jun 2017 11:01:05 +0800