(regression?) openipmi fails to start on kernels 3.13.0-11-generic and 3.13.0-12-generic

Bug #1284334 reported by Kent Baxley on 2014-02-24
30
This bug affects 5 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
High
Unassigned

Bug Description

When updating 14.04 to 3.13.0-12 I noticed that the openipmi service failed to start on reboot. After running 'sudo service openipmi start', the service would still fail to start because it failed to load the ipmi modules.

Kernel 3.13.0-8, which was previously on the system, works fine.

I also updated to linux-image 3.14.0-031400rc4-generic from the mainline site and the openipmi service starts working again.

Steps to reproduce:

Install 14.04 server, amd64 version. Make sure kernel 3.13.0-11 or -12 are installed
Install openipmi.
Try to start the openipmi service (sudo service ipmi start)

Actual results:
openipmi fails to load the ipmi modules and the service fails to start

Expected results:
openipmi starts without problems.

This is an important issue because tools like ipmitool and Dell's iDRAC service module for Ubuntu rely on a working openipmi in order to function properly. The 3.14 kernels seem to have a fix of some sort in them.

Kent Baxley (kentb) wrote :

This is the kernel that worked for me:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14-rc4-trusty/

I can also go back and test the others until I find one that breaks.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1284334

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Kent Baxley (kentb) wrote :

I've uploaded the .apport file, since the server can't get out to the internet very easily. Let me know if you need anything else.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Changed in linux (Ubuntu):
importance: Undecided → High
tags: added: kernel-key trusty
Changed in linux (Ubuntu):
status: Confirmed → Triaged
Joseph Salisbury (jsalisbury) wrote :

The Trusty 3.13.0-12 kernel has been rebased to upstream 3.13.4. However, 3.13.5 is already available. Can you see if this bug is already fixed in 3.13.5:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13.5-trusty/

Joseph Salisbury (jsalisbury) wrote :

Can you provide the apport output from 3.13.0-8 for comparison?

The ipmi_si driver is built into the kernel in 3.13.0-11 and -12. It could be the driver gets started but user-space expects it always to be a module.

Kent Baxley (kentb) wrote :

Here's the apport when booted to 3.13.0-8. I'm wondering if that is indeed the case. I saw a changelog for the -11 kernel stating IPMI modules were set to 'y'.

Kent Baxley (kentb) wrote :

Also just tested the 3.13.5 from mainline and noticed that openipmi does not start there, either. Same symptoms.

Kent Baxley (kentb) wrote :

Any updates on this?

Joseph Salisbury (jsalisbury) wrote :

Yes, you are correct. The latest Saucy kernel does have IPMI as a module. Here is the config entries for Saucy 3.11.0-18.32:

config.common.ubuntu:CONFIG_ACPI_IPMI=m
config.common.ubuntu:CONFIG_IPMI_DEVICE_INTERFACE=m
config.common.ubuntu:CONFIG_IPMI_HANDLER=m
config.common.ubuntu:# CONFIG_IPMI_PANIC_EVENT is not set
config.common.ubuntu:CONFIG_IPMI_POWEROFF=m
config.common.ubuntu:CONFIG_IPMI_SI=m
config.common.ubuntu:CONFIG_IPMI_WATCHDOG=m

Since you confirmed this is fixed in 3.14-rc4, it is probably best for us to perform a reverse bisect to identify the commit that fixes this.

Can you test the v3.14-rc1 kernel to see if that's when this was fixed? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14-rc1-trusty/

If the bug still exists there, we should test rc2, then rc3, etc. Once we find the last bad version and first good version, we can bisect between those two versions.

Joseph Salisbury (jsalisbury) wrote :

In trusty, the CONFIG_IPMI_HANDLER option is set to Y:

config.common.ubuntu:CONFIG_ACPI_IPMI=m
config.common.ubuntu:CONFIG_IPMI_DEVICE_INTERFACE=m
config.common.ubuntu:CONFIG_IPMI_HANDLER=y
config.common.ubuntu:# CONFIG_IPMI_PANIC_EVENT is not set
config.common.ubuntu:CONFIG_IPMI_POWEROFF=m
config.common.ubuntu:CONFIG_IPMI_SI=m
config.common.ubuntu:CONFIG_IPMI_WATCHDOG=m

I can build a Trusty kernel with CONFIG_IPMI_HANDLER set back to m and see if that changes things.

Kent Baxley (kentb) wrote :

Hi Joseph,

Thanks. I can check the 3.14rc1 kernel and see. I'll also be glad to try out a test kernel from you with the handler set back to module.

I'll let you know as soon as I find something out with 3.14rc1

Joseph Salisbury (jsalisbury) wrote :

I built a Trusty test kernel with CONFIG_IPMI_HANDLER=m. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1284334/

Can you test this kernel to see if it exhibits the bug or not? You will need to install both the linux-image and linux-image-extra .deb packages.

Joseph Salisbury (jsalisbury) wrote :

Also, you can confirm the IPMI config options on a running system by executing:

cat /boot/config-$(uname -r) | grep IPMI

Kent Baxley (kentb) wrote :

Trusty 3.13.0-14-generic config shows IPMI_HANDLER option set to Y:

CONFIG_ACPI_IPMI=m
CONFIG_IPMI_HANDLER=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m

..with the 3.13.0-8 kernel it is set to "m", and this was the last kernel that I ran where openipmi didn't have issues

Joseph's test kernel shows this and it works as expected with openipmi in that the service starts properly. IPMI_HANDLER is set to 'm':

CONFIG_ACPI_IPMI=m
CONFIG_IPMI_HANDLER=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m

I will test rc1 tomorrow as I ran out of time today.

Joseph Salisbury (jsalisbury) wrote :

It sounds like this bug didn't exists in Ubuntu 3.13.0-8. That kernel was based off of upstream 3.13.2. That may indicate that this bug was introduced by upstream stable 3.13.3 or 3.13.4. If you have time, can you also test the following two kernels:

3.13.3: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13.3-trusty/
3.13.4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13.4-trusty/

If 3.13.3 has the bug, there is no need to test 3.13.4. However, the priority should be on the reverse bisect, so we can identify what fixes this in 3.14.

Joseph Salisbury (jsalisbury) wrote :

All of the mainline kernels are built with CONFIG_IPMI_HANDLER set to m, so I assume testing will not reproduce the bug for any of the upstream kernels. We now know that the change of CONFIG_IPMI_HANDLER from m to y in Trusty is what causes this bug.

Kent Baxley (kentb) wrote :

Ok. I'll not test those if that's the case.

tags: added: kernel-da-key
removed: kernel-key

CONFIG_IPMI_HANDLER is still y in3.13.0-20, so this is still broken. This is the workaround I'm using at the moment:

# diff /etc/init.d/openipmi.orig /etc/init.d/openipmi
324,326c324,326
< modprobe ipmi_msghandler > /dev/null 2>&1
< modules_loaded ipmi_msghandler
< [ ${OnePlusLoaded} -ne 1 ] && unload_all_ipmi_modules && RETVAL=$((RETVAL | 1)) && return
---
> # modprobe ipmi_msghandler > /dev/null 2>&1
> # modules_loaded ipmi_msghandler
> # [ ${OnePlusLoaded} -ne 1 ] && unload_all_ipmi_modules && RETVAL=$((RETVAL | 1)) && return

Kent Baxley (kentb) wrote :

The OpenIPMI portion of this issue now seems to be fixed in Utopic & Trusty and I will try and verify the fix for Precise:

https://bugs.launchpad.net/ubuntu/+source/openipmi/+bug/1309860

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers