ubuntu18.04: ipmitool failed: Could not open device at /dev/ipmi0 : No such file or directory

[[e-mail]]

ipmitool failed in-band
#ipmitool lan print
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

And it work out-band:
# ipmitool -H {theip} -I lanplus -U {user} -P {passwd} lan print
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback : MD2 MD5 PASSWORD
                        : User : MD2 MD5 PASSWORD
                        : Operator : MD2 MD5 PASSWORD
                        : Admin : MD2 MD5 PASSWORD
                        : OEM : MD2 MD5 PASSWORD
IP Address Source : DHCP Address
...

But the file exist:
# ls -al /dev/ipmi0
crw------- 1 root root 240, 0 Jul 16 13:55 /dev/ipmi0
# cat /proc/devices | grep ipmidev
240 ipmidev

packages that I installed:
# dpkg -l | grep ipmi
ii freeipmi-common 1.4.11-1.1ubuntu4.1 amd64 GNU implementation of the IPMI protocol - common files
ii ipmitool 1.8.18-5ubuntu0.1 amd64 utility for IPMI control with kernel driver or LAN interface (daemon)
ii libfreeipmi16 1.4.11-1.1ubuntu4.1 amd64 GNU IPMI - libraries
ii libopenipmi0 2.0.22-1.1ubuntu2.1 amd64 Intelligent Platform Management Interface - runtime
ii openipmi 2.0.22-1.1ubuntu2.1 amd64 Intelligent Platform Management Interface (for servers)

the mods:
# lsmod| grep ipmi_
ipmi_devintf 20480 0
ipmi_poweroff 16384 0
ipmi_watchdog 28672 0
ipmi_ssif 32768 0
ipmi_si 61440 0
ipmi_msghandler 53248 6 ipmi_devintf,ipmi_si,nvidia,ipmi_watchdog,ipmi_ssif,ipmi_poweroff

the kernel:
# uname -a
Linux A16969628514217 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

the ubuntu version:
# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

[Paride Legovini (paride)]

Thanks for your report. Let's rule out the simplest hypothesis: are you running the command as root? I can reproduce the error you observe running ipmitool as an unprivileged user:

$ ls -l /dev/ipmi0
crw------- 1 root root 239, 0 May 30 13:39 /dev/ipmi0
$ ipmitool lan print
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
$ sudo ipmitool lan print
Set in Progress : Set Complete
Auth Type Support :
Auth Type Enable : Callback :
                        : User :
                        : Operator :
                        : Admin :
                        : OEM :
IP Address Source : Static Address
[...]

Can you please verify this? I'm marking this report as "Incompete" for the moment, please change the status back to "New" after providing the additional information. Thanks!

[[e-mail] (xiaotong-tony)]

I confirm ,run all command with root. After the machine restarts, all ipmitool are back to normal.I don't know why.
I try on other machines , and the problem cannot be reproduced.

Thank you very much.

[Paride Legovini (paride)]

I'm marking this report as Invalid then. Thanks for the follow-up!

[Ventsislav Georgiev (vgeorgiev)]

Not sure why it's marked as invalid. This appears on Ubuntu 18.04 with the same error.

root@x:/dev# sudo ipmitool lan print
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

root@x:/dev# uname -a
Linux x 4.15.0-117-generic #118-Ubuntu SMP Fri Sep 4 20:02:41 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

root@x:/dev# ls -la ipmi0
crw------- 1 root root 240, 0 Sep 11 10:32 ipmi0
root@x:/dev# lsmod | grep ipmi
ipmi_si 57344 0
ipmi_devintf 20480 0
ipmi_ssif 32768 0
ipmi_msghandler 53248 3 ipmi_devintf,ipmi_si,ipmi_ssif

I even added this - "GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity ipmi_si.type=kcs ipmi_si.ports=0xca2" no results.

ipmitool also doesn't work on Mint 19.3 with 5.8.2 kernel.

[Christian Ehrhardt  (paelzer)]

From a machine with actual ipmi support:
$ lsmod | grep ipmi
ipmi_ssif 36864 0
ipmi_si 65536 0
ipmi_devintf 20480 0
ipmi_msghandler 106496 3 ipmi_devintf,ipmi_si,ipmi_ssif

$ ll /dev/ipmi0 /dev/ipmi/0 /dev/ipmidev/0
ls: cannot access '/dev/ipmi/0': No such file or directory
ls: cannot access '/dev/ipmidev/0': No such file or directory
crw------- 1 root root 238, 0 Aug 24 06:27 /dev/ipmi0

$ sudo ipmitool lan print
Set in Progress : Set Complete
Auth Type Support :
Auth Type Enable : Callback :
...

But as one would expect without sudo:
$ ipmitool lan print
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

But the same also triggers on systems without ipmi (obviously).
So when I run it in a container or VM for example.

So we now know this can happen:
a) without root permissions (user error)
b) on a system without ipmi (well, that is ok then)
c) odd error we need to find (this would be worth to debug).

Does your kernel create and populate any of these?
$ ls -laF /dev/ipmi0 /dev/ipmi/0 /dev/ipmidev/0
And which modules are loaded?
$ lsmod | grep ipmi

Adding a kernel task for your reply to be seen there

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1836701

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ventsislav Georgiev (vgeorgiev)]

I've given the mods loaded in my initial comment. The output of ls -laF is the following:

root@x:~# ls -laF /dev/ipmi0 /dev/ipmi/0 /dev/ipmidev/0
ls: cannot access '/dev/ipmi/0': No such file or directory
ls: cannot access '/dev/ipmidev/0': No such file or directory
crw------- 1 root root 240, 0 Sep 11 10:32 /dev/ipmi0

Not sure what logs I can provide, there are errors or anything related to ipmi.

[Christian Ehrhardt  (paelzer)]

Hmm, so your system indeed has /dev/ipmi0 then (also you had that implicitly in comment #4 already) - thanks.
But that means we are back at the beginning with the but triggering for you but being non reproducible in other places :-/

For the sake of generating more ideas what could happen in your case.
$ sudo su -
$ strace -rT -ff -o ipmitool.strace ipmitool lan print

In my cross check I see an openat and then a bunch of ioctls.
We could then try to compare your output ....

[Ventsislav Georgiev (vgeorgiev)]

Yeah I did try that (saw it somewhere among the web) but it pretty much gives nothing, because of the main error.

root@x:~# strace -rT -ff -o ipmitool.strace ipmitool lan print
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

So I guess that ipmitool is just bricked on ubuntu. As mentioned in my first comment, I did try it on my own machine which is Mint (again ubuntu-ish based) and it has the same error.

This has never occurred on CentOS 6/7/8, so I'll just have to move this host away from Ubuntu.

[Christian Ehrhardt  (paelzer)]

On Fri, Sep 18, 2020 at 2:15 PM Ventsislav Georgiev
<email address hidden> wrote:
>
> Yeah I did try that (saw it somewhere among the web) but it pretty much
> gives nothing, because of the main error.
>
> root@x:~# strace -rT -ff -o ipmitool.strace ipmitool lan print
> Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

You this command redirects the logs to the file "ipmitool.strace" it
isn't expected to show anything else on the console.
Just attach that file here please.
Even if you give up :sad: it would help others if they hit the same ...

[Christian Ehrhardt  (paelzer)]

Mine:
openat(AT_FDCWD, "/dev/ipmi0", O_RDWR) = 3 <0.000052

Yours:
openat(AT_FDCWD, "/dev/ipmi0", O_RDWR) = -1 EINVAL (Invalid argument) <0.000053>

So we are back at the kernel backend of this - on your particular hardware/firmware - not behaving correctly. This isn't one of the later ioctls, just the basic open fails.

I re-checked my system and even a "cat /dev/ipmi0" while failing as expected eventually can open the device just fine (only fails later when reading)
# strace -e openat cat /dev/ipmi0 2>&1|grep ipmi0
openat(AT_FDCWD, "/dev/ipmi0", O_RDONLY) = 3

So why isn't it "openable" in his case?
@Kernel people, what would you want Ventsislav to look out for ?