IPMI detection and automatic setting fail in Ubuntu 14.04 maas

[juan c caballero]

I have no ipmi detection or automatic setting of IPMI in Ubuntu 14.04 maas. I have the options pull down menu to select IPMI 1.5 or IPMI 2.0, but none of the ipmi setting work. Manual does not work ether. maas cannot power up servers.

[juan c caballero (juancc2007)]

please swap detension for detection

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1321885/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[juan c caballero (juancc2007)]

thanks for your response, i dont get any errors. it is like the software is there, but not doing anything. i will start to perform some tests so i can try to fingure out, which package it is. but i can say that when i was running ubuntu 12.04 server, that maas package commissioned servers and impi setting where configured automatic. it appears like the automatic detection, is not working.

[juan c caballero (juancc2007)]

i have trace the issue to the package above. i don't have the resources, to go deeper. but this is what i have come up with.

[Julian Edwards (julian-edwards)]

The 1.4 release had known problems around IPMI detection which are mostly fixed in the 1.5 release which is in trusty (14.04). Please upgrade to trusty, or if you are still on precise you can try out the MAAS in the cloud tools archive.

[Julian Edwards (julian-edwards)]

Please confirm if 1.5 solves your problem and can you confirm if you are running on trusty or precise.

[juan c caballero (juancc2007)]

I have verified that I am running ubuntu 14.4, with maas packages (maas 1.5.1 +bzr2252, these also matched the other maas, packages i.e maas-cli, maas-dns, etc).

i am also adding, just in-case this maybe machine specific. I am using 10 dell 1950, gen 2, 8gb ram. ( and 2 dell 1750. just as extras.) for a total of 12.

[juan c caballero (juancc2007)]

hi the error i am getting is (no power type defined)
which to me it means that the detection process is not working. I don't even get the auto detect option.

I watched the screen, I have a 3 min video of the whole process. and there is something wrong with impi
detection. i will post more when i slow down the video, and write down the errors.

[juan c caballero (juancc2007)]

here is a small log from the provisioning servers screen. i do have a video and some photos created from video .

 showing numerous errors.

screen dictation:

setting impitool (1.8.13-1)...
*starting IMPI event deamon
could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/impidev/0: no such file or directory
could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/impidev/0: no such file or directory
ipmi end: using pidfile /var/run/ipievd.pid0 could not open device at /dev/ipmi0 or /dev/ipmi/0
or /dev/impidev/0: no such file or directory
unable to open interface
...fail
invoke-rc-d: initscript ipmievd, action "start" failed.
unable to start ipmievd during installation. trying to disable.

then it executes a few other command::then i get a bunch of what appears to be code: lines programming line errors..too many to list....this one is the 1st one

traceback (most recent call last:)
>
  main ()
 file "/tmp/user_data_sh.FOxjgx/bin/maas-ipmi-autodetect" line 312....line 278,218,195,184 etc, etc,etc..

[Julian Edwards (julian-edwards)]

Can you please post details of the hardware you're using and someone can triage the bug.

[Andres Rodriguez (andreserl)]

Can you please paste the full log? What hardware is this? Can you please provide the output of bmc-config --checkout and ipmi-locate from the machine itself?

Thank you!

[juan c caballero (juancc2007)]

bmi log
juan@region:~$ sudo bmc-config --checkout
[sudo] password for
#
# Section UserX Comments
#
# In the following User sections, users should configure usernames, passwords,
# and access rights for IPMI over LAN communication. Usernames can be set to any
# string with the exception of User1, which is a fixed to the "anonymous"
# username in IPMI.
#
# For IPMI over LAN access for a username, set "Enable_User" to "Yes",
# "Lan_Enable_IPMI_Msgs" to "Yes", and "Lan_Privilege_Limit" to a privilege
# level. The privilege level is used to limit various IPMI operations for
# individual usernames. It is recommened that atleast one username be created
# with a privilege limit "Administrator", so all system functions are available
# to atleast one username via IPMI over LAN. For security reasons, we recommend
# not enabling the "anonymous" User1. For most users, "Lan_Session_Limit" can be
# set to 0 (or ignored) to support an unlimited number of simultaneous IPMI over
# LAN sessions.
#
# If your system supports IPMI 2.0 and Serial-over-LAN (SOL),
# a"SOL_Payload_Access" field may be listed below. Set the "SOL_Payload_Access"
# field to "Yes" or "No" to enable or disable this username's ability to access
# SOL.
#
# Please do not forget to uncomment those fields, such as "Password", that may
# be commented out during the checkout.
#
# Some motherboards may require a "Username" to be configured prior to other
# fields being read/written. If this is the case, those fields will be set to
# <username-not-set-yet>.
#
Section User1
 ## Give Username
 ## Username NULL
 ## Give password or blank to clear. MAX 16 chars (20 chars if IPMI 2.0 supported).
 ## Password
 ## Possible values: Yes/No or blank to not set
 ## Enable_User
 ## Possible values: Yes/No
 Lan_Enable_IPMI_Msgs No
 ## Possible values: Yes/No
 Lan_Enable_Link_Auth No
 ## Possible values: Yes/No
 Lan_Enable_Restricted_to_Callback No
 ## Possible values: Callback/User/Operator/Administrator/OEM_Proprietary/No_Access
 Lan_Privilege_Limit No_Access
 ## Possible values: 0-17, 0 is unlimited; May be reset to 0 if not specified
 ## Lan_Session_Limit
EndSection
Section User2
 ## Give Username
 Username root
 ## Give password or blank to clear. MAX 16 chars (20 chars if IPMI 2.0 supported).
 ## Password
 ## Possible values: Yes/No or blank to not set
 ## Enable_User
 ## Possible values: Yes/No
 Lan_Enable_IPMI_Msgs Yes
 ## Possible values: Yes/No
 Lan_Enable_Link_Auth Yes
 ## Possible values: Yes/No
 Lan_Enable_Restricted_to_Callback No
 ## Possible values: Callback/User/Operator/Administrator/OEM_Proprietary/No_Access
 Lan_Privilege_Limit Administrator
 ## Possible values: 0-17, 0 is unlimited; May be reset to 0 if not specified
 ## Lan_Session_Limit
EndS...

[juan c caballero (juancc2007)]

ipmi locate:

juancc@region:~$ sudo ipmi-locate
Probing KCS device using DMIDECODE... done
IPMI Version: 1.5
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA8
Register spacing: 4

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... done
IPMI Version: 1.5
IPMI locate driver: SMBIOS
IPMI interface: KCS
BMC driver device:
BMC I/O base address: 0xCA8
Register spacing: 4

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

juancc@region:~$

[juan c caballero (juancc2007)]

the servers are 13

10 dell power edge 1950 gen 2, 8gb ram 2x200 gb harddrive, in raid ..2 1gb network interfaces. dual xeon 64 bit. (these are the nodes)

1 dell power edge 1800 8gb ram , 1 500gb harddrive, 2 1gb network interfaces. dual xeon 64 bit (running ubuntu 14.04 maas region/cluster controller...

2 dell power edge 1750, 4 gb ram 2, 100gb harddrive , 2 1gb network interfaces, dual xeon 32 bit (these are also nodes) they are last in the setup. because they will be running 14.4 i386. or precise.

just want to add that this configuration was running ubuntu 12.04.4 maas/juju/openstack..until the release of 14.4 when i did a clean install.

and this is where i am at. stuck

[juan c caballero (juancc2007)]

Done...

On Thursday, May 22, 2014, Andres Rodriguez <email address hidden> wrote:

> Can you please paste the full log? What hardware is this? Can you please
> provide the output of bmc-config --checkout and ipmi-locate from the
> machine itself?
>
> Thank you!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1321885
>
> Title:
> IMPI detection and automatic setting fail in ubuntu 14.4 maas
>
> Status in MAAS:
> New
> Status in "freeipmi" package in Ubuntu:
> New
> Status in "ipmitool" package in Ubuntu:
> New
> Status in "maas" package in Ubuntu:
> New
> Status in "openipmi" package in Ubuntu:
> New
>
> Bug description:
> i have no impi detection or automatic setting of IMPI in ubuntu 14.4
> maas. i have the options pull down menu to select IMPI 1.5 or IMPI 2.0
> but none of the impi setting work . manual does not work ether. maas
> cannot power up servers.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1321885/+subscriptions
>

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jay Farschman (jfarschman)]

Juan,

I'm seeing the exact same problem ( I believe) and I noticed you are having problems getting into your server to see the failure. I addressed this with a backdoor.

https://maas.ubuntu.com/docs/troubleshooting.html#debugging-ephemeral-image

The output of my bmc-config --checkout and ipmi locate are very similar to yours. Let me know if you need help.

[juan c caballero (juancc2007)]

hi

thanks but for what I understand, I believe that this is machine specific.
I am getting these problems on dell 1750 and 1950 but not on my dell 2950.
it like they removed them from IMPI, between 12.04.4 and 14.04.

if you know how to notified, please do so.

I have already tried this before I posted the bug.

juancc

[Jason Hobbs (jason-hobbs)]

Juan,

Can you try running this script http://bazaar.launchpad.net/~maas-maintainers/maas/trunk/view/head:/etc/maas/templates/commissioning-user-data/snippets/maas_ipmi_autodetect.py on one of the servers you're having problems with?

You'll need to install freeipmi-utils first, and run it as root. This is the script that discovers and configures IPMI for MAAS.

Please post its output back to this bug.

Thanks,
Jason

[juan c caballero (juancc2007)]

thanks for the help, we have returned to 12.04.4, where we are currently
stuck on a juju bug. but we have maas running.... hopefully they should
have resolved this by the time we return to 14.4.. for the project must go
forward..

thanks

On Wed, Jun 18, 2014 at 5:24 PM, Jason Hobbs <email address hidden>
wrote:

> Juan,
>
> Can you try running this script http://bazaar.launchpad.net/~maas-
> maintainers/maas/trunk/view/head:/etc/maas/templates/commissioning-user-
> data/snippets/maas_ipmi_autodetect.py on one of the servers you're
> having problems with?
>
> You'll need to install freeipmi-utils first, and run it as root. This is
> the script that discovers and configures IPMI for MAAS.
>
> Please post its output back to this bug.
>
> Thanks,
> Jason
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1321885
>
> Title:
> IMPI detection and automatic setting fail in ubuntu 14.4 maas
>
> Status in MAAS:
> New
> Status in "freeipmi" package in Ubuntu:
> Confirmed
> Status in "ipmitool" package in Ubuntu:
> Confirmed
> Status in "maas" package in Ubuntu:
> Confirmed
> Status in "openipmi" package in Ubuntu:
> Confirmed
>
> Bug description:
> i have no impi detection or automatic setting of IMPI in ubuntu 14.4
> maas. i have the options pull down menu to select IMPI 1.5 or IMPI 2.0
> but none of the impi setting work . manual does not work ether. maas
> cannot power up servers.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1321885/+subscriptions
>

[juan c caballero (juancc2007)]

thanks for looking at this, but we have reinstalled 12.04.4, where we are currently stuck on a juju bug. but maas is working. except for the IMPI on my dell 1750s. which appear to have the same problem detecting IMPI..sorry i cannot continue to supply you with information. but we must have a working concept 12.04.4 or 14.4 ASAP. and it has been almost 3months with no working proof of concept and a lot of bugs. this stuff works great on VM, but it not good on hardware. need to get off VM. and support true metal.

thanks

juancc

[Hamed Hemmati (hamed-5)]

I am having problem with Dell servers too. Just like Juan I decided to go back to 12.04 but ended up with JuJu just not working properly so now I am back to 14.04 trying to figure out why the Dell servers don't get their power type set. The only Dell server that did get its power type set was poweredge 2950 all the other ones have issues for me. Even if I set the power type manually they still don't startup.

[juan c caballero (juancc2007)]

hi

you may just stay at 12.04.4, and wait for the juju issues to be resolved.
know that the next release of juju will include some bug fixes, from bugs
that i have found. hoping for the best.

juancc

On Fri, Jun 20, 2014 at 2:43 PM, Hamed Hemmati <email address hidden> wrote:

> I am having problem with Dell servers too. Just like Juan I decided to
> go back to 12.04 but ended up with JuJu just not working properly so now
> I am back to 14.04 trying to figure out why the Dell servers don't get
> their power type set. The only Dell server that did get its power type
> set was poweredge 2950 all the other ones have issues for me. Even if I
> set the power type manually they still don't startup.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1321885
>
> Title:
> IMPI detection and automatic setting fail in ubuntu 14.4 maas
>
> Status in MAAS:
> New
> Status in "freeipmi" package in Ubuntu:
> Confirmed
> Status in "ipmitool" package in Ubuntu:
> Confirmed
> Status in "maas" package in Ubuntu:
> Confirmed
> Status in "openipmi" package in Ubuntu:
> Confirmed
>
> Bug description:
> i have no impi detection or automatic setting of IMPI in ubuntu 14.4
> maas. i have the options pull down menu to select IMPI 1.5 or IMPI 2.0
> but none of the impi setting work . manual does not work ether. maas
> cannot power up servers.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1321885/+subscriptions
>

[Hamed Hemmati (hamed-5)]

I tried but I just had too many issues with juju and 12.04. So far the only issue I have with 14.04 is the fact that some of my poweredge servers aren't getting their power type set. Other than that juju seems to work fine on the other ones.

I rather just get the power type issue resolved and take advantage of the new maas features than go back to 12.04 and have to upgrade in 6 months.

[juan c caballero (juancc2007)]

hi

so you are saying that besides power up issues with impi juju works in
14.4...
i ask because i have been trying to build a cloud for over 4 months, with
bug after bug.

on my dell metal.

thanks for the input.

juancc

On Fri, Jun 20, 2014 at 3:36 PM, Hamed Hemmati <email address hidden> wrote:

> I tried but I just had too many issues with juju and 12.04. So far the
> only issue I have with 14.04 is the fact that some of my poweredge
> servers aren't getting their power type set. Other than that juju seems
> to work fine on the other ones.
>
> I rather just get the power type issue resolved and take advantage of
> the new maas features than go back to 12.04 and have to upgrade in 6
> months.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1321885
>
> Title:
> IMPI detection and automatic setting fail in ubuntu 14.4 maas
>
> Status in MAAS:
> New
> Status in "freeipmi" package in Ubuntu:
> Confirmed
> Status in "ipmitool" package in Ubuntu:
> Confirmed
> Status in "maas" package in Ubuntu:
> Confirmed
> Status in "openipmi" package in Ubuntu:
> Confirmed
>
> Bug description:
> i have no impi detection or automatic setting of IMPI in ubuntu 14.4
> maas. i have the options pull down menu to select IMPI 1.5 or IMPI 2.0
> but none of the impi setting work . manual does not work ether. maas
> cannot power up servers.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1321885/+subscriptions
>

[Hamed Hemmati (hamed-5)]

Yeah I was able to bootstrap just fine on the dell server that got its power type set. I haven't tested charms yet because I can't get any of my other dell servers to startup since their power types aren't getting set.

Juju has just evolved too much to be compatible with maas that comes with 12.04.

[Hamed Hemmati (hamed-5)]

Jason I ran the script you post and this is what I got on one of the nodes.

ubuntu@tf77q:~$ sudo python maas_ipmi_autodetect.py
Traceback (most recent call last):
  File "maas_ipmi_autodetect.py", line 312, in <module>
    main()
  File "maas_ipmi_autodetect.py", line 278, in main
    configure_ipmi_user(IPMI_MAAS_USER, IPMI_MAAS_PASSWORD)
  File "maas_ipmi_autodetect.py", line 218, in configure_ipmi_user
    apply_ipmi_user_settings(user_settings)
  File "maas_ipmi_autodetect.py", line 193, in apply_ipmi_user_settings
    verify_ipmi_user_settings(ipmi_user_number, user_settings)
  File "maas_ipmi_autodetect.py", line 182, in verify_ipmi_user_settings
    raise IPMIError(message)
__main__.IPMIError: IPMI user setting verification failures: for 'Enable_User', expected 'Yes', actual 'None'.

[Kent Baxley (kentb)]

If the 1950's and 1750's are having problems, what type of response do you get from this command?:

ipmipower -h <bmc ip address> -u <bmc username> -p <bmc password> --stat

Preferably you'll want to run that from the maas server running 14.04. On a very basic level, that's what MAAS is using to bring the nodes up and down. If your MAAS server is not running 14.04 then install freeipmi-tools on another system running 14.04 and see if you can contact the 1950's BMC with it.

In my lab I have an old R805 that was doing simliar things with regard to MAAS picking up the BMC properly. Adding in the ipmi v1.5, ip address, username and password got it working with 14.04 maas.

[Kent Baxley (kentb)]

..and if that doesn't work, then try:

ipmipower -D LAN -h <bmc ip address> -u <bmc username> -p <bmc password> --stat

Plus, I know these machines are older, but, is the latest BMC firmware installed on these affected systems?

[Rick Masters (grick23)]

Please fix the typos in the title of this bug (IMPI/IPMI and 14.4/14.04); they make it hard to find.

The root cause of this issue is that a least one type of BMC does not report back at least one setting (Enable_User) that has been configured correctly.

This was not a problem in 12.04 because earlier versions of maas do not attempt to verify that the BMC settings have been set correctly after configuring them. Previously, it just set the values and moved on. Now that the maas_ipmi_autodetect.py code attempts to read back the values, it is failing, even though the correct values appear to be in effect.

This issue is affecting Dell PowerEdge SC1435 and Dell 2970 models and probably others that use the same BMC. Note that these are fairly popular models. This was reproduced on a Dell SC1435 with 2.2.5 BIOS, which I believe is the latest. This is reproducable on multiple machines. Other Dell models (710s and 620s) do not exhibit this issue.

Here is an example on an SC1435:

manager@maas-ctrl-2:~$ sudo bmc-config --checkout --key-pair=User1:Username
Section User1
        ## Give Username
        ## Username NULL
EndSection
manager@maas-ctrl-2:~$ sudo bmc-config --checkout --key-pair=User2:Username
Section User2
        ## Give Username
        Username root
EndSection
manager@maas-ctrl-2:~$ sudo bmc-config --checkout --key-pair=User3:Username
Section User3
        ## Give Username
        Username maas
EndSection
manager@maas-ctrl-2:~$ sudo bmc-config --checkout --key-pair=User2:Enable_User
Section User2
        ## Possible values: Yes/No or blank to not set
        ## Enable_User
EndSection
manager@maas-ctrl-2:~$ sudo bmc-config --checkout --key-pair=User3:Enable_User
Section User3
        ## Possible values: Yes/No or blank to not set
        ## Enable_User
EndSection
manager@maas-ctrl-2:~$ sudo bmc-config --commit --key-pair="User2:Enable_User=Yes"
manager@maas-ctrl-2:~$ sudo bmc-config --checkout --key-pair=User2:Enable_User
Section User2
        ## Possible values: Yes/No or blank to not set
        ## Enable_User
EndSection
manager@maas-ctrl-2:~$

Note that if you disable the verification by commenting it out, then everything works fine.

Steps to perform the workaround on the maas server:

sudo vi /etc/maas/templates/commissioning-user-data/snippets/maas_ipmi_autodetect.py

Change this function to comment out the last line as shown:

def apply_ipmi_user_settings(user_settings):
    """Commit and verify IPMI user settings."""
    username = user_settings['Username']
    ipmi_user_number = pick_user_number(username)

    for key, value in user_settings.iteritems():
        bmc_user_set(ipmi_user_number, key, value)

    #verify_ipmi_user_settings(ipmi_user_number, user_settings)

While it seems clear the BMC should return the proper values, even if that got fixed (which I'll guess won't happen anytime soon), upgrading IPMI firmware is not an easy task for the casual user with Dell machines. See http://www.yellow-bricks.com/2011/06/27/dell-firmware-updates. Therefore, please consider this a request for MaaS to work with this beha...

[Kent Baxley (kentb)]

Fixed the bug title from the original reporter so that it's easier to find.

[Kent Baxley (kentb)]

grick23's workaround also seems to 'fix' this for a PowerEdge R805. Previously, this machine also had trouble with MAAS auto-detecting the IPMI settings with 14.04. So, it looks like many Dell 10th generation (10G) and older machines will run into problems based on the type of DRAC / BMC installed in them. As was stated in comment #33, the more modern 11G and 12G systems have no issues with the IPMI auto-detection.

[Kent Baxley (kentb)]

Took a shot at trying to address this. We'll see what the maintainers say.

[Hamed Hemmati (hamed-5)]

The work around fixed the problem with r200 and poweredge 850 as well.

[juan c caballero (juancc2007)]

The work around fixed the problem with poweredge 1950 not poweredge 1750.

[Kent Baxley (kentb)]

Try the script mentioned above and post the output:

https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1321885/comments/22

[Mark W Wenning (mwenning)]

Verified that the fix works on Dell PowerEdge R805 .

maas 1.7.0+bzr3299-0ubuntu1~trusty