MAAS ipmi fails on OCPv3 Roadrunner

[David Duffey]

The OCPv3 Roadrunner machine has been fully enabled and passes certification testing. When testing ipmitool locally I'm able to setup the BMC and users, etc.

When using MAAS, MAAS is able to setup the BMC network information (I see that it changes that), but it appears to fail to set a username and password. If I try to use the username and password as defined in the MAAS GUI, it fails. Therefore commissioning and juju bootstrapping the node has to be done manually (by physically pushing the power button).

If I use the username/password I've set on the BMC I can see that MAAS fails to set the username 'maas' and the password as defined in the MAAS gui.

Since the commissioning/enlisting process is temporary and I'm not sure how to login to this phase to gather data, troubleshooting tips are welcome.

[David Duffey (dduffey)]

I should add, that using ipmitool manually from a remote host using the 'admin' user I created also works fine. It simply seems that MAAS thinks it has set a user 'maas' when it has not.

[Julian Edwards (julian-edwards)]

Andres, you know more about this than me, any idea?

[Jeff Marcom (jeffmarcom)]

This error is present regardless whether maas is in use. Even using freeipmi locally fails to set ipmi parameters or get sdr readings with an error similar to:

map pfn expected mapping type uncached-minus for d7fa0000-d7fa1000, got write-back

This is present in saucy using openipmi and freeipmi.

[Jeff Marcom (jeffmarcom)]

BMC info:

<lshw:node id="serial" claimed="true" class="bus" handle="PCI:0000:01:00.6">
         <lshw:description>IPMI SMIC interface</lshw:description>
         <lshw:product>Broadcom Corporation</lshw:product>
         <lshw:vendor>Broadcom Corporation</lshw:vendor>
         <lshw:physid>0.6</lshw:physid>
         <lshw:businfo>pci@0000:01:00.6</lshw:businfo>
         <lshw:version>00</lshw:version>
         <lshw:width units="bits">64</lshw:width>
         <lshw:clock units="Hz">33000000</lshw:clock>
         <lshw:configuration>
          <lshw:setting id="driver" value="ipmi_si"/>
          <lshw:setting id="latency" value="0"/>
         </lshw:configuration>
         <lshw:capabilities>
          <lshw:capability id="pm">Power Management</lshw:capability>
          <lshw:capability id="pciexpress">PCI Express</lshw:capability>
          <lshw:capability id="bus_master">bus mastering</lshw:capability>
          <lshw:capability id="cap_list">PCI capabilities listing</lshw:capability>
         </lshw:capabilities>
         <lshw:resources>
          <lshw:resource type="irq" value="0"/>
          <lshw:resource type="memory" value="d0010000-d001ffff"/>
          <lshw:resource type="memory" value="d0000000-d000ffff"/>

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1210393

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jeff Marcom (jeffmarcom)]

ApportVersion: 2.12.1-0ubuntu4
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: ubuntu 2918 F.... pulseaudio
CasperVersion: 1.336
DistroRelease: Ubuntu 13.10
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
LiveMediaBuild: Ubuntu 13.10 "Saucy Salamander" - Alpha amd64 (20130916)
MachineType: empty empty
MarkForUpload: True
Package: linux (not installed)
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 radeondrmfb
ProcKernelCmdLine: noprompt cdrom-detect/try-usb=true file=/cdrom/preseed/hostname.seed boot=casper initrd=/casper/initrd.lz quiet splash -- maybe-ubiquity
ProcVersionSignature: Ubuntu 3.11.0-7.13-generic 3.11.0
PulseList:
 Error: command ['pacmd', 'list'] failed with exit code 1: Home directory not accessible: Permission denied
 No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.11.0-7-generic N/A
 linux-backports-modules-3.11.0-7-generic N/A
 linux-firmware 1.114
RfKill:

Tags: saucy
Uname: Linux 3.11.0-7-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

dmi.bios.date: 03/04/2013
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 4.6.5
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: S8237
dmi.board.vendor: TYAN
dmi.board.version: empty
dmi.chassis.asset.tag: empty
dmi.chassis.type: 28
dmi.chassis.vendor: empty
dmi.chassis.version: empty
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr4.6.5:bd03/04/2013:svnempty:pnempty:pvrempty:rvnTYAN:rnS8237:rvrempty:cvnempty:ct28:cvrempty:
dmi.product.name: empty
dmi.product.version: empty
dmi.sys.vendor: empty

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Jeff Marcom (jeffmarcom)]

apport information

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1210393

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jeff Marcom (jeffmarcom)]

Troubleshooting steps:

modprobe ipmi_msghandler
modprobe ipmi_devintf
modprobe ipmi_si

Will enable the bmc device parameters to be changed locally and eliminate the write back errors, something I feel the Saucy ephemeral images are not doing.

Secondly, trying to enable link authentication fails to set:
 ipmitool lan set 1 auth ADMIN MD5,PASSWORD

Channel link access still states:
User ID : 4

User Name : admin

Fixed Name : No

Access Available : call-in / callback

Link Authentication : disabled

IPMI Messaging : enabled

Also, setting link access for channel 1 fails:

sudo ipmitool lan set 1 access on
Set Channel Access for channel 1 failed: Unknown (0x83)

[Jeff Marcom (jeffmarcom)]

BMC is detected on the network, just cannot establish connection via link:

ipmiping 10.193.37.122
ipmiping 10.193.37.122 (10.193.37.122)
response received from 10.193.37.122: rq_seq=32
response received from 10.193.37.122: rq_seq=33
response received from 10.193.37.122: rq_seq=34
response received from 10.193.37.122: rq_seq=35
response received from 10.193.37.122: rq_seq=36

[Jeff Marcom (jeffmarcom)]

I don't understand. Why was this set to incomplete? What specific information do you need?

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1210393

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jeff Marcom (jeffmarcom)]

Confirmed that the ipmi-locate command fails during the MAAS enlistment process on OCV3 bmc.

ipmi-locate failure:
Probing KCS device using DMIDECODE... done
IPMI Version: 2.0
IPMI locate driver: DMIDECODE
IPMI interface: KCS
BMC driver device:
BMC memory base address: 0x0
Register spacing: 1

Probing SMIC device using DMIDECODE... FAILED

Probing BT device using DMIDECODE... FAILED

Probing SSIF device using DMIDECODE... FAILED

Probing KCS device using SMBIOS... done
IPMI Version: 2.0
IPMI locate driver: SMBIOS
IPMI interface: KCS
BMC driver device:
BMC memory base address: 0x0
Register spacing: 1

Probing SMIC device using SMBIOS... FAILED

Probing BT device using SMBIOS... FAILED

Probing SSIF device using SMBIOS... FAILED

Probing KCS device using ACPI... FAILED

Probing SMIC device using ACPI... FAILED

Probing BT device using ACPI... FAILED

Probing SSIF device using ACPI... FAILED

Probing KCS device using PCI... FAILED

Probing SMIC device using PCI... FAILED

Probing BT device using PCI... FAILED

Probing SSIF device using PCI... FAILED

[Jeff Marcom (jeffmarcom)]

ipmi-chassis --get-capabilities command success:

Intrusion sensor : not provided
Front Panel Lockout : provided
Diagnostic Interrupt : provided
Power interlock : not provided
FRU Info Device Address : 20h
SDR Device Address : 20h
SEL Device Address : 20h
Sys Mgmt Device Address : 0h
Bridge Device Address : 20h

IPMI modules are loaded

[Jeff Marcom (jeffmarcom)]

User list after cloud-init/ipmitool has tried to set maas username/password:

ID Name Callin Link Auth IPMI Msg Channel Priv Limit
1 true false true USER
2 admin true false true ADMINISTRATOR
3 Operator true false true OPERATOR
4 admin true false true ADMINISTRATOR

[Jeff Marcom (jeffmarcom)]

Just to clarify, I'm able to successfully update the username+password in the ephemeral saucy image by hand using ipmitool

[Jeff Marcom (jeffmarcom)]

So basically I was able to get this to work in maas.

Basically you need to change the following line in :
/etc/maas/templates/commissioning-user-data/snippets/maas_ipmi_autodetect.py

    def commit_ipmi_user_settings(user, password):
    ipmi_user_number = get_ipmi_user_number(user)
    if ipmi_user_number is None:
        (status, output) = commands.getstatusoutput('bmc-config --commit --key-pair="User10:Username=%s"' % user)
        ipmi_user_number = get_ipmi_user_number(user)

You have to change User10 to User4, as this bmc config maxes out at 4 User sections.

[Jeff Marcom (jeffmarcom)]

Also, it seems you have to set the power management option to ipmi_2.0 in the MAAS UI because the power command will work.

[Julian Edwards (julian-edwards)]

Ok so here's the bug:

maas_ipmi_autodetect.py looks for a user slot called "maas", and reprograms that first. If it does not exist then it defaults to User10. User10 does not exist on this hardware.

What it should be doing:
 * Look for a user called maas and re-use or re-program
 * Failing that, look for an empty slot
 * Failing that, go for User4 (or perhaps make this depend on the type of BMC detected?)

[Jeff Lane  (bladernr)]

why not have it look for an existing user called maas, and if none is present, have it walk the slots starting at 0 until it finds an empty or runs out of spaces?

Or is that one of those weird things that seem simple on paper, but in practice are really actually difficult to do because of some IPMI issue or design?

Anyway, just curious.

[Jason Hobbs (jason-hobbs)]

I've started work on a patch to fix this. It will find either an existing maas user, or will find the first disabled user with an empty username. If it can't find either it will bail and give up on automatic IPMI config.

[David Duffey (dduffey)]

@ Jason,

Is this targeted for 14.04 LTS?

[Andres Rodriguez (andreserl)]

Hi David!

Yes this is targeted for 14.04.

Cheers.

[Jason Hobbs (jason-hobbs)]

I've posted a branch with a fix to lp:~jason-hobbs/maas/lp-1210393

I've manually tested this, but for lack of access, not on OCPv3 Roadrunner.

[David Duffey (dduffey)]

Thanks Jason, we will get this tested (Rod/Samantha)

[Jason Hobbs (jason-hobbs)]

Cool David - let me know how it works out. The branch is otherwise complete/reviewed and ready to land.

[Dustin Kirkland  (kirkland)]

Hey Jason! Thanks!

Could you please re-assign this bug, then, to Samantha or Rod? And Samantha/Rod, could you please update the status (in-progress) when you're working on it?

That helps those of us here watching/waiting for this work to complete :-)

Thanks!
Dustin

[Jason Hobbs (jason-hobbs)]

Hey Dustin - I reassigned to David since I'm not sure who will be testing it. David/Samantha/Rod - please reassign to whoever is doing the test!

[Jason Hobbs (jason-hobbs)]

I don't believe there are plans to fix this against Saucy.

[Joseph Salisbury (jsalisbury)]

This bug was nominated against a series that is no longer supported, ie saucy. The bug task representing the saucy nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.