P8 node "modoc" is not very stable for kernel SRU testing

[Po-Hsu Lin]

This Power8 node "modoc" is not very stable for kernel SRU testing.

MAAS deployment is not working reliably recently.

Fail rate is 45 out 63 attempts

And when you try to release it with disk erasing, sometimes it will "Failed disk erasing"

I have tried to revive it with the following attempts:
1. ipmi chassis power cycle command to power cycle it
2. Manually turn off / on the pdu from bos01-b-07-fa1-12 and bos01-b-07-fb1-12

But yet it's still very unstalbe.

[Frank Heimes (fheimes)]

If modoc failed to deploy with MAAS in the past, did you noticed any error messages in MAAS?
Especially in the 'Tests', 'Logs' or 'Events' section of the Machine details in MAAS or did it just hang?

[Po-Hsu Lin (cypressyew)]

IIRC there will be a web consle for the BMC, but when I tried to access it the web console is not accessible (refused to connect)

[Andrew Cloke (andrew-cloke)]

If modoc is giving inconsistent results, do you want to switch kernel testing to witchita while we investigate?

[Po-Hsu Lin (cypressyew)]

I have watched one deployment failure event, the deployment will fail in ~1 minute with "Installation has failed and no output was given."

The event suggest that the BMC is not working:
    Marking node failed - Power on for the node failed: Failed talking to node's BMC: Failed to power c3mfb7. BMC never transitioned from off to on.

I need to power off / on modoc_psu1 and modoc_psu2 to make the power supply back to work.

[Frank Heimes (fheimes)]

Depending on the Power system the boot up process (especially cold start, but also restart/recycle) may take quite some time (several minutes).
I'm wondering if MAAS sometimes just times out - does not wait long enough (and if such a timeout is configurable ...)
Looks like there is no official MAAS UI way to tweak the ipmi timeout, but found this:
https://bugs.launchpad.net/maas/+bug/1521290/comments/8
(Anyway, I will not just change any config w/o getting a +1 from an expert ...)

[Andrew Cloke (andrew-cloke)]

Following a firmware update, modoc appears to be functioning better now.
Marking as "Fix Released" for now, but if it starts to misbehave again, please feel free to reopen.

[Po-Hsu Lin (cypressyew)]

Yes I have modoc stressed with 4 consecutive jenkins testing jobs, with different series deployed. The result is positive. I didn't see the "Failed talking to node's BMC" error in the events again.

Thanks for your help.