[2.3, snap] Rack Controller from the snap fails to power manage on IPMI

[Jeff Lane ]

This was reported to me by a user trying to deploy a note on one of our MAAS networks for testing purposes. We're on 2.3.0~alpha3 from the experimental3 ppa (was on this PPA after consulting a while back with the MAAS team).

So Sam reported that when he tried deploying a node, he got a Power Error message and a popup error warning that said:

Error:[Errno 2] No such file or directory: '/usr/sbin/ipmipower'

/usr/sbin/ipmipower is clearly there, and clearly works from a console:

ubuntu@landmaas:~$ which ipmipower
/usr/sbin/ipmipower
ubuntu@landmaas:~$ /usr/sbin/ipmipower -s -h 172.24.25.55 -u maas -p <PASSWORD>
172.24.25.55: on

This is also an intermittent error. I am able to easily reproduce this by simply clicking the Check Power link a few times. It seems to happen anywhere between about 40 - 80% of the time for me. I've attached a short video showing the reproduction in action.

(Note, ignore the Failed Deployment message, this is due to a separate Petitboot issue and is not related to power control).

ubuntu@landmaas:~$ apt-cache policy maas
maas:
  Installed: 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
  Candidate: 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
  Version table:
 *** 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1 500
        500 http://ppa.launchpad.net/maas-maintainers/experimental3/ubuntu xenial/main amd64 Packages
        500 http://ppa.launchpad.net/maas-maintainers/experimental3/ubuntu xenial/main i386 Packages
        100 /var/lib/dpkg/status
     2.2.2-6099-g8751f91-0ubuntu1~16.04.1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
     2.2.2-6099-g8751f91-0ubuntu1~16.04.1 500
        500 http://ppa.launchpad.net/maas/stable/ubuntu xenial/main amd64 Packages
        500 http://ppa.launchpad.net/maas/stable/ubuntu xenial/main i386 Packages
     2.0.0~beta3+bzr4941-0ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial/main i386 Packages

[Andres Rodriguez (andreserl)]

Hi Jeff,

Can you please attach logs? That is :

- /var/log/Maas/*log
- events for the machine (all of them, you can grab them via the "events" endpoint)

[Andres Rodriguez (andreserl)]

Also, please:

 dpkg -l | grep maas

Also, is this a single region/rack of multiple region rack?

[Jeff Lane  (bladernr)]

Just to double check that this wasn't just some cached data wonkiness or something, I rebooted the MAAS server completely and the problem persists. The only difference is that now I get the Power Error status, but no popup telling me why.

[Andres Rodriguez (andreserl)]

ack, we wont be able to really know what's wrong without the requested information!

THanks!

[Jeff Lane  (bladernr)]

Event Time
Failed to query node's BMC - [Errno 2] No such file or directory: '/usr/sbin/ipmipower' Wed, 30 Aug. 2017 15:51:13
Queried node's BMC - Power state queried: off Wed, 30 Aug. 2017 15:46:12
Queried node's BMC - Power state queried: off Wed, 30 Aug. 2017 15:41:12
Failed to query node's BMC - [Errno 2] No such file or directory: '/usr/sbin/ipmipower' Wed, 30 Aug. 2017 15:35:58
Failed to query node's BMC - [Errno 2] No such file or directory: '/usr/sbin/ipmipower' Wed, 30 Aug. 2017 15:30:58
Queried node's BMC - Power state queried: off Wed, 30 Aug. 2017 15:25:57
Queried node's BMC - Power state queried: off Wed, 30 Aug. 2017 15:20:57
Queried node's BMC - Power state queried: off Wed, 30 Aug. 2017 15:15:57
Failed to query node's BMC - [Errno 2] No such file or directory: '/usr/sbin/ipmipower' Wed, 30 Aug. 2017 15:10:43
Failed to query node's BMC - [Errno 2] No such file or directory: '/usr/sbin/ipmipower' Wed, 30 Aug. 2017 15:05:43
Failed to query node's BMC - [Errno 2] No such file or directory: '/usr/sbin/ipmipower' Wed, 30 Aug. 2017 15:00:43
Queried node's BMC - Power state queried: off Wed, 30 Aug. 2017 14:55:42
Queried node's BMC - Power state queried: off Wed, 30 Aug. 2017 14:50:42
Failed to query node's BMC - [Errno 2] No such file or directory: '/usr/sbin/ipmipower' Wed, 30 Aug. 2017 14:45:28
Failed to query node's BMC - [Errno 2] No such file or directory: '/usr/sbin/ipmipower' Wed, 30 Aug. 2017 14:40:28
Queried node's BMC - Power state queried: off Wed, 30 Aug. 2017 14:35:27

[Jeff Lane  (bladernr)]

Attached is the last 1000 events for this node

[Jeff Lane  (bladernr)]

ubuntu@landmaas:~$ dpkg -l |grep maas|awk '{print $1" "$2"\t\t"$3}'
ii maas 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
ii maas-cert-server 0.2.30-0~76~ubuntu16.04.1
ii maas-cli 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
rc maas-cluster-controller 1.9.3+bzr4577-0ubuntu1~trusty1
ii maas-common 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
ii maas-dhcp 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
ii maas-dns 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
ii maas-proxy 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
ii maas-rack-controller 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
ii maas-region-api 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
ii maas-region-controller 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
rc maas-region-controller-min 1.9.3+bzr4577-0ubuntu1~trusty1
ii python3-django-maas 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
ii python3-maas-client 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1
ii python3-maas-provisioningserver 2.3.0~alpha3-6223-gaca9fbb-0ubuntu1~16.04.1

The setup is all on one box, so region and rack together.

Here are the maas logs also

[Andres Rodriguez (andreserl)]

So I have tried to reproduce this over and over and over against different type of hardware and I have been unable to.

I did find this in the logs, for example:

Aug 31 14:10:58 landmaas maas.node: [error] baltar: Marking node failed: Power off for the node failed: Failed talking to node's BMC: [Errno 2] No such file or directory: '/usr/sbin/ipmi-chassis-config'

But, I wonder if this could be two things:

1. no permissions to access the file
2. Or the error is a false possitive for another underlying error ?
3. Someone messing up with permissions or the file itself, or something removed packages?

[Jeff Lane  (bladernr)]

After chatting with Newell, I looked at the apt history log and it shows three recent updates to MAAS:
Start-Date: 2017-08-23 10:54:10
Commandline: apt install maas
Requested-By: ubuntu (1000)
Upgrade: python-curtin:amd64 (0.1.0~bzr511-0ubuntu1~ubuntu16.04.1, 0.1.0~bzr518-0ubuntu1~ubuntu16.04.1), python3-maas-provisioningserver:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), python3-django-maas:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), python3-curtin:amd64 (0.1.0~bzr511-0ubuntu1~ubuntu16.04.1, 0.1.0~bzr518-0ubuntu1~ubuntu16.04.1), maas-common:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), maas-rack-controller:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), maas-cli:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), maas-dns:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), maas:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), python3-maas-client:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), maas-region-controller:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), maas-dhcp:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), maas-region-api:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1), curtin-common:amd64 (0.1.0~bzr511-0ubuntu1~ubuntu16.04.1, 0.1.0~bzr518-0ubuntu1~ubuntu16.04.1), maas-proxy:amd64 (2.3.0~alpha1-6163-gd7f5060-0ubuntu1~16.04.1, 2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1)
End-Date: 2017-08-23 11:01:14

Start-Date: 2017-08-29 11:08:47
Commandline: apt install maas
Requested-By: ubuntu (1000)
Upgrade: python3-maas-provisioningserver:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), python3-django-maas:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), maas-common:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), maas-rack-controller:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), maas-cli:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), maas-dns:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), maas:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), python3-maas-client:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), maas-region-controller:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), maas-dhcp:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3.0~alpha3-6214-g6c2ae15-0ubuntu1~16.04.1), maas-region-api:amd64 (2.3.0~alpha3-6199-g057d34c-0ubuntu1~16.04.1, 2.3....

[Andres Rodriguez (andreserl)]

I figured it out. There was a rack controller installed on a snap, and the snap was failing to power manage the machine.