Comment 3 for bug 1704489

I added some logging in the ipmi power code and in power_query_all() where it's building its request list. It's sending the power query request once for each RPC client it is - which is quite a few. Here's some output from that:

http://paste.ubuntu.com/25115285/

Jul 17 23:36:08 lairon maas.power: [error] making query for client yb87bd
Jul 17 23:36:08 lairon maas.power: message repeated 11 times: [ [error] making query for client yb87bd]

That means controller yb87bd will make 12 ipmi power queries, all to the same node, and power_query_all() will wait for them all to complete.

You can see the 8 requests issued on the rackd that's on the same node as the region log:
Jul 17 23:36:08 lairon maas.drivers.power.ipmi: [error] JASON: issuing command to ('/usr/sbin/ipmipower', '-W', 'opensesspriv', '--driver-type', 'LAN_2_0', '-h', '192.168.246.112', '-u', 'Administrator', '-p', 'insecure', '--stat'): 192.168.246.112
Jul 17 23:36:08 lairon maas.drivers.power.ipmi: message repeated 7 times: [ [error] JASON: issuing command to ('/usr/sbin/ipmipower', '-W', 'opensesspriv', '--driver-type', 'LAN_2_0', '-h', '192.168.246.112', '-u', 'Administrator', '-p', 'insecure', '--stat'): 192.168.246.112]

So for one node, that's 20 ipmi power queries. That's probably why it's taking so long.

It should only be doing one ipmi power query per rack controller. I'm not sure if it's right for there to be multiple RPC clients per rack controller, but if so, the code in power_query_all shouldn't be using each of them.