[2.4] MAAS UI becomes unresponsive (slow) when there's disconnected controllers/RPC errors

[Andres Rodriguez]

Upgrading my primary region/rack, MAAS bacame bvery unresponsive and in the logs I saw the following errors.

This seems like a new recurrence of this old bug: https://bugs.launchpad.net/maas/+bug/1360004

2018-04-09 15:39:05 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'maas00' (6g7yrg).
2018-04-09 15:39:06 maasserver: [error] Error while calling DescribePowerTypes: Unable to get RPC connection for rack controller 'node01' (4rcbwp).
2018-04-09 15:39:10 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'maas00' (6g7yrg).
2018-04-09 15:39:10 maasserver: [error] Error while calling DescribePowerTypes: Unable to get RPC connection for rack controller 'node01' (4rcbwp).
2018-04-09 15:39:16 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'maas00' (6g7yrg).
2018-04-09 15:39:16 maasserver: [error] Error while calling DescribePowerTypes: Unable to get RPC connection for rack controller 'node01' (4rcbwp).
2018-04-09 15:39:20 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'maas00' (6g7yrg).
2018-04-09 15:39:20 maasserver: [error] Error while calling DescribePowerTypes: Unable to get RPC connection for rack controller 'node01' (4rcbwp).

[Andres Rodriguez (andreserl)]

This seems very similar to:

https://bugs.launchpad.net/maas/+bug/1360004

[Andres Rodriguez (andreserl)]

https://bugs.launchpad.net/maas/+bug/1761601 is a duplicate of this.

[Andres Rodriguez (andreserl)]

PLease see 1762461 for more details too.

[Mark Shuttleworth (sabdfl)]

Not sure where we are tracking this, but I see it in the GMAAS too. I think I have one sick rack controller. Might it have to do with the IP addresses on which RPC services listen?

[Mark Shuttleworth (sabdfl)]

I see this:

2018-05-06 12:47:16 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'lapsi' (4nqs36).
2018-05-06 12:47:23 -: [critical] Unhandled error in EventualResult
 Traceback (most recent call last):
 Failure: twisted.internet.defer.CancelledError:

2018-05-06 12:47:26 maasserver: [error] Error while calling DescribePowerTypes: RPC connection timed out to rack controller 'lapsi' (4nqs36).

[Phil Merricks (seffyroff)]

I'm getting a similar experience on 2.5.0 final here. 3 rack controllers offline and I've been waiting 45 mins to bounce the region service. This is my second attempt - the first time the box eventually ground to a complete halt.

[Lee Trager (ltrager)]

Hi Phil,

Could you please open a new bug and attach your MAAS logs?

[Phil Merricks (seffyroff)]

I will do that next time I'm running a deployment. FYI I resolved this by:

1: Restoring the racks and region to their prior state (before I shut down the racks to migrate them)
2: Deleting the racks from the region controller via the Controllers UI

With that, the Region Controller played nice and allowed me to migrate it very smoothly. Hurrah!