MAAS

[2.3] rackd — 12% connected to region controllers.

  1. Series 2.2
  2. Bug #1727073
Bug #1727073 reported by Andres Rodriguez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse
MAAS 2.3.0beta3
2.2
Fix Released
Critical
Blake Rouse
MAAS 2.2.3

Bug Description

I have 2 region/racks but the racks show maas00 -> primary, node01 -> secondary):

 rackd — 12% connected to region controllers.

There's no errors in the logs, no failures, no nothing.

The one thing I did see constantly is:

2017-10-24 21:33:09 regiond: [info] 10.90.90.7 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:33:32 regiond: [info] 10.90.90.1 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:33:39 regiond: [info] 10.90.90.7 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:34:02 regiond: [info] 10.90.90.1 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:34:09 regiond: [info] 10.90.90.7 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:34:25 maasserver.regiondservices.active_discovery: [info] Active network discovery: Active scanning is not enabled on any subnet. Skipping periodic scan.
2017-10-24 21:34:32 regiond: [info] 10.90.90.1 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:34:39 regiond: [info] 10.90.90.7 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:35:02 regiond: [info] 10.90.90.1 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:35:09 regiond: [info] 10.90.90.7 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:35:32 regiond: [info] 10.90.90.1 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)2017-10-24 21:35:39 regiond: [info] 10.90.90.7 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2017-10-24 21:36:02 regiond: [info] 10.90.90.1 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)

2017-10-24 21:36:09 regiond: [info] 10.90.90.7 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)

See original description

Related branches

~blake-rouse/maas:ha-fixes-2.2
Blake Rouse (community): Approve
Diff: 874 lines (+509/-34)
13 files modified
src/maasserver/locks.py (+9/-6)
src/maasserver/models/node.py (+12/-2)
src/maasserver/models/tests/test_node.py (+1/-1)
src/maasserver/rpc/rackcontrollers.py (+1/-1)
src/maasserver/rpc/regionservice.py (+19/-3)
src/maasserver/rpc/tests/test_rackcontrollers.py (+2/-2)
src/maasserver/rpc/tests/test_regionservice.py (+37/-0)
src/maasserver/utils/orm.py (+11/-0)
src/maasserver/utils/tests/test_orm.py (+16/-0)
src/provisioningserver/rackdservices/tests/test_tftp.py (+223/-0)
src/provisioningserver/rackdservices/tftp.py (+41/-1)
src/provisioningserver/rpc/clusterservice.py (+89/-17)
src/provisioningserver/rpc/tests/test_clusterservice.py (+48/-1)
~blake-rouse/maas:fix-rack-rpc-conn
Andres Rodriguez (community): Approve
Diff: 395 lines (+195/-23)
6 files modified
src/maasserver/models/node.py (+1/-1)
src/maasserver/models/tests/test_node.py (+1/-1)
src/maasserver/rpc/regionservice.py (+19/-3)
src/maasserver/rpc/tests/test_regionservice.py (+37/-0)
src/provisioningserver/rpc/clusterservice.py (+89/-17)
src/provisioningserver/rpc/tests/test_clusterservice.py (+48/-1)
Andres Rodriguez (andreserl)
Changed in maas:
assignee: nobody → Blake Rouse (blake-rouse)
milestone: none → 2.3.0beta3
importance: Undecided → Critical
status: New → Triaged
description: updated
Revision history for this message
Andres Rodriguez (andreserl) wrote :

After being idled, I'm constantly seeing this:

"One rack controller is not yet connected to the region. Visit the rack controllers page for more information."

I can confirm the same happens in other clusters.

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Something strange I noticed:

1. restarted secondary (node01) region/rack

2. Primary (maas00) UI now shows primary region/rack as "50% connected to region controllers." which is correct, provided that node01 is down and the rack running on maas00 is only connected to one region.

3. After secondary region/rack (node01) comes back up, node01 shows all connected. But maas00 shows "38% connected to region controllers." (restarting the maas-rackd didn't help).

4. restarted postgresql and all processes for maas00 and the same issue continues to happen.

description: updated
Blake Rouse (blake-rouse)
Changed in maas:
status: Triaged → In Progress
MAAS Lander (maas-lander)
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.