[2.4] Registering RPC connections with master can occur out of order
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
Critical
|
Blake Rouse |
Bug Description
When a regiond process is reporting the RPC connections to a master process the order of actions is not deterministic being async. That means a new re-connection can tell the master the process has that connection and an old disconnection defer can later come back and say it has disconnected. This out of order is what causes the master to think the regiond process doesn't have that RPC connection, when in reality it actually does have that connection.
Since the rack controller actually does have that connection it will not perform an re-connection to fix this issue.
018-04-17 21:13:43 maasserver.
Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
current.result = callback(
File "/usr/lib/
_inlineCal
File "/usr/lib/
result = g.send(result)
--- <exception caught here> ---
File "/usr/lib/
yield self.initRespon
File "/usr/lib/
return func(*args, **kwargs)
File "/usr/lib/
self.host = self.transport.
builtins.
2018-04-17 21:13:43 maasserver.ipc: [info] Worker pid:22906 lost RPC connection to ('mar3yn', '10.244.40.32', 5250).
2018-04-17 21:13:43 maasserver.ipc: [info] Worker pid:22905 lost RPC connection to ('mar3yn', '10.244.40.32', 5252).
2018-04-17 21:13:43 provisioningser
2018-04-17 21:13:44 twisted.
2018-04-17 21:13:44 twisted.
2018-04-17 21:13:44 twisted.
2018-04-17 21:13:44 twisted.
2018-04-17 21:13:44 maasserver.
2018-04-17 21:13:44 maasserver.
2018-04-17 21:13:44 maasserver.
2018-04-17 21:13:44 maasserver.
2018-04-17 21:13:45 maasserver.ipc: [info] Worker pid:22906 registered RPC connection to ('mar3yn', '10.244.40.32', 5250).
2018-04-17 21:13:47 maasserver.ipc: [info] Worker pid:22907 registered RPC connection to ('mar3yn', '10.245.219.208', 5251).
2018-04-17 21:13:47 maasserver.ipc: [info] Worker pid:22905 registered RPC connection to ('mar3yn', '10.244.40.32', 5252).
2018-04-17 21:13:48 maasserver.ipc: [info] Worker pid:22904 registered RPC connection to ('mar3yn', '10.244.40.32', 5253).
2018-04-17 21:13:51 maasserver.ipc: [info] Worker pid:22904 lost RPC connection to ('mar3yn', '10.244.40.32', 5253).
Related branches
- Andres Rodriguez (community): Approve
-
Diff: 272 lines (+43/-36)4 files modifiedsrc/maasserver/ipc.py (+29/-29)
src/maasserver/rpc/regionservice.py (+5/-2)
src/maasserver/rpc/tests/test_regionservice.py (+3/-2)
src/maasserver/tests/test_ipc.py (+6/-3)
Changed in maas: | |
milestone: | 2.4.0beta3 → 2.4.0rc1 |
description: | updated |
Changed in maas: | |
status: | In Progress → Fix Committed |
Changed in maas: | |
milestone: | 2.4.0rc1 → 2.4.0beta3 |
Changed in maas: | |
status: | Fix Committed → Fix Released |