[2.0b4] twisted.internet.error.ConnectionDone: Connection was closed cleanly.

Bug #1580817 reported by Andres Rodriguez on 2016-05-12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gavin Panella

Bug Description

1. updated the ephemeral image repo to use daily
2. maas webui lagged for a sec.
3. controller status showed yellow (warning)
4. saw this error.

I don't know whether this is reproduceable.

==> /var/log/maas/rackd.log <==
2016-05-12 08:00:22+0800 [ClusterClient,client] ClusterClient connection lost (HOST:IPv4Address(TCP, '', 57846) PEER:IPv4Address(T
CP, '', 5252))
2016-05-12 08:00:22+0800 [ClusterClient,client] Querying node power states.
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in errback
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
            _inlineCallbacks(r, g, deferred)
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python3/dist-packages/provisioningserver/pserv_services/node_power_monitor_service.py", line 64, in query_nodes
            ListNodePowerParameters, uuid=client.localIdent)
        twisted.internet.error.ConnectionDone: Connection was closed cleanly.

==> /var/log/maas/maas.log <==
May 12 08:00:22 trusty-maas9 maas.power_monitor_service: [ERROR] Failed to query nodes' power status: Connection was closed cleanly.

Related branches

description: updated
Changed in maas:
milestone: none → 2.1.0
milestone: 2.1.0 → 2.0.0
importance: Undecided → Critical
description: updated
summary: - twisted.internet.error.ConnectionDone: Connection was closed cleanly.
+ [2.0b4] twisted.internet.error.ConnectionDone: Connection was closed
+ cleanly.
Lee Trager (ltrager) wrote :

I wasn't able to reproduce this. MAAS periodically checks the power status of all nodes a rack controller has access to. When you switched boot sources from releases to daily this triggered new images to be downloaded. This action may of starved some resource on the system enough that the RPC action timed out and the connection was closed.

Changed in maas:
status: New → Invalid
Changed in maas:
status: Invalid → Incomplete
Gavin Panella (allenap) wrote :

I think this happens when the rack loses its connection to the region while looping in NodePowerMonitorService.query_nodes(). It's not critical at all, but I'll improve the error message.

Changed in maas:
status: Incomplete → In Progress
importance: Critical → Medium
assignee: nobody → Gavin Panella (allenap)
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers