MAAS

[master] Twisted seems to not handle disconnect from client correctly

Bug #1741913 reported by Blake Rouse
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Medium
Dougal Matthews
MAAS 2.9.0b8
twisted (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

[Impact]

Web http/wsgi doesn't fail gracefully when the client closes the connection, causing 'Traceback' in MAAS /var/log/maas/rackd.log as follows:

....
   File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException
     raise self.value.with_traceback(self.tb)
 builtins.AttributeError: 'NoneType' object has no attribute 'writeHeaders'
....
   File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1474, in loseConnection
     self.channel.loseConnection()
 builtins.AttributeError: 'NoneType' object has no attribute 'loseConnection'
....

[Test Case]

[Regression Potential]

[Other Information]

* Upstream bug:
https://twistedmatrix.com/trac/ticket/9410

* Upstream fix:
https://github.com/twisted/twisted/commit/169fd1d93b7af06bf0f6893b193ce19970881868

# git describe --contains 169fd1d93b
twisted-19.7.0~20^2~13

# rmadison
 => python3-twisted | 17.9.0-2ubuntu0.1 | bionic-updates | all
 => python3-twisted | 18.9.0-11 | focal | all
 => python3-twisted | 18.9.0-11 | groovy | all

[Original Description]

When latest master just moving around the UI caused a wierd stacktrace. It didn't seem to affect the experience of using MAAS, but needs to be looked into.

2018-01-08 15:33:45 -: [critical] WSGI application error
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 87, in callWithContext
     self.contexts.pop()
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 522, in run
     self.started = True
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 500, in run
     self.write(elem)
   File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 455, in write
     self.reactor, wsgiWrite, self.started)
   File "/usr/lib/python3/dist-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
     result.raiseException()
   File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException
     raise self.value.with_traceback(self.tb)
 builtins.AttributeError: 'NoneType' object has no attribute 'writeHeaders'

2018-01-08 15:33:45 -: [critical] WSGI application error
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 87, in callWithContext
     self.contexts.pop()
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 522, in run
     self.started = True
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 500, in run
     self.write(elem)
   File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 455, in write
     self.reactor, wsgiWrite, self.started)
   File "/usr/lib/python3/dist-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
     result.raiseException()
   File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException
     raise self.value.with_traceback(self.tb)
 builtins.AttributeError: 'NoneType' object has no attribute 'writeHeaders'

2018-01-08 15:33:45 -: [critical] Unhandled Error
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 396, in startReactor
     self.config, oldstdout, oldstderr, self.profiler, reactor)
   File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 311, in runReactorWithLogging
     reactor.run()
   File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1243, in run
     self.mainLoop()
   File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1252, in mainLoop
     self.runUntilCurrent()
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 851, in runUntilCurrent
     f(*a, **kw)
   File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 510, in wsgiError
     self.request.loseConnection()
   File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1474, in loseConnection
     self.channel.loseConnection()
 builtins.AttributeError: 'NoneType' object has no attribute 'loseConnection'

2018-01-08 15:33:45 -: [critical] Unhandled Error
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 396, in startReactor
     self.config, oldstdout, oldstderr, self.profiler, reactor)
   File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 311, in runReactorWithLogging
     reactor.run()
   File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1243, in run
     self.mainLoop()
   File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1252, in mainLoop
     self.runUntilCurrent()
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 851, in runUntilCurrent
     f(*a, **kw)
   File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 510, in wsgiError
     self.request.loseConnection()
   File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1474, in loseConnection
     self.channel.loseConnection()
 builtins.AttributeError: 'NoneType' object has no attribute 'loseConnection'

See original description
Tags: sts seg

Related branches

~d0ugal/maas:twisted-patch-disconnect
Adam Collard (community): Approve
MAAS Lander: Approve
Diff: 105 lines (+73/-1)
2 files modified
src/maasserver/monkey.py (+33/-1)
src/maasserver/tests/test_monkey.py (+40/-0)
Revision history for this message
Adam Collard (adam-collard) wrote :

This bug has not seen any activity in the last 6 months, so it is being automatically closed.

If you are still experiencing this issue, please feel free to re-open.

MAAS Team

Changed in maas:
status: Triaged → Invalid
Revision history for this message
Eric Desrochers (slashd) wrote :

Same situations observed on MAAS (DEB install):

ii maas-cli 2.7.1-8262-g.9a49af7df-0ubuntu1~18.04.1 all MAAS client and command-line interface
ii maas-common 2.7.1-8262-g.9a49af7df-0ubuntu1~18.04.1 all MAAS server common files
ii maas-dhcp 2.7.1-8262-g.9a49af7df-0ubuntu1~18.04.1 all MAAS DHCP server
ii maas-proxy 2.7.1-8262-g.9a49af7df-0ubuntu1~18.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.7.1-8262-g.9a49af7df-0ubuntu1~18.04.1 all Rack Controller for MAAS
ii maas-region-api 2.7.1-8262-g.9a49af7df-0ubuntu1~18.04.1 all Region controller A

Trace:
2020-10-11 06:25:54 -: [critical] WSGI application error
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 87, in callWithContext
self.contexts.pop()
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 885, in callInContext
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 522, in run
self.started = True
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 500, in run
self.write(elem)
File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 455, in write
self.reactor, wsgiWrite, self.started)
File "/usr/lib/python3/dist-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
result.raiseException()
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException
raise self.value.with_traceback(self.tb)
builtins.AttributeError: 'NoneType' object has no attribute 'writeHeaders'

2020-10-11 06:25:54 asyncio: [error] Exception in callback <function AsyncioSelectorReactor.callLater.<locals>.run at 0x7fda5ce7be18>
handle: <Handle AsyncioSelectorReactor.callLater.<locals>.run>
Traceback (most recent call last):
File "uvloop/cbhandles.pyx", line 47, in uvloop.loop.Handle._run
File "/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 290, in run
f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 510, in wsgiError
self.request.loseConnection()
File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1474, in loseConnection
self.channel.loseConnection()
AttributeError: 'NoneType' object has no attribute 'loseConnection'
...

Changed in maas:
status: Invalid → New
Revision history for this message
Eric Desrochers (slashd) wrote :

https://twistedmatrix.com/trac/ticket/9410

Revision history for this message
Eric Desrochers (slashd) wrote :

https://github.com/twisted/twisted/commit/169fd1d93b7af06bf0f6893b193ce19970881868

# git describe --contains 169fd1d93b
twisted-19.7.0~20^2~13

# rmadison
 python3-twisted | 17.9.0-2ubuntu0.1 | bionic-updates | all
 python3-twisted | 18.9.0-11 | focal | all
 python3-twisted | 18.9.0-11 | groovy | all

Revision history for this message
Eric Desrochers (slashd) wrote :

It also seems like this has the effect for the impacted user I'm currently helping to always have this message in the UI:

"X rack controllers are not yet connected to the region. Visit the rack controllers page for more information".

Eric Desrochers (slashd)
tags: added: seg sts
Eric Desrochers (slashd)
Changed in twisted (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in twisted (Ubuntu):
status: New → Confirmed
Revision history for this message
Eric Desrochers (slashd) wrote :

# Test package available on my PPA:

sudo add-apt-repository ppa:slashd/sf294743
sudo apt-get update

Eric Desrochers (slashd)
description: updated
description: updated
Revision history for this message
Eric Desrochers (slashd) wrote :

With this test package I want to see if the 'traceback' will disapear and if this will also solve the MAAS UI telling "X rack controllers are not yet connected to the region. Visit the rack controllers page for more information" despite that everything else look good (network, firewall rules, nc test, ...)

With that test package we will be able to see if it's 2 separate situations or if they have this bug in common as the culprit.

Revision history for this message
Adam Collard (adam-collard) wrote :

@MAAS - let's monkey patch Twisted with the fix from upstream

Changed in maas:
milestone: 2.4.x → 2.9.0b7
Adam Collard (adam-collard)
Changed in maas:
status: New → Triaged
Lee Trager (ltrager)
Changed in maas:
milestone: 2.9.0b7 → 2.9.0b8
Dougal Matthews (d0ugal)
Changed in maas:
assignee: nobody → Dougal Matthews (d0ugal)
MAAS Lander (maas-lander)
Changed in maas:
status: Triaged → Fix Committed
Lee Trager (ltrager)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.