Twisted tests failing due to port conflict

[Fernando Perez]

With my fixes in trunk-dev we now have a test suite that passes again, except for a ton of failures in the twisted tests. But I think they all come from the same problem, because the traceback is identical in all failures:

===============================================================================
[ERROR]: IPython.kernel.tests.test_taskfc.TaskTest.test_task_ids

Traceback (most recent call last):
  File "/home/fperez/usr/lib/python2.6/site-packages/IPython/kernel/tests/test_taskfc.py", line 70, in setUp
    self.controller_tub.startService()
  File "/usr/lib/pymodules/python2.6/foolscap/pb.py", line 594, in startService
    service.MultiService.startService(self)
  File "/usr/lib/python2.6/dist-packages/twisted/application/service.py", line 278, in startService
    service.startService()
  File "/usr/lib/python2.6/dist-packages/twisted/application/internet.py", line 91, in startService
    self._port = self._getPort()
  File "/usr/lib/python2.6/dist-packages/twisted/application/internet.py", line 116, in _getPort
    *self.args, **self.kwargs)
  File "/usr/lib/python2.6/dist-packages/twisted/internet/posixbase.py", line 356, in listenTCP
    p.startListening()
  File "/usr/lib/python2.6/dist-packages/twisted/internet/tcp.py", line 858, in startListening
    raise CannotListenError, (self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on 127.0.0.1:10105: [Errno 98] Address already in use.
-------------------------------------------------------------------------------

Brian, if you can pitch in for this one, we might get trunk into a testable state again. Thanks!

[Brian Granger (ellisonbg)]

> With my fixes in trunk-dev we now have a test suite that passes again,
> except for a ton of failures in the twisted tests.  But I think they all
> come from the same problem, because the traceback is identical in all
> failures:

Wow, that is great. How did you fix everything? I barely even
remember the state of the testing system, I just remember it wasn't
working? Did you incorporate the testing approaches we came up with
and you implemented previously?

Can you commit everything in your branch - I will check it out and
debug this one using your branch. How should I run the test suite?

Ohh, one silly question that might resolve this....

Do you have ipcontroller running somewhere else on your local system?
If so stop it and rerun the test suite.

This is great though!

Brian

> ===============================================================================
> [ERROR]: IPython.kernel.tests.test_taskfc.TaskTest.test_task_ids
>
> Traceback (most recent call last):
>  File "/home/fperez/usr/lib/python2.6/site-packages/IPython/kernel/tests/test_taskfc.py", line 70, in setUp
>    self.controller_tub.startService()
>  File "/usr/lib/pymodules/python2.6/foolscap/pb.py", line 594, in startService
>    service.MultiService.startService(self)
>  File "/usr/lib/python2.6/dist-packages/twisted/application/service.py", line 278, in startService
>    service.startService()
>  File "/usr/lib/python2.6/dist-packages/twisted/application/internet.py", line 91, in startService
>    self._port = self._getPort()
>  File "/usr/lib/python2.6/dist-packages/twisted/application/internet.py", line 116, in _getPort
>    *self.args, **self.kwargs)
>  File "/usr/lib/python2.6/dist-packages/twisted/internet/posixbase.py", line 356, in listenTCP
>    p.startListening()
>  File "/usr/lib/python2.6/dist-packages/twisted/internet/tcp.py", line 858, in startListening
>    raise CannotListenError, (self.interface, self.port, le)
> twisted.internet.error.CannotListenError: Couldn't listen on 127.0.0.1:10105: [Errno 98] Address already in use.
> -------------------------------------------------------------------------------
>
> Brian, if you can pitch in for this one, we might get trunk into a
> testable state again.  Thanks!
>
> ** Affects: ipython
>     Importance: High
>         Status: Confirmed
>
> --
> Twisted tests failing due to port conflict
> https://bugs.launchpad.net/bugs/504515
> You received this bug notification because you are a member of IPython
> Developers, which is subscribed to IPython.
>
> Status in IPython - Enhanced Interactive Python: Confirmed
>
> Bug description:
> With my fixes in trunk-dev we now have a test suite that passes again, except for a ton of failures in the twisted tests.  But I think they all come from the same problem, because the traceback is identical in all failures:
>
> ===============================================================================
> [ERROR]: IPython.kernel.tests.test_taskfc.TaskTest.test_task_ids
>
> Traceback (most recent call last):
>  File "/home/fperez/usr/lib/python2.6/site-packages/IPython/kernel/tests/test_taskfc.py", line 70, in setUp
>    self.controller_tub.startService()
>  File "/usr/lib/pymodules...

[Fernando Perez (fdo.perez)]

Hey,

On Thu, Jan 7, 2010 at 4:13 PM, Brian Granger <email address hidden> wrote:
> Wow, that is great.  How did you fix everything?  I barely even
> remember the state of the testing system, I just remember it wasn't
> working?  Did you incorporate the testing approaches we came up with
> and you implemented previously?

Lots of work :) Yes, I did add the new testing stuff (better
parametric tests for example). Not all of it is cleanly integrated
everywhere, but all of it is committed now and usable.

> Can you commit everything in your branch - I will check it out and
> debug this one using your branch.  How should I run the test suite?

Done, I just pushed a second ago.

> Ohh, one silly question that might resolve this....
>
> Do you have ipcontroller running somewhere else on your local system?
> If so stop it and rerun the test suite.

Actually yes! So things are even better than we thought, though we do
have a small (different) issue. The reason I had a leftover
controller was because earlier today I had to stop a hung test suite
and the trial process stayed around. This test often (but not always)
hangs:

IPython.kernel.tests.test_multienginefc
  FullSynchronousMultiEngineTestCase
    testClearProperties ...

Do you know why? I guess now what we need to do is figure this one
out instead, because the port problems were all due to the stale trial
process.

One more thing though: in addition to this, perhaps we should try to
find the next available port instead of relying on a hardcoded one?
This would make it easier to run the test suite even if 10105 is in
use...

> This is great though!

Yes, I think we're getting close :) Thanks!

f

[Fernando Perez (fdo.perez)]

On Thu, Jan 7, 2010 at 4:28 PM, Fernando Perez <email address hidden> wrote:
> Actually yes!  So things are even better than we thought, though we do
> have a small (different) issue.  The reason I had a leftover
> controller was because earlier today I had to stop a hung test suite
> and the trial process stayed around.  This test often (but not always)
> hangs:
>
> IPython.kernel.tests.test_multienginefc
>  FullSynchronousMultiEngineTestCase
>    testClearProperties ...
>
> Do you know why?  I guess now what we need to do is figure this one
> out instead, because the port problems were all due to the stale trial
> process.

BTW, I just pushed r1258 to my branch which adds stale subprocess
cleanup when possible, to mitigate this problem; I checked and it does
help on my system by killing the stale trial processes. The mystery
of why testClearProperties hangs often remains though...

f

[Fernando Perez (fdo.perez)]

Lowered priority, but just to record the conversation Brian and I had: an easy partial fix Brian suggested is just to have the test suite use a different port from the default. This is easy to do and will help a lot.

Once we do that and with my already committed subprocess cleanup, this can then be closed.