TestTCPServerInAThread.test_server_crash_while_responding random failure

[Martin Packman]

PQM just failed to land a branch with no bzrlib changes, with a single test error on:

bt.test_test_server.TestTCPServerInAThread.test_server_crash_while_responding(TestingThreadingTCPServer)

Two traceback attachments, the first from test case failing the assertion because FailedToRespond is not raised:

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/testtools/runtest.py", line 144, in _run_user
    return fn(*args)
  File "/usr/lib/python2.6/dist-packages/testtools/testcase.py", line 465, in _run_test_method
    testMethod()
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_test_server.py", line 224, in test_server_crash_while_responding
    self.assertRaises(FailToRespond, server.pending_exception)
AssertionError: FailToRespond not raised

The second from run cleanups where that exception belatedly pops up:

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/testtools/testcase.py", line 178, in _runCleanups
    function(*arguments, **keywordArguments)
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_server.py", line 575, in stop_server
    self._server_thread.join()
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_server.py", line 256, in join
    super(TestThread, self).join(timeout)
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/cethread.py", line 126, in run
    super(CatchingExceptionThread, self).run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_server.py", line 538, in run_server
    self.server.serve()
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_server.py", line 299, in serve
    self.handle_request()
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_server.py", line 316, in handle_request
    self.handle_error(request, client_address)
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_server.py", line 314, in handle_request
    self.process_request(request, client_address)
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_server.py", line 467, in process_request
    t.pending_exception()
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/cethread.py", line 154, in pending_exception
    self.join(timeout=0)
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_server.py", line 256, in join
    super(TestThread, self).join(timeout)
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/cethread.py", line 126, in run
    super(CatchingExceptionThread, self).run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_server.py", line 443, in process_request_thread
    self, request, client_address)
  File "/usr/lib/python2.6/SocketServer.py", line 561, in process_request_thread
    self.handle_error(request, client_address)
  File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
    self.handle()
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_test_server.py", line 84, in handle
    self.handle_connection()
  File "/home/pqm/pqm-workdir/srv/+trunk/bzrlib/tests/test_test_server.py", line 215, in handle_connection
    raise FailToRespond()
FailToRespond

[Vincent Ladeuil (vila)]

In a nutshell:

So the fix is that we should check both the connection thread and the server
thread with some care:

- first the connection thread, if we get the exception, we're good,

- if it's not in the connection thread, we need to wait for the server to
  stop before getting... our precious.