tests.test_transport.TestSSHConnections.test_bzr_connect_to_bzr_ssh considered harmful
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | Bazaar |
Critical
|
John A Meinel | ||
Bug Description
This test:
- has failed intermittently for months on babune across almost all platforms,
- contains several XXX and one FIXME,
- is huge
- is blocking PQM as of today
My gut feeling is that it tries to test too much at once and should be split.
It may also be related to bug #579530 fixed by spiv in a private ppa
- fails on hardy/py2.
Traceback (most recent call last):
File "/home/
return fn(*args)
File "/home/
testMethod()
File "/home/
t.mkdir('foo')
File "/home/
self.
File "/home/
return self._client.
File "/home/
result, protocol = self.call_
File "/home/
return self._call_
File "/home/
response_tuple = response_
File "/home/
self.
File "/home/
self.
File "/home/
raise errors.
ConnectionReset: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.
------------
- succeds on hardy/py2.
(this is a paramiko private thread):
Exception in thread Thread-6:
Traceback (most recent call last):
File "/usr/lib/
self.run()
File "/usr/lib/
self.
File "/home/
write(bytes)
File "/home/
raise socket.
error: Socket is closed
- succeeds on karmic/py2.4/ haha, skipped, no paramiko here :(
- fails on karmic/
- succeeds on karmic/
(note the switch between 2.4 and 2.5 when going from hardy to karmic)
Related branches
- Vincent Ladeuil: Approve on 2010-09-02
-
Diff: 112 lines (+31/-24)3 files modifiedNEWS (+3/-0)
bzrlib/tests/stub_sftp.py (+28/-22)
bzrlib/tests/test_transport.py (+0/-2)
| Changed in bzr: | |
| status: | New → Confirmed |
| Changed in bzr: | |
| importance: | Undecided → Critical |
| Vincent Ladeuil (vila) wrote : | #1 |
| Vincent Ladeuil (vila) wrote : | #2 |
succeeds on gentoo/
same warning as freebsd about paramiko
same traceback as hardy/py25 that can be ignored
| Vincent Ladeuil (vila) wrote : | #3 |
I think that's enough evidence that this test should be rewritten in a more granular way to better
capture the possible failures.
| John A Meinel (jameinel) wrote : | #4 |
So far, the only 'granularity' I've seen is to remove the 'mkdir' call. Basically, I feel that it *is* important to have a real connection to an ssh server in the test suite if we can. The test is a little bit long, but only because the ssh server code is inline in the test.
As such, there doesn't seem a lot that could be done. If it is an interaction with threads and sockets, etc, then we could arguably start an ssh server in a subprocess, connect to *that* and then shut it down afterwards. That might provide better *isolation* but I don't see it as a *granularity* thing.
| Vincent Ladeuil (vila) wrote : | #5 |
> If it is an interaction with threads and sockets
Let's look at what is involved in that test then:
This will start a thread for the server and since it's threading server, one for each connection. Each of them have a socket.
Since we're using paramiko, that also means one thread per socket as per paramiko implementation.
Then we have:
This will also create a socket and its paramiko thread.
When the server receives the connection, it will start a subprocess and 3 threads for stdin, stdout and stderr each of them
associated with a pipe, so 3 file handles in the parent process and 3 file handles in the server process.
Paramiko use a 0.1 timeout for handling his sockets adding a lot of possible failures.
I can see a lot a *granularity* to add to this design to better track where the failures occur.
Or, come to think of it, I think the design is just to brittle to continue with and we'd better start again from scratch :)
| John A Meinel (jameinel) wrote : | #6 |
I'm getting this on Windows w/ python 2.6 and paramiko 1.7.5. Sorry if there are formatting issues.
Note that this is ConnectionReset, but paramiko isn't handling the Windows failure code the same way as it seems to do on Linux. (Or *we* aren't translating it, or something).
=======
ERROR: bzrlib.
-------
_StringException: Text attachment: log
------------
1.862 failed to load system host keys: [Errno 2] No such file or directory: 'C:/users/
ata/local/
1.862 failed to load bzr host keys: [Errno 2] No such file or directory: 'C:/users/
/local/
1.862 creating config parent directory: 'C:/users/
.test_bzr_
1.862 creating config directory: 'C:/users/
zr_connect_
DEBUG starting thread (client mode): 0x279ca70L
INFO Connected (version 2.0, client paramiko_1.7.5)
DEBUG kex algos:[
r', 'aes256-ctr', 'aes128-cbc', 'blowfish-cbc', 'aes256-cbc', '3des-cbc', 'arcfour128', 'arcfour256'
] server encrypt:
, 'arcfour128', 'arcfour256'] client mac:['hmac-sha1', 'hmac-md5', 'hmac-sha1-96', 'hmac-md5-96'] se
rver mac:['hmac-sha1', 'hmac-md5', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none'] server co
mpress:['none'] client lang:[''] server lang:[''] kex follows?False
DEBUG Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
DEBUG using kex diffie-
emote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
DEBUG Switch to new keys ...
WARNING Adding ssh-rsa host key for localhost: 60733844CB51866
2.131 Trying SSH agent key 9EB9CEAB13B9027
DEBUG userauth is OK
ERROR Socket exception: An existing connection was forcibly closed by the remote host (10054)
2.340 opening working tree 'C:/users/
------------
Text attachment: traceback
------------
Traceback (most recent call last):
File "C:\Python26\
_run_user
return fn(*args)
File "C:\Python26\
_run_test_method
testMethod()
File "C:\Users\
nect_to_bzr_ssh
t.mkdir('foo')
File "C:\Users\
self.
File "C:\Users\
| Vincent Ladeuil (vila) wrote : | #7 |
And one more failure on xp/py2.
./bzr selftest -s bt.test_
bzr selftest: C:/cygwin/
C:\cygwin\
bzr-2.3.0dev1 python-2.5.4 Windows-
ERROR: test_transport.
Text attachment: log
------------
0.821 failed to load compiled extension: No module named _annotator_pyx
0.821 failed to load compiled extension: No module named _knit_load_data_pyx
0.851 failed to load compiled extension: No module named _known_graph_pyx
0.871 failed to load compiled extension: No module named _groupcompress_pyx
0.871 creating repository in file://
0.871 creating branch <bzrlib.
0.881 opening working tree 'C:/windows/
1.041 failed to load system host keys: [Errno 2] No such file or directory: 'C:/windows/
1.041 failed to load bzr host keys: [Errno 2] No such file or directory: 'C:/windows/
1.041 creating config parent directory: 'C:/windows/
1.041 creating config directory: 'C:/windows/
DEBUG starting thread (client mode): 0x155f390L
INFO Connected (version 2.0, client paramiko_1.7.6)
DEBUG kex algos:[
DEBUG Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
DEBUG using kex diffie-
DEBUG Switch to new keys ...
WARNING Adding ssh-rsa host key for localhost: 60733844CB51866
DEBUG userauth is OK
INFO Authentication (password) successful!
DEBUG [chan 1] Max packet in: 34816 bytes
DEBUG [chan 1] Max packet out: 34816 bytes
INFO Secsh channel 1 opened.
DEBUG [chan 1] Sesch channel 1 request ok
ERROR Socket exception: Connection reset by peer (10054)
1.662 opening working tree 'C:/windows/
------------
Text attachment: traceback
------------
Traceback (most recent call last):
File "\\VBOXSRV\
return fn(*args)
File "\\VBOXSRV\
| Vincent Ladeuil (vila) wrote : | #8 |
This currently skipped and should be re-enabled as fast as we can, hence the 'Critical' importance.
| Changed in bzr: | |
| status: | Confirmed → In Progress |
| assignee: | nobody → John A Meinel (jameinel) |
| status: | In Progress → Confirmed |
| status: | Confirmed → In Progress |
| John A Meinel (jameinel) wrote : | #9 |
The final fix is that we just weren't waiting for the paramiko.Transport to finish before we closed the server socket when the Handler returned.
| Changed in bzr: | |
| status: | In Progress → Fix Released |
| milestone: | none → 2.3b1 |
fails on freebsd/
Traceback (most recent call last):
File "/home/
return fn(*args)
File "/home/
testMethod()
File "/usr/home/
t.mkdir('foo')
File "/usr/home/
self.
File "/usr/home/
return self._client.
File "/usr/home/
result, protocol = self.call_
File "/usr/home/
method, args, expect_
File "/usr/home/
readv_
File "/usr/home/
encoder.
File "/usr/home/
self.
File "/usr/home/
self.flush()
File "/usr/home/
self.
File "/usr/home/
self.
File "/usr/home/
self.
File "/usr/home/
self.
File "/usr/home/
'--
File "/usr/home/
channel.
File "/usr/local/
self.
File "/usr/local/
raise e
EOFError
------------
Note that there is also a warning there:
/usr/local/
RandomPool_