SSHException("Channel closed.") thrown

[Patrick J. McNerthney]

My use paramiko via Fabric is failling intermittently. Every so often, the following exception occurs:

Traceback (most recent call last):
  File "/home/pmcnerthney/cpm/internal/cloud/management/fabric/main.py", line 480, in main
    commands[name](*args, **kwargs)
  File "/home/pmcnerthney/cpm/internal/cloud/management/storm.py", line 52, in rebuild
    run('sed -i "s/#alias /alias /g" .bashrc')
  File "/home/pmcnerthney/cpm/internal/cloud/management/fabric/network.py", line 381, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/home/pmcnerthney/cpm/internal/cloud/management/fabric/operations.py", line 418, in run
    channel.exec_command(real_command)
  File "/home/pmcnerthney/cpm/internal/cloud/management/paramiko/channel.py", line 212, in exec_command
    self._wait_for_event()
  File "/home/pmcnerthney/cpm/internal/cloud/management/paramiko/channel.py", line 1077, in _wait_for_event
    raise e
paramiko.SSHException: Channel closed.

This is occurring due to the change in behavior of the _wait_for_event method between versions 1.7.4 and 1.7.5. Before, _wait_for_event was returning whenever self.event.isSet() was true, regardless of the self.closed value. In version 1.7.5, the following sequence of events is now possible:

1. A request is made, clearing self.event.
2. The request completes, setting self.event.
3. The channel is closed, setting self.closed.
4. _wait_for_event is finally called, tests that self.closed is true, and raises SSHException('Channel closed.')

99 times out of 100, steps 3 and 4 happen in the opposite order, but if the thread swapping happens just right, the above occurs.

[Patrick J. McNerthney (pmcnerthney)]

Just in case the attached svn diff is hard to use, I have attached the complete channel.py source.

[bitprophet (jeff-bitprophet)]

I alluded to this over on Github, but this bug affects all Fabric users who are using, or may upgrade to, Paramiko 1.7.5, so I'd be ecstatic if Robey could check this out and push a 1.7.5.1 or 1.7.6 to PyPI sometime soon. I'm almost ready to release a new Fabric version, but would rather not do so if new users are going to be hit with this bug :)

[Pug (pfaffflo)]

I also get SSHException("Channel closed.") when I connect to an embedded NAS device using paramiko.

The strange things is:

 - it works when i connect for example to an linux host
 - exception thrown with paramiko 1.7.4
 - exception thrown with the patched channel.py (from above)

will attach the following:

1. log and traceback from paramiko 1.7.4
2. log and traceback from patched paramiko 1.7.5
3. log from ssh -vvv to both a linux host and the nas host ( did this to check if the nas device does something strange but personally I cannot see anything wrong)

[flxkid (oliver-ootbc)]

I can duplicate this bug 100% of the time on my ubuntu box. For me the "thread swapping" occurs 100% of the time in the code I use. reverted to the 1.7.4 _wait_for_event code. Should I actually move to the patched code given here?

[Augusto Becciu (augusto-becciu)]

the patch by Patrick J. McNerthney works like a charm for me.

[Todd Whiteman (toddw)]

This has also been reported to occur in Komodo when using Paramiko for SSH access:
http://bugs.activestate.com/show_bug.cgi?id=84740

The attached patch (#1) is reported to have solved this issue for a Komodo user.

[Robey Pointer (robey)]

patch 1 looks right to me. committing. sorry for the trouble.