listdir testfails with LANG=fr_CH.UTF-8

[Jelmer Vernooij]

bzrlib.tests.per_transport.TransportTests.test_list_dir_result_is_url_escaped and various other tests fail only when LANG=fr_CH.UTF-8 is set *and* translations are installed (suggesting this is caused by translations).

ERROR: bzrlib.tests.per_transport.TransportTests.test_list_dir(SFTPTransport,SFTPHomeDirServer)
----------------------------------------------------------------------
Traceback (most recent call last):
testtools.testresult.real._StringException: log: {{{
DEBUG [chan bzr SocketAsChannelAdapter] Started sftp server on channel <bzrlib.transport.ssh.SocketAsChannelAdapter object at 0x7fc45d35c5d0>
   DEBUG [chan bzr SocketAsChannelAdapter] Request: opendir
   DEBUG [chan bzr SocketAsChannelAdapter] Request: readdir
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: open
   DEBUG [chan bzr SocketAsChannelAdapter] Request: write
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: open
   DEBUG [chan bzr SocketAsChannelAdapter] Request: write
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: open
   DEBUG [chan bzr SocketAsChannelAdapter] Request: write
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: mkdir
   DEBUG [chan bzr SocketAsChannelAdapter] Request: open
   DEBUG [chan bzr SocketAsChannelAdapter] Request: write
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: open
   DEBUG [chan bzr SocketAsChannelAdapter] Request: write
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: mkdir
   DEBUG [chan bzr SocketAsChannelAdapter] Request: opendir
   DEBUG [chan bzr SocketAsChannelAdapter] Request: readdir
   DEBUG [chan bzr SocketAsChannelAdapter] Exception in server processing: Unknown type
   DEBUG [chan bzr SocketAsChannelAdapter] Traceback (most recent call last):
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/sftp_server.py", line 111, in start_subsystem
   DEBUG [chan bzr SocketAsChannelAdapter] self._process(t, request_number, msg)
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/sftp_server.py", line 386, in _process
   DEBUG [chan bzr SocketAsChannelAdapter] self._read_folder(request_number, folder)
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/sftp_server.py", line 239, in _read_folder
   DEBUG [chan bzr SocketAsChannelAdapter] msg.add_string(attr)
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/message.py", line 285, in add_string
   DEBUG [chan bzr SocketAsChannelAdapter] s = asbytes(s)
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/common.py", line 170, in asbytes
   DEBUG [chan bzr SocketAsChannelAdapter] raise Exception('Unknown type')
   DEBUG [chan bzr SocketAsChannelAdapter] Exception: Unknown type
   DEBUG [chan bzr SocketAsChannelAdapter]
   DEBUG [chan bzr SocketAsChannelAdapter] EOF -- end of session
}}}

Traceback (most recent call last):
  File "/build/bzr-2.6.0+bzr6606/build/lib.linux-x86_64-2.7/bzrlib/tests/per_transport.py", line 1155, in test_list_dir
    ['a', 'a%2525b', 'b', 'c', 'c2'], sorted_list('', t))
  File "/build/bzr-2.6.0+bzr6606/build/lib.linux-x86_64-2.7/bzrlib/tests/per_transport.py", line 1139, in sorted_list
    l = list(transport.list_dir(d))
  File "/build/bzr-2.6.0+bzr6606/build/lib.linux-x86_64-2.7/bzrlib/transport/sftp.py", line 787, in list_dir
    self._translate_io_exception(e, path, ': failed to list_dir')
  File "/build/bzr-2.6.0+bzr6606/build/lib.linux-x86_64-2.7/bzrlib/transport/sftp.py", line 698, in _translate_io_exception
    raise failure_exc(path, str(e) + more_info)
bzrlib.errors.PathError: Generic path error: '': Failure: failed to list_dir)

======================================================================
ERROR: bzrlib.tests.per_transport.TransportTests.test_copy_tree_to_transport(SFTPTransport,SFTPHomeDirServer)
----------------------------------------------------------------------
Traceback (most recent call last):
testtools.testresult.real._StringException: log: {{{
DEBUG [chan bzr SocketAsChannelAdapter] Started sftp server on channel <bzrlib.transport.ssh.SocketAsChannelAdapter object at 0x7fc455259e10>
   DEBUG [chan bzr SocketAsChannelAdapter] Request: mkdir
   DEBUG [chan bzr SocketAsChannelAdapter] Request: mkdir
   DEBUG [chan bzr SocketAsChannelAdapter] Request: open
   DEBUG [chan bzr SocketAsChannelAdapter] Request: write
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: open
   DEBUG [chan bzr SocketAsChannelAdapter] Request: write
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: open
   DEBUG [chan bzr SocketAsChannelAdapter] Request: write
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: open
   DEBUG [chan bzr SocketAsChannelAdapter] Request: write
   DEBUG [chan bzr SocketAsChannelAdapter] Request: close
   DEBUG [chan bzr SocketAsChannelAdapter] Request: mkdir
   DEBUG [chan bzr SocketAsChannelAdapter] Request: opendir
   DEBUG [chan bzr SocketAsChannelAdapter] Request: readdir
   DEBUG [chan bzr SocketAsChannelAdapter] Exception in server processing: Unknown type
   DEBUG [chan bzr SocketAsChannelAdapter] Traceback (most recent call last):
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/sftp_server.py", line 111, in start_subsystem
   DEBUG [chan bzr SocketAsChannelAdapter] self._process(t, request_number, msg)
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/sftp_server.py", line 386, in _process
   DEBUG [chan bzr SocketAsChannelAdapter] self._read_folder(request_number, folder)
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/sftp_server.py", line 239, in _read_folder
   DEBUG [chan bzr SocketAsChannelAdapter] msg.add_string(attr)
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/message.py", line 285, in add_string
   DEBUG [chan bzr SocketAsChannelAdapter] s = asbytes(s)
   DEBUG [chan bzr SocketAsChannelAdapter] File "/usr/lib/python2.7/dist-packages/paramiko/common.py", line 170, in asbytes
   DEBUG [chan bzr SocketAsChannelAdapter] raise Exception('Unknown type')
   DEBUG [chan bzr SocketAsChannelAdapter] Exception: Unknown type
   DEBUG [chan bzr SocketAsChannelAdapter]
   DEBUG [chan bzr SocketAsChannelAdapter] EOF -- end of session
}}}

Traceback (most recent call last):
  File "/build/bzr-2.6.0+bzr6606/build/lib.linux-x86_64-2.7/bzrlib/tests/per_transport.py", line 1517, in test_copy_tree_to_transport
    from_transport.copy_tree_to_transport(to_transport)
  File "/build/bzr-2.6.0+bzr6606/build/lib.linux-x86_64-2.7/bzrlib/transport/__init__.py", line 1095, in copy_tree_to_transport
    for path in self.list_dir(dir):
  File "/build/bzr-2.6.0+bzr6606/build/lib.linux-x86_64-2.7/bzrlib/transport/sftp.py", line 787, in list_dir
    self._translate_io_exception(e, path, ': failed to list_dir')
  File "/build/bzr-2.6.0+bzr6606/build/lib.linux-x86_64-2.7/bzrlib/transport/sftp.py", line 698, in _translate_io_exception
    raise failure_exc(path, str(e) + more_info)
bzrlib.errors.PathError: Generic path error: 'from': Failure: failed to list_dir)

[Jelmer Vernooij (jelmer)]

argh, so paramiko has this gem:

def asbytes(s):
    if not isinstance(s, bytes_types):
        if isinstance(s, string_types):
            s = b(s)
        else:
            try:
                s = s.asbytes()
            except Exception:
                raise Exception('Unknown type')
    return s

Which means that if "asbytes()" raises exceptions for one reason or another, like "Exception: Unknown type <SFTPAttributes: [ size=16 uid=1000 gid=1000 mode=0100644 atime=1449250574 mtime=1449250574 ]>: UnicodeDecodeError('ascii', '-rw-r--r-- 1 1000 1000 16 04 d\xc3\xa9c 17:36 ', 46, 47, 'ordinal not in range(128)')" it gets silently hidden :(

[Martin Packman (gz)]

Filed downstream as <https://github.com/paramiko/paramiko/issues/985> and proposed a fix there.