SSH backend paramiko 1.12.0 presents truncated array of volumes

[Jonathan Elchison]

NOTE: This is a paramiko bug, not a Duplicity bug. This report has been created at edso's request to save other Duplicity users the trouble.

Symptoms:
1. Backup operation succeeds (presumably with > 1287 volumes)
2. 'collection-status' operation...
** ...only reports 1287 volumes
** ...deletes local cache (sigtar) due to "not authoritative at backend"
** ... finds no primary backup chains
3. 'list-current-files' operation excepts with "CollectionsError: No signature chains found"
4. Restore operation also fails

Symptoms 3 and 4 are presumably because of symptom 2.

Link to original question: https://answers.launchpad.net/duplicity/+question/241395

Link to paramiko bug: https://github.com/paramiko/paramiko/issues/246

This bug report should be closed when the paramiko bug is closed.

[Kenneth Loafman (kenneth-loafman)]

As a workaround please try pexpect+sftp:// instead of paramiko. Let me know how that goes.

[Jonathan Elchison (jelchison)]

Your recommended workaround was successful as of 2014-01-02 (I assume nothing has changed since then). For full context, please see #6 on https://answers.launchpad.net/duplicity/+question/241395.

[Matthias Larisch (matthiaslarisch)]

This bug is not a paramiko bug but a usage bug in duplicity.

Using duplicity 0.7.12, I can confirm that the list of files are truncated in paramiko+scp backend.

The problem is, that in paramiko backend runremote() returns recv(-1) which does not return _all_ data but only _all available_ data.

Internet sources I briefly read:
http://jessenoller.com/blog/2009/02/05/ssh-programming-with-paramiko-completely-different
https://stackoverflow.com/questions/14643861/paramiko-channel-stucks-when-reading-large-ouput

Both make me think that just using readlines() in runremote(), so directly returning an array of strings (filenames in the case of _list()), should make it work.

I will try in some time as I first need to get my backup, currently using paramiko+sftp which is not affected of runremote() bug.

[Matthias Larisch (matthiaslarisch)]

Appended patch uses higher level channel handling in paramiko that uses the BufferedFile implementation that internally handles collecting all output.

Confirmed it working in my case at least :-)

[edso (ed.so)]

hey Matthias,

wouldn't you have to check *ch_err* before assuming that *ch_out* is actually a readable object?

..ede/duply.mnet

On 23.05.2017 12:05, Matthias Larisch wrote:
> Appended patch uses higher level channel handling in paramiko that uses
> the BufferedFile implementation that internally handles collecting all
> output.
>
> Confirmed it working in my case at least :-)
>
> ** Patch added: "paramiko_ssh_not_truncate_command_output.patch"
> https://bugs.launchpad.net/duplicity/+bug/1265765/+attachment/4881879/+files/paramiko_ssh_not_truncate_command_output.patch
>

[edso (ed.so)]

ok, my bad
  http://docs.paramiko.org/en/2.1/api/client.html#paramiko.client.SSHClient.exec_command
is like a local system exec and returns STDIN,OUT,ERR io streams as file objects

hence it should probably look like (warning not tested)

    def runremote(self, cmd, ignoreexitcode=False, errorprefix=""):
        """small convenience function that opens a shell channel, runs remote
        command and returns stdout of command. throws an exception if exit
        code!=0 and not ignored"""
        try:
            ch_in, ch_out, ch_err = self.client.exec_command(cmd, -1, globals.timeout)
            output = ch_out.read(-1)
            return output
        except Exception as e:
            if not ignoreexitcode:
                raise BackendException("%sfailed: %s \n %s" % (errorprefix,
                                                         ch_err.read(-1), e))

ignoreexitcode seems to be False currently for all runremote calls, but it does not hurt to leave it there.

d'accord? would you mind testing something along these lines? i am currently not set up.

..ede/duply.net

On 23.05.2017 12:27, edso wrote:
> hey Matthias,
>
> wouldn't you have to check *ch_err* before assuming that *ch_out* is
> actually a readable object?
>
> ..ede/duply.mnet
>
>
> On 23.05.2017 12:05, Matthias Larisch wrote:
>> Appended patch uses higher level channel handling in paramiko that uses
>> the BufferedFile implementation that internally handles collecting all
>> output.
>>
>> Confirmed it working in my case at least :-)
>>
>> ** Patch added: "paramiko_ssh_not_truncate_command_output.patch"
>> https://bugs.launchpad.net/duplicity/+bug/1265765/+attachment/4881879/+files/paramiko_ssh_not_truncate_command_output.patch
>>
>

[Kenneth Loafman (kenneth-loafman)]

This seems to be working for me. I tested against a backup with over 2,000
volumes and it worked. Will get it committed soon.

On Tue, May 23, 2017 at 6:21 AM, edso <email address hidden> wrote:

> ok, my bad
> http://docs.paramiko.org/en/2.1/api/client.html#paramiko.
> client.SSHClient.exec_command
> is like a local system exec and returns STDIN,OUT,ERR io streams as file
> objects
>
> hence it should probably look like (warning not tested)
>
> def runremote(self, cmd, ignoreexitcode=False, errorprefix=""):
> """small convenience function that opens a shell channel, runs
> remote
> command and returns stdout of command. throws an exception if exit
> code!=0 and not ignored"""
> try:
> ch_in, ch_out, ch_err = self.client.exec_command(cmd, -1,
> globals.timeout)
> output = ch_out.read(-1)
> return output
> except Exception as e:
> if not ignoreexitcode:
> raise BackendException("%sfailed: %s \n %s" % (errorprefix,
> ch_err.read(-1),
> e))
>
> ignoreexitcode seems to be False currently for all runremote calls, but
> it does not hurt to leave it there.
>
> d'accord? would you mind testing something along these lines? i am
> currently not set up.
>
> ..ede/duply.net
>
> On 23.05.2017 12:27, edso wrote:
> > hey Matthias,
> >
> > wouldn't you have to check *ch_err* before assuming that *ch_out* is
> > actually a readable object?
> >
> > ..ede/duply.mnet
> >
> >
> > On 23.05.2017 12:05, Matthias Larisch wrote:
> >> Appended patch uses higher level channel handling in paramiko that uses
> >> the BufferedFile implementation that internally handles collecting all
> >> output.
> >>
> >> Confirmed it working in my case at least :-)
> >>
> >> ** Patch added: "paramiko_ssh_not_truncate_command_output.patch"
> >> https://bugs.launchpad.net/duplicity/+bug/1265765/+
> attachment/4881879/+files/paramiko_ssh_not_truncate_command_output.patch
> >>
> >
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1265765
>
> Title:
> SSH backend paramiko 1.12.0 presents truncated array of volumes
>
> Status in Duplicity:
> Confirmed
>
> Bug description:
> NOTE: This is a paramiko bug, not a Duplicity bug. This report has
> been created at edso's request to save other Duplicity users the
> trouble.
>
> Symptoms:
> 1. Backup operation succeeds (presumably with > 1287 volumes)
> 2. 'collection-status' operation...
> ** ...only reports 1287 volumes
> ** ...deletes local cache (sigtar) due to "not authoritative at backend"
> ** ... finds no primary backup chains
> 3. 'list-current-files' operation excepts with "CollectionsError: No
> signature chains found"
> 4. Restore operation also fails
>
> Symptoms 3 and 4 are presumably because of symptom 2.
>
> Link to original question:
> https://answers.launchpad.net/duplicity/+question/241395
>
> Link to paramiko bug: https://github.com/paramiko/paramiko/issues/246
>
> This bug report should be closed when the paramiko bug is closed.
>
> To manage notif...

[Matthias Larisch (matthiaslarisch)]

Sorry, I did not get your updates until now.

First: Thanks for merging/fixing the original issue. But I see a problem with the patch how it is implemented: recv() will not throw an exception when the file gets closed, so the exitcode might never be read.

This is not a problem as the exitcode is not used anyway, but that code is not necessary then as well...

Maybe leave out the exitcode stuff completely? I don't have time to finally fix it now.

[edso (ed.so)]

hey Matthias,

On 14.06.2017 12:30, Matthias Larisch wrote:
> Sorry, I did not get your updates until now.
>
> First: Thanks for merging/fixing the original issue. But I see a problem
> with the patch how it is implemented: recv() will not throw an exception
> when the file gets closed, so the exitcode might never be read.
>
> This is not a problem as the exitcode is not used anyway, but that code
> is not necessary then as well...
>
> Maybe leave out the exitcode stuff completely? I don't have time to
> finally fix it now.
>

currently there is no call to recv() anymore
  http://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/view/head:/duplicity/backends/ssh_paramiko_backend.py#L377

afaics does SSHClient.exec_command() not support returning the error code
  http://docs.paramiko.org/en/2.1/api/client.html#paramiko.client.SSHClient.exec_command
but throws an SSHException instead, which is properly handled now.

..ede/duply.net