dput's sftp method is very slow

Bug #1814791 reported by Roman Shipovskij on 2019-02-05
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
dput (Ubuntu)
Low
Unassigned

Bug Description

Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic
dput: 1.0.1ubuntu1

New dput's sftp method is very slow, ~1.5Mbit/s, after some investigation I found that increasing buffer size up to 65536 increase upload speed up to ~8Mbit/s, but this is still much slower than old method.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in dput (Ubuntu):
status: New → Confirmed
tags: added: disco
Changed in dput (Ubuntu):
importance: Undecided → Low
Mathew Hodson (mhodson) on 2019-02-09
tags: added: patch
Julian Andres Klode (juliank) wrote :

We should probably also or only adjust the reads to be 64 KiB large. I'm not sure if the sftp code is always buffering - I'd hope if I pass it really large buffers like that, that it just uses those directly, rather than buffer again.

Changed in dput (Ubuntu):
status: Confirmed → In Progress
Julian Andres Klode (juliank) wrote :

Changed it to use 256 KiB large buffers when reading and turned of buffering in sftp, in order to get things to hopefully speed up; although progress reporting quality will suffer obviously.

Changed in dput (Ubuntu):
status: In Progress → Fix Released
status: Fix Released → Fix Committed
Julian Andres Klode (juliank) wrote :

Not sure why I picked 256 KiB vs 64 KiB, though. Let me know how it performs for you.

I just installed dput 1.0.3ubuntu1 on Bionic from https://launchpad.net/ubuntu/+source/dput/1.0.3ubuntu1 and now upload speed is ~11Mbit/s.

For example, upload speed of Midnight Commander or console sftp is 100Mbit/s which is the physical limit of my network.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dput - 1.0.3ubuntu1

---------------
dput (1.0.3ubuntu1) eoan; urgency=medium

  [ Julian Andres Klode ]
  * Merge from Debian unstable. Remaining changes:
    - dput:
      + Recognize 0ubuntu1 as a debian version that requires
        orig.tar.gz to be included in the upload.
      + Correctly handle cases where a non-existant host or no host at all is
        supplied when -o option is passed. (Thanks to David Futcher for
        spotting)
      + Be more careful about setting a variable in a section that does not
        exist in host argument handling.
    - dput.1: Updated to document host argument feature and sftp support.
    - dput.cf:
      + Set 'default_main_host = ubuntu'
      + Set 'progress_indicator = 2'
      + Updated ppa stanza to make use of argument support.
    - dput.cf.5: Updated to note support for sftp transport and host args.
    - dput.cf: Switch ubuntu stanza to upload to "/ubuntu" rather than "/"
      (LP: #1340130).
    - dput.cf: Drop trailing "/ubuntu" from ppa stanza, to support the new
      form of the upload path needed for PPAs based on derived distributions
      (LP: #1340130).
    - Implement a new sftp method that connects via openssh and then
      uses paramiko's sftp support. Some code copied from bzr.
  * sftp: Use bigger chunks when copying files (LP: #1814791)
  * sftp: Handle exceptions during upload, try to remove file

  [ David Lechner ]
  * sftp: fix atomic upload regression for sftp method (LP: #1762572)

dput (1.0.3) unstable; urgency=medium

  * The “سعد راشد محمد الفقيه‎‎ (Sa'ad Rashed Mohammad al-Faqih)” release.

  [ Ben Finney ]
  * Specify current VCS for this code base.
  * Declare “Standards-Version: 4.3.0”. No additional changes needed.
  * Declare Debhelper compatibility level 12.
  * debian/compat:
    * Remove obsolescent configuration file.
  * Update publication years in copyright notices.

  [ Ondřej Nový ]
  * Remove specification of minimum Python versions.
    The required versions are now in all supported Debian releases.

 -- Julian Andres Klode <email address hidden> Mon, 22 Apr 2019 12:31:01 +0200

Changed in dput (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers