dput stalling forever during upload without returning error

[Scott Ritchie]

Binary package hint: dput

When trying to upload a particularly large source package (160 megabytes in this case), dput fails for me:

scott@scott-desktop:~/debian/spring$ dput spring-ppa spring-maps-teamplay_1.0_source.changes
gpg: /home/scott/.gnupg/options:112: deprecated option "honor-http-proxy"
gpg: please use "keyserver-options http-proxy" instead
gpg: Signature made Wed 20 Feb 2008 01:18:40 AM PST using DSA key ID 387EE263
gpg: Good signature from "Scott Ritchie <email address hidden>"
gpg: aka "Scott Ritchie <email address hidden>"
gpg: aka "Scott Ritchie <email address hidden>"
Good signature on /home/scott/debian/spring/spring-maps-teamplay_1.0_source.changes.
Checking Signature on .dsc
gpg: /home/scott/.gnupg/options:112: deprecated option "honor-http-proxy"
gpg: please use "keyserver-options http-proxy" instead
gpg: Signature made Wed 20 Feb 2008 01:18:35 AM PST using DSA key ID 387EE263
gpg: Good signature from "Scott Ritchie <email address hidden>"
gpg: aka "Scott Ritchie <email address hidden>"
gpg: aka "Scott Ritchie <email address hidden>"
Good signature on /home/scott/debian/spring/spring-maps-teamplay_1.0.dsc.
Uploading to spring-ppa (via ftp to ppa.launchpad.net):
  spring-maps-teamplay_1.0.dsc: done.
  spring-maps-teamplay_1.0.tar.gz:

After uploading for a while, something will go wrong and dput will stay there indefinitely. Looking at my internet connection, I have no upload activity, however dput is not attempting to restore the connection, or even give a failure warning. Preferably, dput would auto resume the transfer if it gets interrupted for some reason, however at the very least it should fail with an error here rather than leaving me stuck.

[Sarah Kowalik (hobbsee-deactivatedaccount)]

I've added a default option of progress_indicator = 2 which will tell people if the package is still uploading. Hopefully this will help a little.

[Chris Cheney (ccheney)]

I think I see this problem or maybe bug 183624. When uploading it just hangs forever with 1KB left to upload. I am currently retrying with debugging enabled to see if it tells me anything useful.

Thanks,

Chris Cheney

[Chris Cheney (ccheney)]

Ok, here is my debugging output, it hangs with no output as to why it hung. And it does this repeatedly at the same spot 1KB left.

Thanks,

Chris Cheney

dput -d *.changes
D: dput 0.9.2.30
D: Login: root
D: Parsing Configuration File /etc/dput.cf
No such file or directory: /root/.dput.cf, skipping
D: File: /usr/share/dput/http.py
D: Base: http
Imported: /usr/share/dput/http.py
D: File: /usr/share/dput/scp.py
D: Base: scp
Imported: /usr/share/dput/scp.py
D: File: /usr/share/dput/https.py
D: Base: https
Imported: /usr/share/dput/https.py
D: File: /usr/share/dput/local.py
D: Base: local
Imported: /usr/share/dput/local.py
D: File: /usr/share/dput/rsync.py
D: Base: rsync
Imported: /usr/share/dput/rsync.py
D: File: /usr/share/dput/ftp.py
D: Base: ftp
Imported: /usr/share/dput/ftp.py
Upload package to host ubuntu
D: Validating contents of changes file /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2_source.changes
D: Architecture: source
D: dsc-File: openoffice.org_2.4.0~rc2-1ubuntu2.dsc
D: .changes-File: /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2_source.changes
D: .dsc-File: /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2.dsc
Checking Signature on .changes
gpg: Signature made Mon 03 Mar 2008 12:39:41 PM CST using DSA key ID 8B1AE688
gpg: Good signature from "Chris Cheney <email address hidden>"
gpg: aka "Chris Cheney <email address hidden>"
gpg: aka "Chris Cheney <email address hidden>"
gpg: aka "Chris Cheney <email address hidden>"
Good signature on /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2_source.changes.
Checking Signature on .dsc
gpg: Signature made Mon 03 Mar 2008 12:39:40 PM CST using DSA key ID 8B1AE688
gpg: Good signature from "Chris Cheney <email address hidden>"
gpg: aka "Chris Cheney <email address hidden>"
gpg: aka "Chris Cheney <email address hidden>"
gpg: aka "Chris Cheney <email address hidden>"
Good signature on /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2.dsc.
D: Package Version: 1:2.4.0~rc2-1ubuntu2
D: Epoch found
D: Upstream Version: 2.4.0~rc2
D: Debian Version: 1ubuntu2
D: File to upload: /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2.dsc
D: Checksum for /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2.dsc is fine
D: File to upload: /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2.diff.gz
D: Checksum for /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2.diff.gz is fine
D: Checking: distribution hardy matches (?!UNRELEASED)
D: File to upload: /root/chroot/hardy-amd64-chroot/root/ooo.24/openoffice.org_2.4.0~rc2-1ubuntu2_source.changes
D: Default Method: ftp
D: Host Method: ftp
D: Login anonymous from section ubuntu used
Uploading to ubuntu (via ftp to upload.ubuntu.com):
D: FQDN: upload.ubuntu.com
D: Login: anonymous
D: Incoming: /
D: FTP port: 21
D: Using passive ftp
D: FTP-Connection to host: upload.ubuntu.com
D: Directory to upload to: /
D:...

[Chris Cheney (ccheney)]

I think that my problem has something to do with my computer/router/ISP, since I tried uploading using lftp and it hung in the same way.

Thanks,

Chris

[Scott Ritchie (scottritchie)]

I have Comcast for an ISP, and I get this problem with any moderately large package. It wouldn't surprise me if their awful filtering is screwing this up somehow.

SFTP never fails for me. So currently what I've been doing is using that to move large packages to another server with a better connection, and then dput from there. That works perfectly.

[Patrice Vetsel (vetsel-patrice)]

I also have this problem under 8.10 (also on 8.04).

On little packages it's ok, but on bigger, i upload to 80% and freeze, retry and freeze at 55%, retry and freeze at size-1Kb ...

[Scott Ritchie (scottritchie)]

Interesting note: if I use a VPN box to tunnel from my Comcast connection to a more serious network, the FTP upload never fails as well; the problem is clearly between me and the Internet.

Perhaps the best solution is to make dput default to sftp, and then make Ubuntu and all the PPAs compatible with that?

[Scott Lewin (sclewin)]

I don't know if I am having the same problem or not. On both my Ubuntu 8.10 Laptop and my Kubuntu 8.04 Desktop I am not able to get any http or ftp upload except for tiny bits of data, like this message.

[Scott Lewin (sclewin)]

I just noticed that my laptop works fine if I plug it directly into my modem. So, at least my problem, is a argument between my router and my Ubuntu Laptop and Kubuntu Desktop. If it helps I have a D-link WBR-2310 router. I am going to write a separate bug as I don't think my problem is the same as others here.

[Rudy Godoy (rudy)]

I'm having the same issue in a Debian sid box while uploading to Debian ftp-master. Apparently, after strace and -d, the server closes the connection and dput is unable to reinstantiate it so it continues sending the remaining files.
A more detailed debug should be done under this conditions. I'm uploading a file which size is circa 10MB.

[Max Bowsher (maxb)]

Revert unjustified status change.

[Mahyuddin Susanto (udienz)]

In Maverick i still have this problem, dput hang for a while while 1KB left. i use ftp method

$ dput -f ppa-experimental-natty ../zabbix_1.8.4-2ubuntu1_source.changes
Checking signature on .changes
gpg: Signature made Mon Feb 21 13:07:27 2011 WIT using DSA key ID 08E6A8A5
gpg: Good signature from "Mahyuddin Susanto <email address hidden>"
Good signature on ../zabbix_1.8.4-2ubuntu1_source.changes.
Checking signature on .dsc
gpg: Signature made Mon Feb 21 13:07:22 2011 WIT using DSA key ID 08E6A8A5
gpg: Good signature from "Mahyuddin Susanto <email address hidden>"
Good signature on ../zabbix_1.8.4-2ubuntu1.dsc.
Package includes an .orig.tar.gz file although the debian revision suggests
that it might not be required. Multiple uploads of the .orig.tar.gz may be
rejected by the upload queue management software.
Uploading to ppa-experimental-natty (via ftp to ppa.launchpad.net):
  Uploading zabbix_1.8.4-2ubuntu1.dsc: done.
  Uploading zabbix_1.8.4.orig.tar.gz: done.
  Uploading zabbix_1.8.4-2ubuntu1.debian.tar.gz: 42k/43k

[Radu Cristescu (radu.c)]

This could be unrelated: I upload using the dput and the sftp mechanism and dput gets stuck in the "terminate" function of /usr/lib/python2.6/dist-packages/bzrlib/transport/ssh.py waiting for ssh to finish (which doesn't happen). I just commented out this: _close_ssh_proc(proc) in the specified file and my problem went away without side effects (no ssh processes left behind).

Looking at dotted_names = ['stdin.close', 'stdout.close', 'wait'], I assume stdin and stdout don't actually close. gdb says my processes is already in "wait". Pressing Control+C without the modification mentioned above says it's ignored in function "terminate" and then proceeds to tell me the upload was successful and pretend the lockup never happened.

[Anton Eliasson (eliasson)]

Those of you whose upload fails with 1 kB left to go can check out https://bugs.launchpad.net/launchpad/+bug/251685 which suggests configuring dput to upload via SFTP instead of FTP.

[Jeremy Bícha (jbicha)]

Specifically, to workaround this and enable SFTP for dputting to the Ubuntu archive see http://blog.launchpad.net/cool-new-stuff/sftp-uploads#comment-40017

[Filip Brcic (brcha)]

I had the same problem with uploading qt sources (~240 MB). I wrote this into my ~/.dput.cf and now it works fine

[sppa]
fqdn = ppa.launchpad.net
method = sftp
incoming = ~%(sppa)s/ubuntu
login = brcha

Just copy the snippet above to your ~/.dput.cf file, change username (unless you have my private ssh key which is not likely :) ) and upload to dput sppa:<yourname>/<yourrepo> <somefile>.changes. My advice is to use regular ppa:x/y for transfers of smaller size since sftp doesn't show the progress indicator which is a cool feature to have :)

[Alexander List (alexlist)]

I can confirm this problem using dput with ftp.

After around 4MB of upload, I get ICMP network unreachable from my router, after a few minutes dput crashes.

I am behind an OpenWRT router which connects to my DSL.

[Phillip Susi (psusi)]

What is the state of the socket when it is hung? Can you fire up wireshark and see if there are any interesting packets?

[Phillip Susi (psusi)]

Also can you upload such files via ftp to other sites besides launchpad?

[Launchpad Janitor (janitor)]

[Expired for dput (Ubuntu) because there has been no activity for 60 days.]