Wget does not support large file sizes using HTTP

[Bruno Santos]

$ wget -c http://darkstar.ist.utl.pt/ubuntu/6.10/dvd/ubuntu-6.10-dvd-i386.iso
--00:54:36-- http://darkstar.ist.utl.pt/ubuntu/6.10/dvd/ubuntu-6.10-dvd-i386.iso
           => `ubuntu-6.10-dvd-i386.iso'
Resolving darkstar.ist.utl.pt... 193.136.198.175
Connecting to darkstar.ist.utl.pt|193.136.198.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: -569,649,152 [text/plain]

    [ <=> ] 0 --.--K/s

Aborted (core dumped)

[magilus (magilus)]

Maybe the Ubuntu wget does not have large file support compiled in?

[Bruno Santos (bsantos)]

What? Why did you change my report?!

I've downloaded ISOs lots of times, this specific time the server reported the size of the file as negative and wget crashed...

[magilus (magilus)]

The same appears to me, too.

But the actual problem - as far as I can see - is that wget does not support large files. By this, the negative file sizes and the "Aborted (core dumped) is caused. That's why I changed the title to the actual problem.

Here is an example:

http://darkstar.ist.utl.pt/ubuntu/6.10/dvd/ubuntu-6.10-dvd-i386.iso (ubuntu dvd image, large) does *not* work.

http://darkstar.ist.utl.pt/ubuntu/6.10/ubuntu-6.10-desktop-i386.iso (ubuntu cd image, small) *works*.

If you do not agree with me here, please tell me that you want to have it changed back.

[Bruno Santos (bsantos)]

I can change it back, Martin, the issue's not that. ;-)

I thought that on a server side error (negative file size) wget would
segfault. You suggest that it may be lack of large file size support,
although I'm almost sure of downloading other dvd iso files with wget
without issues... But maybe I didn't, and you're right.

I just tried another mirror and the error has repeated itself.

Thanks for correcting my report. :-)

On Thu, 2006-11-30 at 14:33 +0000, Martin Jürgens wrote:
> The same appears to me, too.
>
> But the actual problem - as far as I can see - is that wget does not
> support large files. By this, the negative file sizes and the "Aborted
> (core dumped) is caused. That's why I changed the title to the actual
> problem.
>
> Here is an example:
>
> http://darkstar.ist.utl.pt/ubuntu/6.10/dvd/ubuntu-6.10-dvd-i386.iso
> (ubuntu dvd image, large) does *not* work.
>
> http://darkstar.ist.utl.pt/ubuntu/6.10/ubuntu-6.10-desktop-i386.iso
> (ubuntu cd image, small) *works*.
>
>
> If you do not agree with me here, please tell me that you want to have it changed back.
>

[Bruno Santos (bsantos)]

A fedora DVD works well:

--15:20:27--
http://iso.linuxquestions.org/download/ftp/ftp.fu-berlin.de/FC-6-i386-DVD.iso
           => `FC-6-i386-DVD.iso'
Resolving iso.linuxquestions.org... 207.44.182.114
Connecting to iso.linuxquestions.org|207.44.182.114|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location:
ftp://ftp.fu-berlin.de/linux/fedora/core/6/i386/iso/FC-6-i386-DVD.iso
[following]
--15:20:28--
ftp://ftp.fu-berlin.de/linux/fedora/core/6/i386/iso/FC-6-i386-DVD.iso
           => `FC-6-i386-DVD.iso'
Resolving ftp.fu-berlin.de... 130.133.3.130
Connecting to ftp.fu-berlin.de|130.133.3.130|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /linux/fedora/core/6/i386/iso ... done.
==> PASV ... done. ==> RETR FC-6-i386-DVD.iso ... done.
Length: 3,525,195,776 (3.3G) (unauthoritative)

What's the limit?

On Thu, 2006-11-30 at 14:33 +0000, Martin Jürgens wrote:
> The same appears to me, too.
>
> But the actual problem - as far as I can see - is that wget does not
> support large files. By this, the negative file sizes and the "Aborted
> (core dumped) is caused. That's why I changed the title to the actual
> problem.
>
> Here is an example:
>
> http://darkstar.ist.utl.pt/ubuntu/6.10/dvd/ubuntu-6.10-dvd-i386.iso
> (ubuntu dvd image, large) does *not* work.
>
> http://darkstar.ist.utl.pt/ubuntu/6.10/ubuntu-6.10-desktop-i386.iso
> (ubuntu cd image, small) *works*.
>
>
> If you do not agree with me here, please tell me that you want to have it changed back.
>

[magilus (magilus)]

The link provided from you is a FTP link (the HTTP link forwards to a FTP server), which seems to work fine using wget.

I believe that wget only fails when downloading a big file using HTTP.

[Bruno Santos (bsantos)]

Nice catch!

Confirmed:

--16:00:58--
ftp://darkstar.ist.utl.pt/pub/ubuntu/releases/dvd/ubuntu-6.10-dvd-i386.iso
           => `ubuntu-6.10-dvd-i386.iso'
Resolving darkstar.ist.utl.pt... 193.136.198.175
Connecting to darkstar.ist.utl.pt|193.136.198.175|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/ubuntu/releases/dvd ... done.
==> PASV ... done. ==> RETR ubuntu-6.10-dvd-i386.iso ... done.
Length: 3,725,318,144 (3.5G) (unauthoritative)

The same file through ftp works.

On Thu, 2006-11-30 at 15:47 +0000, Martin Jürgens wrote:
> The link provided from you is a FTP link (the HTTP link forwards to a
> FTP server), which seems to work fine using wget.
>
> I believe that wget only fails when downloading a big file using HTTP.
>
> ** Summary changed:
>
> - Wget does not support large file sizes
> + Wget does not support large file sizes using HTTP
>

[magilus (magilus)]

I compiled wget with something like --enable-largefile which was mentioned in the INSTALL. But the same issue appears.

[magilus (magilus)]

It also may be caused by the webserver not supporting files bigger than 2 GB (Apache 2.0.50 and older do not support this). But then I wonder why it works with Firefox..

[Bruno Santos (bsantos)]

It doesn't happen as of 04/04/07 in 1.10.2-2ubuntu2 (Feisty).

[Andreas Gustafsson (gson)]

wget still dumps core in 8.04.1 if the HTTP server returns a large negative content-length.
To reproduce:

wget http://www.gson.org/bugs/ubuntu/wget/test

[Andreas Gustafsson (gson)]

The bug is also triggered by attempting to download a file between 2G and 4G in size from an Apache 1.3 server, because it has a bug causing it to return a negative content-length:

https://issues.apache.org/bugzilla/show_bug.cgi?id=22827

[Micah Cowan (micahcowan)]

This issue should be resolved in Wget-1.11.4, which is available in Ubuntu 8.10, so this bug should be closed, as it has been fixed in recent releases of Ubuntu (please re-open if you find this is not in fact the case).

I expect this qualifies for a Stable Release Update for Hardy, so if you would like to start the process for that, please complete steps one and two from https://wiki.ubuntu.com/StableReleaseUpdates#Procedure

In the meantime, I believe you can actually just grab the binary package from Intrepid at https://launchpad.net/ubuntu/intrepid/+source/wget/1.11.4-1ubuntu1 (binary packages are available from a panel on the right).