since oneiric ocelot wget mirror with -c -N option leads to memory exhaustion

[ray]

i try to mirror a public server. i interrupt the download and continue later.

somehow wget gets slow overally comparing the sizes and continuing a session used to be lightning fast. now it slows down.

also for some files or large sets of files it already downloaded before it somehow at some point just suddenly spikes the memory usage (visible at the system load indicators) and then it terminates telling something like 'memory exhausted'

so maybe this is caused somewhere in the decision logic where wget keeps the files it already found in memory.

maybe someone can reproduce this. i see this on a 32 bit system. wget version is 1.12 if i remember correctly.

mirror a site with the options --mirror -c (continue partial downloads) and -N (timestamp checking)

interrupt and continue. sometimes this happens. most of the time it doesn't. but when the session got to a point when it failed once it will fail again at the same stage.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dennis (dennis.guse)]

For reproduction:

wget --convert-links --domains=www.jvm.com,secure.jvm.com --page-requisites --span-hosts -r http://www.jvm.com

Does not work on FreeBSD 9.1 and Ubuntu 13.04; first shows memory full, later segfault

FreeBSD 9.1 WGET version

root@fsc1:/usr/home/user # wget --version
GNU Wget 1.13.4 built on freebsd9.1.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie +ssl/openssl

Wgetrc:
    /usr/local/etc/wgetrc (system)
Locale: /usr/local/share/locale
Compile: cc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
    -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib
    -I/usr/include -I/usr/local/include -D_THREAD_SAFE
    -I/usr/local/include -O2 -pipe -fno-strict-aliasing
Link: cc -O2 -pipe -fno-strict-aliasing -rpath=/usr/lib:/usr/local/lib
    -L/usr/lib -L/usr/local/lib -L/usr/local/lib
    /usr/local/lib/libiconv.so -Wl,-rpath -Wl,/usr/local/lib
    /usr/local/lib/libintl.so /usr/local/lib/libiconv.so -Wl,-rpath
    -Wl,/usr/local/lib /usr/lib/libssl.so /usr/lib/libcrypto.so
    /usr/lib/libz.so -lz -lidn ftp-opie.o openssl.o http-ntlm.o
    ../lib/libgnu.a