apt can't handle installing more than 4 Gb of packages at once

[Alkis Georgopoulos]

Binary package hint: apt

I've been seeing this in Jaunty, Karmic and now Lucid alpha 2 so it must be a long standing problem.

`apt-get install package1 package2`, where those 2 packages exceed 4 Gb halts with the following error:
"Failed to fetch http://ts.sch.gr/repo/pool/main/g/gymnasio/gymnasio_0.14_all.deb Bad header line
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?"

If I try the installation separately, `apt-get install package1` and later `apt-get install package2` it works fine (of course if each one of them is less than 4 Gb).
It also works fine if I try the command twice (see the attached file): the first time it downloads 4 Gb of packages, and it fails, and the second time it downloads the rest Gb, and it succeeds.

To reproduce the problem, just install a whole lot of packages *that you do not already have in your apt cache* at once from the Ubuntu repositories.

I'm attaching the output I get while trying to install 5 Gb of some Greek educational packages.
I also tried with aptitude and on a different web server, see the attached file.

[Michael Vogt (mvo)]

Thanks for your bugreport.

Can you please attach the output of:
$ sudo apt-get install package1 package -o Debug::acquire::http=true

that should include the http headers so that we can diagnose the problem.

[Alkis Georgopoulos (alkisg)]

I'm attaching the output for the requested command.
The headers are completely broken after downloading the first 4GB, so I believe that means that either the web server or apt have a buffer overflow problem.
I had tried with another web server and I had also tried downloading (without apt) files > 4GB from that web server without problems. So I believe the buffer overflow happens in apt.

Thanks,
Alkis

[Michael Vogt (mvo)]

Thanks for the output. The log looks like apt gets confused about where the header ends and when the data starts. I'm trying to reproduce the bug now using:

"deb http://ts.sch.gr/repo stable main"
in my sources.list and installing
sudo apt-get install -d dimotiko gymnasio

this will take a while until I have the packages in my local cache :)

[Michael Vogt (mvo)]

Is this on a i386 or a amd64 machine?

[Alkis Georgopoulos (alkisg)]

I've tried on many i386 machines, and I think I've once tried on an amd64 machine as well with the same symptoms.

Once the packages are in your apt cache, the problem will "disappear", as http headers are needed to trigger it. It might be easier/faster to install a local apache and clone the whole repository - I could give you ftp access if a web spider won't work for the cloning...

Thanks!

[Julian Andres Klode (juliank)]

Still reproducible?

[Alkis Georgopoulos (alkisg)]

It's still reproducible on Lucid...

...but I suppose the question is about Natty.

Since installing Natty and 4+ Gb of packages would be very time consuming, please allow me to ask about what feedback is required, before proceeding with the installation.

If a "yes it's still reproducible on Natty/i386" is enough, then I can just install Natty in a VM.
If more feedback will be needed, I'd better install apache + reprepro locally so that I can then reproduce it many times quickly, without waiting about a day for the packages to get downloaded each time.

[Alkis Georgopoulos (alkisg)]

I cannot reproduce this problem in Precise, I just installed 6 Gb of packages at once.
I'm marking it "Fix Released" even if I don't know exactly at which point it was fixed.