Apt-get does not support packages of 2.5 gig...

[stg]

This might be a little unusual but I created a 2.5 gig debian package that I've put on my apt server...

When I do an apt-get install It fails when checking the size of the file and it outputs it as an exponential value.

Since 2gig is the maximum for 32 bits I assume that this could be the problem...

[Chris Coulson (chrisccoulson)]

Probably dpkg, so reassigning

Could you please provide a little bit more information:

1. The Ubuntu version you're using (type "lsb_release -rd" in to a terminal)
2. The version of dpkg that you are using ("apt-cache policy dpkg")
3. Does the package install using dpkg directly (cd in to the directory containing your package, then type "sudo dpkg -i <package_name>.deb")

Thanks

[Adam Conrad (adconrad)]

The actual terminal output of the failure might be nice. This could easily be breakage from 3 areas:

1) A web server that doesn't properly serve files over 2GB.
2) apt unable to correctly deal with large files.
3) dpkg unable to correctly deal with large files.

The check you're describing *sounds* like it's apt failing with (2), but it's hard to know for sure with a verbal description instead of a good ol' copy and paste.

[stg (steve-garon-deactivatedaccount)]

First off, sorry for the lack of details on my previous post.

I'm 99.9 % sure its a problem with apt because the download was successful but the error message was at the just before the installation (apt was validating if the download was the same size). Also, when I realized it failed I pulled the downloaded file from the var/cache/apt/archives folder, ran dpkg -i and it worked like a charm.

So this is definatly not an Apache nor Dpkg problem.

Here are the versions I'm running.
Desktop:
8.04.1 with latest patches
apt:
  Installed: 0.7.9ubuntu17
  Candidate: 0.7.9ubuntu17
dpkg:
  Installed: 1.14.16.6ubuntu4
  Candidate: 1.14.16.6ubuntu4

Server:
7.10 with latest patches
apache2:
  Installed: 2.2.4-3ubuntu0.1
  Candidate: 2.2.4-3ubuntu0.1

As off right now, I cannot give you the actual error message but I'll be able to do so tomorrow if its still needed.

Thank you.

[stg (steve-garon-deactivatedaccount)]

Ok here are more details about the bug...

when you type the apt-get install command you get the following:
...
0 upgraded, 1 newly installed, 0 to remove and 0 upgraded.
2.54425e+09, -1.75072e+09
How odd.. The sizes didn't match, email <email address hidden>
Need to get 2544 MB archives.
...

If you say yes to the installation, while its downloading you'll get something like this for the download status:

10% [1 pkg 41515444/18.4EB 0%] 55.5MB/s 19s

And when the transfer is finished you get the following error:

Failed to fetch http://server.domain/amd64/pkg.deb Size mismatch
W: Bizarre Error - File size is not what the server reported 2544251876 18446744071958836196
E: Unable to fetch some archives, try running apt-get update or apt-get --fix-missing

When I saw the error message that the server reported a different file size, I browsed to the server directly with firefox and try to download the file with the browser and everything went perfectly. So I'm now 100% sure that the problem resides inside apt.

Tell me if you need more info.

[James Westby (james-w)]

Hi,

The url of the .deb has amd64 in it. Are either your client or your
server 64 bits?

I'm having trouble reproducing this with 32 bits for both machines.

Thanks,

James

[stg (steve-garon-deactivatedaccount)]

True! I forgot that this could impact ...

My server is 32 bit and my client is 64 bit.

Steve

[James Westby (james-w)]

Hi,

I can't reproduce this with 32 bits on both sides, and I don't have a
64 bit machine, so I'm afraid you will have to wait until someone
who has the right set up can work on the bug.

Thanks,

James

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 0.7.26~exp12ubuntu1

---------------
apt (0.7.26~exp12ubuntu1) maverick; urgency=low

  * ABI break upload
  * merged from debian/experimental, remaining changes:
    - use ubuntu keyring and ubuntu archive keyring in apt-key
    - run update-apt-xapian-index in apt.cron
    - support apt-key net-update and verify keys against master-keyring
    - run apt-key net-update in cron.daily
    - different example sources.list
  * debian/apt.postinst
    - drop set_apt_proxy_from_gconf(), no longer needed in maverick
  * apt-pkg/pkgcache.cc:
    - re-evaluate the architectures cache when the cache is (re)opened

apt (0.7.26~exp12) experimental; urgency=low

  [ Michael Vogt ]
  * debian/control:
    - add dependency on zlib-dev for libapt-pkg-dev

  [ David Kalnischkies ]
  * apt-pkg/cacheset.cc:
    - [ABI BREAK] add an ErrorType option to CacheSetHelper
  * cmdline/apt-cache.cc:
    - use Notice instead of Error in the CacheSetHelper messages
      for compat reasons. Otherwise tools like sbuild blow up
    - return success in show if a virtual package was given
  * debian/control:
    - remove libcurl3-gnutls-dev alternative as the package is gone
    - increase needed version of libcurl4-gnutls-dev to >= 7.19.0
      as we use CURLOPT_{ISSUERCERT,CRLFILE} (Closes: #589642)

apt (0.7.26~exp11) experimental; urgency=low

  [ Julian Andres Klode ]
  * apt-pkg/deb/dpkgpm.cc:
    - Write architecture information to history file.
    - Add to history whether a change was automatic or not.
  * apt-pkg/contrib/fileutl.cc:
    - Add FileFd::OpenDescriptor() (needed for python-apt's #383617).
  * cmdline/apt-get.cc:
    - Support large filesystems by using statvfs64() instead of statvfs()
      and statfs64() instead of statfs() (Closes: #590513).
  * apt-pkg/cdrom.cc:
    - Use link() instead of rename() for creating the CD database backup;
      otherwise there would be a short time without any database.

  [ David Kalnischkies ]
  * apt-pkg/depcache.cc:
    - handle "circular" conflicts for "all" packages correctly
  * cmdline/apt-cache.cc:
    - be able to omit dependency types in (r)depends (Closes: #319006)
    - show in (r)depends the canidate per default instead of newest
    - share the (r)depends code instead of codecopy
  * apt-pkg/cacheset.cc:
    - move them back to the library as they look stable now
    - add a 'newest' pseudo target release as in pkg/newest
  * apt-pkg/pkgcache.cc:
    - prefer non-virtual packages in FindPreferredPkg (Closes: #590041)
  * test/integration/*:
    - add with bug#590041 testcase a small test "framework"
  * apt-pkg/orderlist.cc:
    - try to install another or-group member in DepRemove before
      breaking the or group (Closes: #590438)
    - configure also the replacement before remove by adding Immediate flag

  [ Michael Vogt ]
  * apt-pkg/contrib/error.{cc,h}
    - docstring cleanup
    - add inline DumpError() to avoid subtle API break

apt (0.7.26~exp10) experimental; urgency=low

  [ David Kalnischkies ]
  * apt-pkg/contrib/error.{cc,h}:
    - remove constness of va_list parameter to fix build on amd64 and co
      Thanks Eric Valette! (Closes: #588610)
  * apt...

[Todd Taft (taft)]

This fix also needs to be released for other supported versions of Ubuntu (lucid, karmic, etc.)

[Julian Andres Klode (juliank)]

This fix has been released. Old releases are no reason to keep this open.

[Alex Waite (alexqw85)]

This affects my entire lab with Ubuntu Lucid. Can we either get the fix for this backported to 0.7.25 or (even better) can 0.7.26 be released for Lucid?

---Alex

[Julian Andres Klode (juliank)]

Since this requires an ABI break, backporting the fix to an old release is not a good idea, as it would require recompiling every package using APT.

[Michael Vogt (mvo)]

Attached is the relevant bzr part for the fix as a patch. I don't think its safe to apply this in a SRU as it requires a rebuild of python-apt with a additional patch (acquire-items is expoed to python-apt too)

[Bevan (bevan-bi-co)]

We are still encountering this problem with natty and oneiric.

With natty packages larger than 2GB can not be installed because of a size mismatch:

The following NEW packages will be installed:
  matlab
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 2172 MB of archives. After unpacking 4479 MB will be used.
Get:1 http://example.com/ubuntu/site/ natty/all matlab amd64 7.13-1-4 [2172 MB]
Fetched 18.4 EB in 1min 32s (200 PB/s)
W: Bizarre Error - File size is not what the server reported 2171943158 18446744071586527478
E: Failed to fetch http://example.com/ubuntu/site/dists/natty/all/binary-amd64/matlab/matlab_7.13-1-4_amd64.deb: Size mismatch

With oneiric packages larger than 2GB can be installed but you get several error and warning messages:

The following NEW packages will be installed:
  matlab
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 2172 MB of archives. After unpacking 4479 MB will be used.
Get: 1 http://example.com/ubuntu/site/ oneiric/all matlab amd64 7.13-1-4 [2172 MB]
Fetched 2172 MB in 1min 10s (30.8 MB/s)
E: Unable to seek ahead 2168965456
E: Prior errors apply to /var/cache/apt/archives/matlab_7.13-1-4_amd64.deb
debconf: apt-extracttemplates failed: Illegal seek
Selecting previously deselected package matlab.
(Reading database ... 278578 files and directories currently installed.)
Unpacking matlab (from .../matlab_7.13-1-4_amd64.deb) ...
Setting up matlab (7.13-1-4) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
W: Size of file /var/cache/apt/archives/partial/matlab_7.13-1-4_amd64.deb is not what the server reported 2171943158 18446744071586527478

W: Size of file /var/cache/apt/archives/partial/matlab_7.13-1-4_amd64.deb is not what the server reported 2171943158 18446744071586527478

[Alex Bachmeier (cebalrai)]

I can confirm this bug also exists on precise!

The following packages will be upgraded:
  android-sdk
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/2,789 MB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue [Y/n]? y
E: Unable to seek ahead 2789297700
E: Prior errors apply to /mnt/.auto/install/repo/pool/main/a/android-sdk/android-sdk_17_amd64.deb
debconf: apt-extracttemplates failed: No such file or directory
(Reading database ... 749373 files and directories currently installed.)
Preparing to replace android-sdk 16 (using .../android-sdk_17_amd64.deb) ...
Unpacking replacement android-sdk ...
Setting up android-sdk (17) ...

The package installs, but with errors.

[Yaroslav Leonov (alximik43)]

I've merged some diff from debian-sid branch in addition to Michael Vogt (mvo) patch.
Thus i've solved for myself following error on lucid-amd64:
====
Fetched 18.4 EB in 1min 32s (200 PB/s)
W: Bizarre Error - File size is not what the server reported 2171943158 18446744071586527478
E: Failed to fetch http://example.com/ubuntu/site/dists/natty/all/binary-amd64/matlab/matlab_7.13-1-4_amd64.deb: Size mismatch
====

I'll attach my diff, in case someone will need it. Supposedly this won't work on i386;

[Landon Thomas (lthomas)]

I'm seeing the bug in Ubuntu 12.04 both with apt-get and on the repo end with apt-ftparchive:

me@server:/$ sudo apt-ftparchive --contents=false generate /path/to/local/config/apt-ftparchive-precise.conf
 /path/to/local/repo/: 18 files 19.8 MB 0s
 /path/to/local/repo/:
E: Unable to seek ahead 3747275956
E: Errors apply to file '/path/to/local/repo/pool/local/m/matlab/matlab2013a_8.1.0.604_amd64.deb'
 17 files 3,758 MB 0s
 /path/to/local/repo/:
E: Unable to seek ahead 3747275956
E: Errors apply to file '/path/to/local/repo/pool/local/m/matlab/matlab2013a_8.1.0.604_amd64.deb'
 17 files 3,758 MB 0s
 /path/to/local/repo/: 18 files 16.5 MB 0s
 /path/to/local/repo/: 16 files 4,244 kB 0s
 /path/to/local/repo/: 16 files 4,244 kB 0s
me@server:/$ apt-ftparchive -v
apt 0.8.16~exp12ubuntu10.15 for amd64 compiled on Oct 3 2013 15:14:39

My apt-ftparchive-precise.conf

Default {
        Packages::Compress ".";
        Contents::Compress ".";
}

Dir {
 ArchiveDir "/path/to/local/repo/";
 OverrideDir "/path/to/local/indices";
 CacheDir "/path/to/local/cache/";
};

TreeDefault {
 Directory "/path/to/local/repo/";
 SrcDirectory "/path/to/local/repo";
 Packages "../repo-output/dists/$(DIST)/$(SECTION)/binary-$(ARCH)/Packages";
 Sources "../repo-output/dists/$(DIST)/$(SECTION)/source/Packages";
 FileList "/path/to/local/flist/$(DIST)-$(ARCH)-$(SECTION).flist.inc";
 BinCacheDB "cache-$(ARCH)-$(DIST)-$(SECTION).db";
};

Tree "precise" {
 Sections "main";
 Architectures "i386 amd64";
 SrcOverride "override.precise.$(SECTION).src";
 BinOverride "override.precise.$(SECTION)";
 ExtraOverride "override.precise.extra.$(SECTION)";
};

Tree "precise-updates" {
 Sections "main";
 Architectures "i386 amd64";
 SrcOverride "override.precise-updates.$(SECTION).src";
 BinOverride "override.precise-updates.$(SECTION)";
 ExtraOverride "override.precise-updates.extra.$(SECTION)";
};

Tree "precise-local" {
 Sections "main";
 Architectures "i386 amd64";
 SrcOverride "override.precise-local.$(SECTION).src";
 BinOverride "override.precise-local.$(SECTION)";
 ExtraOverride "override.precise-local.extra.$(SECTION)";
-----

Strangely, the same apt-ftparchive command on my old repo server running Ubuntu 8.04 has no error with the same 3.7 GB MATLAB package:

me@old-server:/$ sudo apt-ftparchive --contents=false generate /path/to/local/config/apt-ftparchive-precise.conf
 /path/to/local/repo/: 18 files 19.8MB 0s
 /path/to/local/repo/: 17 files 3758MB 0s
 /path/to/local/repo/: 17 files 3758MB 0s
 /path/to/local/repo/: 18 files 16.5MB 0s
 /path/to/local/repo/: 16 files 4244kB 0s
 /path/to/local/repo/: 16 files 4244kB 0s
me@old-server:/$ apt-ftparchive -v
apt 0.7.9ubuntu17.6 for amd64 compiled on Jun 15 2012 15:12:30

I've checked the source from Yaroslav Leonov's and Michael Vogt's patches above, and they're incorporated into 12.04's apt-0.8.16~exp12ubuntu10.15 source.

[Landon Thomas (lthomas)]

Followup to comment #17: when I run apt-ftparchive on a mirror server running Ubuntu 14.04, I do not have the same problem. So some change between Ubuntu 12.04 and 14.04 fixed the problem I was having.

[Rolf Leggewie (r0lf)]

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".