Download size discrepancies

[Jonathan Meek]

python-apt returns incorrect values for some API functions on ARM due to broken signed/unsigned and long/int integer conversions.

Impact:
bogus values (e.g. read from stack) for some function calls such as size of a package in cache; this causes user visible issues such as update-manager download sizes, or breaks linaro-image-tools when creating Linaro images (maverick host)

Fix:
This was fixed by using MkPyNumber(...) instead of Py_BuildValue("i", ...) or Py_BuildValue("k", ...)

TEST CASE:
run lp610820.py attached to this bug; if the output is a single entry in the size -> number of packages in cache hash, then python-apt is broken:
        0 => 33936

instead, it should look like a long distribution of values in the form of:
      922 => 4
     1798 => 6
     1800 => 4
     1802 => 5
     1828 => 6
     1830 => 10
     1832 => 7

Regression potential:
Could break the patches functions or could regress applications which relied on the bogus behavior; the builtin testsuite which is run during package build will help a bit with catching the first class of issues, and we'd likely want to fix the other issues by themselves. Testsuite coverage is likely not 100% perfect though

[hrhnick (hrhnick)]

i also have download size discrepancies, except for me, no matter what the actual size is, update manager displays 1kb.

[Noel J. Bergman (noeljb)]

I have the "1K" download size issue, too.

[Noel J. Bergman (noeljb)]

See also: http://ubuntuforums.org/showthread.php?p=9681455#post9681455

[Noel J. Bergman (noeljb)]

This appears to be fixed as of earlier this week.

[kyleabaker (kyleabaker)]

@Noel J. Bergman
You are referring to a completely different bug. The bug you were talking about (which I also saw) was where the download size was always only 1kb in the main window.

The one that this bug is filed against is for when the download size appears to be correct at first, but then a different size is used in the downloading dialog. See:
http://ubuntuforums.org/showthread.php?t=1561222

[Jonathan Meek (shroudedcloud)]

This bug is fixed in Maverick, but it also affected Lucid, I will need to check later if it still applies. (That is, the ORIGINAL issue, not the separate one of 1kb being reported.)

[kyleabaker (kyleabaker)]

@Jonathan
Unless it was fixed in updates in the last few hours, then its still active in Maverick. The screenshot I posted was taken less than 20 hours ago and I saw it occur with updates about an hour ago as well.

[roffik (roffik)]

If I'm right, the values shown on the list are correct, and if you add them up, they match the total download size shown while downloading. But the total download size value shown in the main window is not summed right, I think it doesn't count the decimal parts. For example, if you have one package of 8.8MB and another one of 1.7MB, it will be shown as 9.0MB and not 10.5MB.

Maverick with all updates.

[Steev Klimaszewski (steev-gentoo)]

I'm not sure if this is the same bug as everywhere else, however on an Ubuntu 10.10 installation on an ARM machine, every package says package (Size: 704.0 MB); although the size under the updates window is correct in how much it wants to download, I.E. 12MB of downloads. Attaching a screenshot to show one package that I currently see, which is upstart.

[Steev Klimaszewski (steev-gentoo)]

Is there anything being done with this? No response to my screenshot even letting me know if I have the right bug, or if this is about something else...

[Loïc Minier (lool)]

Could you try running the attached script with:
python list-upgradeable-sizes

and reporting the output here when this bug occurs?

Thanks!

[Loïc Minier (lool)]

Also, could you please mention your version of the update-manager package and your version of Ubuntu; thanks!

[Steev Klimaszewski (steev-gentoo)]

Hi Loic,

Thanks for the test case to run. This is my output:

steev@sbtesting:~$ ./list-upgradeable-sizes
./list-upgradeable-sizes:7: DeprecationWarning: Accessed deprecated property Package.packageSize, please see the Version class for alternatives.
  print "%s: %s" % (pkg.name, pkg.packageSize)
avahi-autoipd: 722034688
avahi-daemon: 722034688
avahi-utils: 722034688
firefox: 722034688
firefox-branding: 722034688
firefox-gnome-support: 722034688
libavahi-client3: 722034688
libavahi-common-data: 722034688
libavahi-common3: 722034688
libavahi-core7: 722034688
libavahi-glib1: 722034688
libavahi-gobject0: 722034688
libavahi-ui0: 722034688
libpoppler-glib5: 722034688
libpoppler7: 722034688
libtiff4: 722034688
linux-firmware: 722034688
poppler-utils: 722034688
python-avahi: 722034688
xulrunner-1.9.2: 722034688

This is on Ubuntu 10.10 (Maverick) and using update-manager:

steev@sbtesting:~$ apt-cache policy update-manager
update-manager:
  Installed: 1:0.142.22
  Candidate: 1:0.142.22
  Version table:
 *** 1:0.142.22 0
        500 http://ports.ubuntu.com/ubuntu-ports/ maverick-updates/main armel Packages
        100 /var/lib/dpkg/status
     1:0.142.19 0
        500 http://ports.ubuntu.com/ubuntu-ports/ maverick/main armel Packages

[Loïc Minier (lool)]

it would be nice to find out if it happens with libapt or apt command-line tools too

perhaps with:
apt-cache show avahi-autoipd=x.y.z | grep Size

where x.y.z is the candidate version of avahi-autoipd returned by apt-cache policy avahi-autoipd.

[Manfred Hampl (m-hampl)]

I see three different bugs discussed here:
1. The issue with all packages shown as 1K size (comments #1 - #4), apparently solved
2. The issue that the total size is shown different from the size of the single packages (#0 plus #5 - #9). That one is already discussed in bug 410310
3. The issue that all packages are shown with 704M size (#10 - #14)

Which one should now be discussed here?
As the most recent discussions are about the 704M issue, and the other two are either resolved or covered elsewhere, most probably it's the 704M issue?

[Loïc Minier (lool)]

Yup, seems reasonable

[Julian Andres Klode (juliank)]

Marking as incomplete, as we're missing the output from apt-cache.

[Steev Klimaszewski (steev-gentoo)]

Sorry, as I had already updated the machine, and natty is in development mode, there just haven't been any updates on my Maverick box that I've seen in order for me to reproduce this again.

[Steev Klimaszewski (steev-gentoo)]

Fwiw, the command that Loic provided, with the package installed gives:

Installed-Size: 188
Size: 38864

Do these comments help?

[Julian Andres Klode (juliank)]

Try running the script and attach the output.

[Steev Klimaszewski (steev-gentoo)]

722042880 => 1304

[Steev Klimaszewski (steev-gentoo)]

> 722042880 =>  1304
>

I just ran it on a fresh install, ran apt-get update, and then ran the
script before installing any of the updates.

I get the following:

722042880 => 33216

[Julian Andres Klode (juliank)]

Thank you. Seems the bug is still there then

[Julian Andres Klode (juliank)]

Could you check whether the attached patch fixes the bug? It seems to have to do with unsigned/signed and long/int stuff, and the attached patch changes all conversions from a numerical type to a Python object to use the correct conversion function with the help of overloaded MkPyNumber functions.

[Julian Andres Klode (juliank)]

A fix has been committed to the debian-experimental branch.

It can be seen at
http://bzr.debian.org<email address hidden>

The message for this commit is:

------------------------------------------------------------
revno: 515
revision-id: <email address hidden>
parent: <email address hidden>
fixes bug(s): https://launchpad.net/bugs/610820
committer: Julian Andres Klode <email address hidden>
branch nick: debian-experimental
timestamp: Tue 2011-04-12 11:38:25 +0200
message:
  apt_pkg: Fix unsigned/long-vs-int issues (LP: #610820)

  This fix is large, but simple in concept. Instead of relying
  on Py_BuildValue and type signatures, or type-specific
  conversion functions, create a new set of overloaded
  MkPyNumber() functions that automatically do the right
  thing for each numerical type.

[Steev Klimaszewski (steev-gentoo)]

The cache.cc portion has 1 failed hunk, at 221. The version in Maverick (0.7.96.1ubuntu11.1) Looks like whatever is in Debian is a few paces ahead, as it doesn't have that function. I built it with that portion left off, and running the script from earlier in the bug, I get the following output.

steev@builder:~/sandbox$ python lp610820.py
        0 => 21
      806 => 4
      904 => 4
      922 => 4
     1798 => 6
     1800 => 4
     1802 => 5
     1828 => 6
     1830 => 10
     1832 => 7
     1834 => 9
     1836 => 10
     1838 => 10
     1840 => 5
     1842 => 9
     1934 => 5
     1936 => 8
     1946 => 4
     1952 => 5
     1956 => 5
     1958 => 6
     1960 => 6
     1968 => 4
     2006 => 6
     2008 => 5
     2010 => 5
     2012 => 4
     2014 => 8
     2016 => 10
     2018 => 10
     2020 => 11
     2022 => 10
     2024 => 7
     2026 => 8
     2028 => 11
     2030 => 14
     2032 => 8
     2034 => 18
     2036 => 26
     2038 => 16
     2040 => 20
     2042 => 10
     2044 => 19
     2046 => 14
     2048 => 23
     2050 => 21
     2052 => 16
     2054 => 15
     2056 => 11
     2058 => 14
     2060 => 11
     2062 => 14
     2064 => 10
     2066 => 5
     2068 => 5
     2070 => 7
     2072 => 12
     2074 => 5
     2076 => 6
     2078 => 4
     2084 => 5
     2086 => 4
     2088 => 4
     2094 => 4
     2130 => 4
     2374 => 4
     2606 => 4
     2608 => 4
     3384 => 4
     3386 => 5
     3388 => 5
     3390 => 6
     3392 => 5
     4066 => 4
     4854 => 4
     4856 => 4
     5068 => 8
     5560 => 4
     6378 => 4
     6470 => 4
     6600 => 4
     6936 => 4
     7108 => 4
     7306 => 4
     7822 => 4
     8056 => 5
     8832 => 5
     8896 => 4
     9564 => 5
     9858 => 4
     9944 => 4
    10122 => 5
    10144 => 4
    10272 => 5
    10540 => 4
    10570 => 5
    10644 => 5
    10894 => 4
    11280 => 4
    11460 => 4
    11476 => 4
    11866 => 4
    11870 => 4
    11878 => 4
    12138 => 4
    12204 => 4
    12342 => 5
    12392 => 5
    12808 => 4
    12840 => 5
    13032 => 5
    13360 => 4
    14028 => 4
    14032 => 4
    14136 => 4
    14332 => 4
    14368 => 4
    14690 => 4
    14696 => 4
    14700 => 4
    14706 => 4
    14720 => 6
    15066 => 4
    15100 => 4
    15146 => 4
    15210 => 4
    15436 => 6
    16400 => 5
    16498 => 4
    16956 => 4
    17856 => 4
    18340 => 4
    18570 => 4
    19168 => 4
    19222 => 4
    19580 => 4
    19594 => 4
    19786 => 4
    20362 => 4
    21134 => 4
    21140 => 4
    21672 => 4
    23122 => 5
    24910 => 4
    26160 => 4
    29026 => 4
    30266 => 4
    31284 => 4
    32626 => 4
    36970 => 4
    3...

[Steev Klimaszewski (steev-gentoo)]

Looks like that fixed it!

[Julian Andres Klode (juliank)]

Yes, the patch was for natty. It could be backported for maverick, though

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 0.8.0~exp4ubuntu1

---------------
python-apt (0.8.0~exp4ubuntu1) oneiric; urgency=low

  * Merged from debian/experimental, remaining changes:
    - updated mirror list
    - do not disable 0.7 compat API yet

python-apt (0.8.0~exp4) experimental; urgency=low

  * apt_pkg: Add OrderList, wanted for mancoosi (Closes: #623485)
  * apt_pkg: Add subclassing fun to PackageManager, for #623485 as well
  * apt.cache: Emit change signals in ProblemResolver
  * apt.Cache: Add a _changes_count member for later use

python-apt (0.8.0~exp3) experimental; urgency=low

  [ Stéphane Graber ]
  * Update enable_component to also apply to -src entries (LP: #758732)

  [ Julian Andres Klode ]
  * apt_pkg: Add apt_pkg.Version.multi_arch and friends

python-apt (0.8.0~exp2) experimental; urgency=low

  * aptsources: Parse multi-arch sources.list files correctly
  * aptsources: Allow insertion of new multi-arch entries
  * aptsources: Various cleanup work
  * all: Fix all instances of ResourceWarning about unclosed files
  * tests/test_apt_cache.py: Use assertTrue() instead of assert_()
  * apt_pkg: Raise error when parse_commandline gets empty argv (LP: #707416)
  * apt_pkg: Fix time_to_str, time_rfc1123 to accept more correct values
    (time_to_str accepts unsigned long, time_rfc1123 long long, y2k31-correct).
  * apt.progress: Use long for ETA, natural type for size (LP: #377375)
  * aptsources/sourceslist.py: s/aptsource.py/sourceslist.py/ (LP: #309603)
  * doc/examples: Add example on how to get architecture names (LP: #194374)
  * apt_pkg: Fix unsigned/long-vs-int issues (LP: #610820)
  * apt.cache: Document that update() may need an open() (Closes: #622342)
  * apt.cache: Add a fetch_archives() method (Closes: #622347)
  * doc: Fix a minor formatting error, patch by Jakub Wilk (Closes: #608914)
  * apt.package: Add 'tasks' to Version, improve doc (Closes: #619574)
  * doc: Fix documentation of BaseDependency.relation (Closes: #607031)

python-apt (0.8.0~exp1) experimental; urgency=low

  * Disable the old-style API, and break all packages using it
  * Add an 'is_multi_arch' attribute to apt_pkg.Cache
  * Add apt_pkg.Group class, wrapping pkgCache::GrpIterator
  * Change apt_pkg.Cache() so that passing None for 'progress' results in
    no progress output
  * Support (name, arch) tuples in apt_pkg.Cache mappings, wrapping
    FindPkg() with two string parameters.
  * Introduce apt_pkg.Cache.groups and apt_pkg.Cache.group_count
  * Fix debian/rules to work correctly with tilde in version number
 -- Michael Vogt <email address hidden> Tue, 24 May 2011 10:08:56 +0200

[Loïc Minier (lool)]

I tested this debdiff in a maverick non-virtual PPA (linaro-maintainers/tools) on armel and it allowed the test case to pass.

[Loïc Minier (lool)]

Corresponding natty debdiff; untested but in a position to test once it's in -proposed

[Martin Pitt (pitti)]

Accepted python-apt into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Pitt (pitti)]

Accepted python-apt into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Steev Klimaszewski (steev-gentoo)]

Thanks for this Martin, I've tested on the EfikaMX smartbook, and it
looks to be working here. (ARM Cortex A-8 running Maverick)

[Loïc Minier (lool)]

maverick:
with 0.7.96.1ubuntu11:
1086308352 => 31305
after upgrading to 0.7.96.1ubuntu11.2:
      922 => 4
     1798 => 6
[...]
    37918 => 4
    43612 => 4
=> PASS

natty:
with 0.7.100.3ubuntu6:
        0 => 32870
with 0.7.100.3ubuntu6.1:
        0 => 137
      940 => 4
[...]
    39236 => 4
    52916 => 4
=> PASS

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 0.7.96.1ubuntu11.2

---------------
python-apt (0.7.96.1ubuntu11.2) maverick-proposed; urgency=low

  [ Loïc Minier ]
  * Backport change below to fix bogus sizes returned by python-apt funcs.

  [ Julian Andres Klode ]
  * apt_pkg: Fix unsigned/long-vs-int issues (LP: #610820)
 -- Loic Minier <email address hidden> Thu, 26 May 2011 18:30:50 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-apt - 0.7.100.3ubuntu6.1

---------------
python-apt (0.7.100.3ubuntu6.1) natty-proposed; urgency=low

  [ Loïc Minier ]
  * Backport change below to fix bogus sizes returned by python-apt funcs.

  [ Julian Andres Klode ]
  * apt_pkg: Fix unsigned/long-vs-int issues (LP: #610820)
 -- Loic Minier <email address hidden> Thu, 26 May 2011 19:26:27 +0200