Latest version is much slower to write rdepends output than older ones

Bug #915569 reported by Daniel Manrique on 2012-01-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
germinate (Ubuntu)
High
Colin Watson

Bug Description

We're using germinate to get a list of dependencies needed to install checkbox, meant to produce a tarball of .debs that can be installed on a server that's offline.

While doing this we noticed the germinate process was taking a really long time (at least an hour, sometimes up to three). What we did was go back to rev376 of germinate, and that one is able to finish the process in a few minutes.

So at least for the way we're using germinate (which could potentially be wrong, but it *is* producing good results) there's a regression with regards to execution time.

How to reproduce:

bzr branch lp:germinate germinate-head
bzr branch -r376 lp:germinate germinate-376

mkdir germ-temp-{1,2}

cd germ-temp-1
echo "Run with rev 376"
PYTHONPATH=../germinate-376 time ../germinate-376/germinate.py -s ubuntu.oneiric -m http://archive.ubuntu.com/ubuntu/ -d oneiric -a amd64 -c main,universe --seed-packages ubuntu-desktop
cd ..

cd germ-temp-2
echo "Run with HEAD"
PYTHONPATH=../germinate-head time ../germinate-head/bin/germinate -s ubuntu.oneiric -m http://archive.ubuntu.com/ubuntu/ -d oneiric -a amd64 -c main,universe ubuntu-desktop
cd ..

Expected results:
both runs run in a reasonable amount of time.

Actual results:
When run on a 2.4-GHz i5 system with 4GB RAM and a 10-Mbit connection, 11.10 amd64, latest is about 34 times slower than r376:

first run (r376):
i5384inputs+1123488outputs (0major+77624minor)pagefaults 0swaps

real 2m16.029s
user 1m51.047s
sys 0m2.656s

Second run (latest revision):
4055.88user 6.01system 1:08:02elapsed 99%CPU (0avgtext+0avgdata 845152maxresident)k
784inputs+1150432outputs (0major+72860minor)pagefaults 0swaps

real 68m2.499s
user 67m35.889s
sys 0m6.016s

Colin Watson (cjwatson) wrote :

I'm going to assume you meant "--seed-packages desktop/checkbox" here, since your original invocation crashes.

Colin Watson (cjwatson) wrote :

The slowdown appears to be in writing out the rdepends/ output tree. As a workaround, I recommend that you use the --no-rdepends option unless you actually use that output; it will speed up both old and new versions of germinate.

Changed in germinate (Ubuntu):
importance: Undecided → High
status: New → Triaged
summary: - Latest version is much slower than older ones
+ Latest version is much slower to write rdepends output than older ones
Colin Watson (cjwatson) on 2012-01-13
Changed in germinate (Ubuntu):
status: Triaged → In Progress
assignee: nobody → Colin Watson (cjwatson)
Colin Watson (cjwatson) wrote :

On porter-amd64.canonical.com, r376:

real 1m33.404s
user 1m30.330s
sys 0m1.780s

r477 (my fix):

real 1m38.430s
user 1m35.630s
sys 0m1.730s

That's close enough to the margin of error to be good enough for me. Just to repeat my point about using --no-rdepends if you can, though, r376 with --no-rdepends:

real 0m10.185s
user 0m8.800s
sys 0m0.350s

r477 with --no-rdepends:

real 0m10.003s
user 0m8.430s
sys 0m0.330s

Changed in germinate (Ubuntu):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package germinate - 2.5

---------------
germinate (2.5) unstable; urgency=low

  * Stop fetching Suggests from Packages; we don't use it.
  * Build a cache of seed entries when analysing reverse-dependencies, so
    that we don't have to expensively call get_seed_entries once per package
    per seed (LP: #915569).
  * Convert to setuptools.
  * Add the beginnings of a test suite.
  * Make the main bodies of scripts into modules so that they can be tested
    more easily.
  * Remove private AtomicUTF8File class; just write all files as UTF-8.

  * Port to Python 3:
    - Use "raise Exception(value)" syntax rather than the old-style "raise
      Exception, value".
    - Use Python 3-style print functions.
    - Use a list comprehension rather than filter (which behaves differently
      in Python 3).
    - Make GraphCycleError a subclass of Exception rather than of
      StandardError.
    - Simplify stringiness tests in TagFile.__init__ and Seed.__init__ using
      basestring.
    - Implement rich comparison methods instead of __cmp__ for Seed and
      GerminatedSeed. (Seed needs all of them for interface-compatibility;
      GerminatedSeed only needs __eq__ and __ne__.)
    - Use Python 3 replacements for urllib, urllib2, and ConfigParser if
      available.
    - When decompressing tag files from the archive, explicitly treat
      everything as binary data.
    - In Python 3, decode seed data read from URLs as UTF-8.
    - Pass universal_newlines=True to subprocess.Popen to get Unicode
      output.
    - Make sure to close stdout of subprocess.Popen objects.
    - Use 2to3 to handle the few remaining 2/3 differences at build time.
    - Add a python3-germinate package.

 -- Colin Watson <email address hidden> Tue, 17 Jan 2012 00:51:28 +0000

Changed in germinate (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers