DVD install takes forever compared to the CD install during the step that it's calculating packages to remove, causing pain for OEM

[Jerone Young]

This bug is related to LP#290400 . OEM is feeling real pain as they use the DVD install. The DVD install is taking an extremely long time to calculate the packages that are to be removed as compared to the CD install.

OEM tries to work around this issue the best they can by some preseed hacks to not let it remove *any* language packs, or fonts or anything during that stuff. The only packages that end up getting removed are the ubiquity ones and their dependencies. I've attached a relative snippet of the hack that OEM is using today as a workaround in there preseed:

<snip>
 # *Horrible* hack to work around bug 290400
 # this should be addressed for Jaunty and no longer necessary then
 d-i pkgsel/language-pack-patterns string language-pack-gnome-$LL icedtea6-plugin ttf-dzongkha ttf-arphic-gbsn00lp ttf-arphic-bsmi00lp openoffice.org-hyphenation openoffice.org-filter-binfilter libgcj-common m17n-db m17n-contrib default-jre default-jre-headless libgcj9-jar libxalan2-java-gcj libxerces2-java-gcj libjaxp1.3-java-gcj libgcj-bc libgcj9-0 gcj-4.3-base bsh-gcj
</snip>

This situation gets worst as there are future machines that are going to be using Intel Atom processors. OEM can't afford to have a 12-25 minute extra step that does nothing. It's taking that long because the atom proc is slow.

[Henrik Nilsen Omma (henrik)]

AFAICS this is really a duplicate of #290400 rather than just related. I've added the extra tracking tasks to #290400 and will mark this as a dupe.

[Mario Limonciello (superm1)]

No, that's related, but even with the workaround for those packages,
the method is taking much longer on the DVD.

On 03/03/2009, Henrik Nilsen Omma <email address hidden> wrote:
> *** This bug is a duplicate of bug 290400 ***
> https://bugs.launchpad.net/bugs/290400
>
> AFAICS this is really a duplicate of #290400 rather than just related.
> I've added the extra tracking tasks to #290400 and will mark this as a
> dupe.
>
> ** This bug has been marked a duplicate of bug 290400
> DVD livefs always removes java packages
>
> --
> DVD install takes forever compared to the CD install during the step that
> it's calculating packages to remove, causing pain for OEM
> https://bugs.launchpad.net/bugs/335596
> You received this bug notification because you are a member of Ubuntu
> Installer Team, which is subscribed to ubiquity in ubuntu.
>

--
Sent from my mobile device

Mario Limonciello
<email address hidden>

[Jerone Young (jerone)]

We broke this up from being a duplicate as it as at a different scope then Bug #209400. This discussion was made during our oem-priority meeting.

[Colin Watson (cjwatson)]

I'm not sure that unduplicating these was all that helpful, to be honest, since I think we ought to deal with the whole lot at once; but what the heck I'll roll with the punches. Assigning to myself.

[Colin Watson (cjwatson)]

There are two key problems here. The first is that trying to remove even a single package, which we do when testing whether we should not bother to copy its files if we're just going to remove the package anyway, is relatively slow (well, on the order of a quarter to half a second for me, but it adds up when you're talking about hundreds of packages). The second is that, if a package can't be removed because it's needed by a language pack that's staying installed, then having to repeatedly try to remove it and then put it back each time we look at one of its dependencies is *really* slow. This second problem is particularly noticeable when you ask for lots of language packs to stay installed.

I've decided to address the second problem first, and have applied this change to ubiquity:

  * Expand dependencies of packages we know we want to keep (language packs,
    etc.) before calculating which packages to blacklist from file copying
    or to remove. This is more correct in the presence of Recommends of
    language packs, and furthermore saves considerable time when
    blacklisting. My test results for various language pack sets:
    - en: 4:00 -> 3:30
    - de+en+es+fr+ja+ko+nb+nds+nl+nn+si+sk+sv: 5:30 -> 2:50
    - all: 14:37 -> 0:10 (!)
    This doesn't quite solve LP #335596 because testing a large number of
    packages for removal when there genuinely are lots of packages to remove
    is still quite slow, but it very significantly improves the worst cases.

Is it acceptable to you to continue installing all language packs? If so, I think this should clear this bug off the oem-priority list; 10 seconds is easily going to be less than the time saved by not having to copy all those files. If that isn't acceptable, then we'll need to do some more work.

[Colin Watson (cjwatson)]

Further partial improvement for ubiquity 1.11.18:

  * broken_packages is fairly slow due to having to iterate over the whole
    cache. Speed it up a bit by stopping when the number of broken packages
    found reaches cache._depcache.BrokenCount; this improves blacklist
    calculation time for the previously-mentioned DVD English-only install
    from 3:30 to 2:30 (see LP #335596).

[Mario Limonciello (superm1)]

This appears to be a *significant* improvement. I tested it on fast hardware with Ubiquity 1.11.18 built from bzr and the DVD livefs from A6. Normally from the start of X in automatic ubiquity mode to reboot it was about 15 minutes. Now it's down to 10 minutes. The portion of the process that normally spun hanging (determining removals) flew by.

Myself or Veronica will verify this with an atom based system on Monday to see that it's adequately quick enough, which I anticipate it is.

[Colin Watson (cjwatson)]

Glad to hear it. Make sure to add 'ubiquity ubiquity/keep-installed string icedtea6-plugin openoffice.org' to your preseeding too (unless you already have something with the same effect); this will be the default for any images built from now on (I just deployed that change on cdimage) but wasn't in Alpha 6. I'd expect it to make a bit of a dent.

[Mario Limonciello (superm1)]

OK, so after some unrelated technical difficulties on the atom platform I was testing with, I've got some data points for it.

Using the same test scenarios as the fast hardware, I tested on an Inspiron Mini 10:
w/ ubiquity 1.11.16 & A6 DVD: 44 minutes
w/ ubiquity 1.11.18 & A6 DVD: 24 minutes

The hard hitters appear to be resolved, and I think this bug is good to mark as Fixed now. I'm thinking that its going to boil down to hardware limitations at this point. If there are any other specific portions of the install that end up being lengthy that can be improved, we can reraise them as separate items.

[Mario Limonciello (superm1)]

Closing the ubiquity task as it looks a character was missing in the changelog to do it automagically:

1.11.18
Published in jaunty-release on 2009-03-16

ubiquity (1.11.18) jaunty; urgency=low

  [ Colin Watson ]
  * broken_packages is fairly slow due to having to iterate over the whole
    cache. Speed it up a bit by stopping when the number of broken packages
    found reaches cache._depcache.BrokenCount; this improves blacklist
    calculation time for the previously-mentioned DVD English-only install
    from 3:30 to 2:30 (see LP #335596).
  * GTK frontend:
    - Restore set_window_hints method for use by windows other than the main
      one (it was still called in the Glade file), just in case we're using
      a window manager that pays attention to this. In these cases
      maximisation doesn't really make sense so we no longer permit that.

  [ Evan Dandrea ]
  * Fix a bit of code that wasn't updated to reflect other changes in
    remove_extras (LP: #342319).
  * Start NetworkManager before ubiquity in only-ubiquity mode
    (LP: #340929).
  * Support the new partman/unmount_active question.
  * Automatic update of included source packages: grub-installer
    1.36ubuntu3, partman-base 129ubuntu2, partman-target 58ubuntu5.

 -- Evan Dandrea < <email address hidden>> Mon, 16 Mar 2009 17:02:01 +0000

[Colin Watson (cjwatson)]

I actually left that character out intentionally since I wanted to wait for feedback from you before closing the bug. :-) Thanks for the testing!

[Timothy R. Chavez (timrchavez)]

The bug task for the somerville project has been removed by an automated script. This bug has been cloned on that project and is available here: https://bugs.launchpad.net/bugs/1305931