[FFe] [MASTER] We need to better handle external payloads (Flash, msttcorefonts) not being available.

[Evan]

At the 11.10 release archive.canonical.com collapsed under heavy load and started returning 503 errors. This caused the flash package to be unable to grab its payload and thus caused the package to fail, which in turn caused the installation to fail.

The current plan is to add metadata to a package where it can add a URL to retrieve its data, so ubiquity and apt-get --download-only can cache it. We also need a way to queue up package installation post-install, for when we commit to installing flash but cannot do so due to network issues, or when the user wishes to have full language support post-install.

This change requires a freeze exception for precise, as it involves adding new features to update-notifier and significant reworking of the maintainer scripts for the affected packages. It also introduces new UI components; however these UI components are only visible in the event of a failure, so I don't think we should block on UI freeze for making this change given the significant benefit to the install experience when a download fails.

[Evan (ev)]

[11:03] ev: so we have an interesting problem around apt/dpkg
[11:03] ev: if the flashplugin installer package fails to get its payload from archive.canonical.com, the package installation fails. Yesterday, a lot of ubiquity installs failed because archive.c.c was under heavy load and returning 503 errors. Ubiquity tries to install the flash plugin as part of the "3rd party extras checkbox". If the package install fails, ubiquity fails, as dpkg will be in a bad state.
[11:03] ev: What are your thoughts on how to solve this?
[11:03] ev: James Troup suggested a canary in the preinst. A few of us came up with the idea of extending the package metadata to allow specifying a payload that can be locally cached by ubiquity or apt-get --download-only, as this is a problem for the Microsoft fonts as well
[11:03] ev: congratulations on the release, by the way
[11:03] ev: (if ubiquity failed to grab the payload, it wouldn't try to install the package)
[11:06] mvo: along the same line, we have bug #859373 where network-manager helpfully shutsdown networking during the upgrade (well, wifi, but still)
[11:07] ev: yay
[11:07] mvo: its a good question how to fix it, all preinst/postinst trickery will not really work because what we want is that the failure is not fatal but something like transient (until we have network again)
[11:08] mvo: but with the current dpkg we can either mark installed or failed, there will be no retry option
[11:08] ev: also, it would be nice to have a way to queue up retries
[11:08] ev: as we have language packs and whatnot that can't always be installed from ubiquity
[11:09] ev: when there's no network initially
[11:09] ev: and it would be great if we could mark them somewhere and have them show up in update-manager on first boot
[11:09] mvo: maybe we simply need a extension where packages just register a url and a script
[11:09] mvo: and something outside of the pakcage will then take care of it
[11:09] mvo: dpkg will consider it installed because the relevant data is on the system
[11:09] mvo: and that "other" can re-try, download etc at its leisure
[11:09] ev: hmm, could work
[11:10] mvo: hm, queing up packages too is interessting
[11:10] ev: yeah, right now it's a mess
[11:10] mvo: indeed
[11:10] ev: we have that update notifier dialog for language packs
[11:10] ev: which is less than ideal
[11:11] mvo: *FAR from it*
[11:11] mvo: yeah
[11:11] ev:
[11:11] mvo: ok, so two new bits: a) download my external resource framework b) queue pkg install for later
[11:11] ev: indeed
[11:11] mvo: we could abuse apt-get dselect-upgrade for the later
[11:12] ev: oh nice
[11:12] ev: I was unaware of that little gem
[11:14] mvo: it means we mess with the dselect states, but I guess only colin and the other remaining dselect users will care
[11:14] mvo: s/users/user/
[11:14] ev: lol
[11:14] ev: I'm sure they'll live
[11:16] mvo: yeah
[11:22] mvo: I get some lunch now and then I check out (b), but I think that is really straightforward with 'echo "2vcard install" | dpkg --set-selections' and u-m should not really need much changes for this either
[11:22] ev: great! thanks!
[12:45] mvo: I commited code to u-m in trunk that will honor the dpkg ...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/876298

[LAZA (laza74)]

Installation routine crashed several times in an VM.

Host: Natty amd64
Virtualbox: 4.1.8
Guest: Precise Beta2 amd64 desktop

[Mark Ellse (markellse)]

Affects USB install on Precise alpha2, but not Precise alpha1.

[intherye (intherye)]

Affects also installation from Xubuntu 12.04 Alpha 2 desktop installation CD.

Disabling installation of third party software (MP3, flash) in the graphical installer seems to be a workaround.

[Barry Warsaw (barry)]

mvo: I see that self._apply_dselect_upgrade() was disabled. Can you provide some background? Will this actually not work or not fix the problem? Do you have any other thoughts on what u-m could do better here?

[Barry Warsaw (barry)]

One thought, which I'm sure is horrible. u-m could have a list of packages in its conf file which contain external payloads, and if one of these packages fails to install, it could essentially ignore that (with a warning, dialog, or setting to control this) and continue on with the installation. It would be kind of automatically retried the next time u-m ran since it would still see that package as needing installation. Not sure what would happen if the failed-to-install package had dependencies though.

[Petr Herynk (pherynk)]

Installation crashed

[Andy Braben (abraben)]

Alpha 2 will still crash with a USB install because it will not be updated.
On 9 Feb 2012 21:27, "Petr Herynk" <email address hidden> wrote:

> Installation crashed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/876298
>
> Title:
> [MASTER] We need to better handle external payloads (Flash,
> msttcorefonts) not being available.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/876298/+subscriptions
>

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[iGadget (igadget)]

@ Andy - It crashes here as well while installing it on a VM using the ISO as virtual CDROM. So it's not limited to USB.

[Mike Siemon (msiemon13)]

I was trying to install to a seperate partition on a Dell Latitude E-6400 via a DVD. This is the first time I have tried installing from the DVD and it is also the first time I've tried installing the 64bit version of Ubuntu on this machine.

[Michael Vogt (mvo)]

So I was thinking about something like:
/usr/share/fetchme/requests.d/

that contains files like "flashplugin.fetchme":
"""
Url: http://example.com/flash.bin
Sha256: 123213

Url: http://example.com/flash-me-more.bin
Sha256 22213

Script: /usr/share/flashplugin/downloaded.sh
"""

We add a dpkg trigger on that dir a cron job (daily?) that regularly scan it and look for new files
there. If a new file is there the script tries download all files and run the script with the local path
of the downloaded files as its argument and records it is successful if the script exits with 0. That
means that if the connection is there essentially nothing changes, but if its not its queued and retried
later without failing the entire operation.

We need to warn that the script can not do any dpkg operations and that the system may be in a
state of flux when its run. This maybe a problem for e.g. update-alternatives calls, we need to think
a bit more what to do here (Script-Run-After-Dpkg-Finished: register.sh maybe?).

The other bit that is tricky is that we need to communicate failures in some way - or at least failures
that are permanent. (for permanent e.g. more than 3 days). One way could be to create a update-notifier
notification and to provide a cli tool like fetchme --show-queue or --show-incomplete. Plus a clear message
in the apt log everytime its triggered what is missing and that it queued it for later if it has to.

[Nelo (nelo)]

Ubiquity crashes when I am installing Ubuntu 12.04 alfa2 on a VM using the ISO as virtual CDROM.

[oCean (alex.harkema)]

Latest daily build same behaviour: installation fails. However, when _not_ selecting 3rd party software, install finishes just fine.

Host: Natty amd64
Virtualbox: 4.1.8
Guest: Precise Alpha 2 i386 Desktop

[fox144 (zacharovannyu)]

Lubuntu daily build 03.03.2012: installation fails

[kenneth bates (kenb8s)]

Hi,
I did a re-install but deselected the Fluendo MP3 Plugin and Ubuntu installed OK this time.Assuming that it would be desirable to have the MP3 plugin, can I install it later?
In Grub Boot Manager, can I change the default OS from Ubuntu to Windows 7 and increase the time delay?Any help gratefully received.
Ken Bates

> Date: Mon, 19 Mar 2012 15:15:13 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 876298] Re: [MASTER] We need to better handle external payloads (Flash, msttcorefonts) not being available.
>
> ** Tags added: rls-p-tracking
>
> ** Changed in: ubiquity (Ubuntu)
> Importance: High => Critical
>
> ** Changed in: update-manager (Ubuntu)
> Importance: High => Critical
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (954521).
> https://bugs.launchpad.net/bugs/876298
>
> Title:
> [MASTER] We need to better handle external payloads (Flash,
> msttcorefonts) not being available.
>
> Status in “diod” package in Ubuntu:
> Confirmed
> Status in “ubiquity” package in Ubuntu:
> Triaged
> Status in “update-manager” package in Ubuntu:
> Triaged
>
> Bug description:
> At the 11.10 release archive.canonical.com collapsed under heavy load
> and started returning 503 errors. This caused the flash package to be
> unable to grab its payload and thus caused the package to fail, which
> in turn caused the installation to fail.
>
> The current plan is to add metadata to a package where it can add a
> URL to retrieve its data, so ubiquity and apt-get --download-only can
> cache it. We also need a way to queue up package installation post-
> install, for when we commit to installing flash but cannot do so due
> to network issues, or when the user wishes to have full language
> support post-install.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/diod/+bug/876298/+subscriptions

[Steve Langasek (vorlon)]

If we add a dpkg trigger for this, the trigger needs to succeed regardless of whether the download was successful, otherwise we run into the same problem again. Michael proposes the use of an update-notifier interactive upgrade hook (https://wiki.ubuntu.com/InteractiveUpgradeHooks) in this case, to ensure the user gets feedback about the failure even though the package has successfully installed.

Then the cron job can take care of retrying, and sending another notification when it's decided the failure is "permanent".

[Steve Langasek (vorlon)]

preliminary flashplugin-nonfree package using this interface available at ppa:vorlon/ppa.

[Steve Langasek (vorlon)]

> We need to warn that the script can not do any dpkg operations and that the
> system may be in a state of flux when its run. This maybe a problem for e.g.
> update-alternatives calls, we need to think a bit more what to do here
> (Script-Run-After-Dpkg-Finished: register.sh maybe?).

I don't see any reason that we shouldn't be able to call update-alternatives - it's normal to run this both in a maintainer script context and outside of it, and running it from a triggered postinst or a cronjob should be no exception. FWIW, flashplugin-nonfree uses u-a and it's working reliably here.

>The other bit that is tricky is that we need to communicate failures in some
> way - or at least failures that are permanent. (for permanent e.g. more than
> 3 days). One way could be to create a update-notifier notification and to
> provide a cli tool like fetchme --show-queue or --show-incomplete. Plus a
> clear message in the apt log everytime its triggered what is missing and that
> it queued it for later if it has to.

So far what I've implemented is a notification popping up each time the script runs and fails to process the data. There's not yet any tracking of whether a download failure is "permanent" or a different error message in that case; nor is there particularly clear error output in the apt log yet.

An open question is how failing downloads should be handled once we've decided the error is permanent. Should we continue to try to download these and just avoid raising notifications about the failures? Or should we stop trying to download them at all? If we stop trying to download them, I guess we want a mechanism for users to forcibly retry these downloads once they've fixed their system.

[Michael Vogt (mvo)]

The update-alternatives warning was mostly a precaution measure as I'm not sure if that is doing proper locking. If it does thats fine, but there is a race otherwise that a package may call it on something when the cron job runs and tries to modify it too. If that is a non-issue even better :)

About the failed downloads> Without knowing the implementation (yet) I would assume we define states like:
 needs-downloading, fetched, failed-permanently
and a way to list the state of the current queue for the user. The failed-permanently one is tricky as it may be that its
a system that is very rarely connected to the net (so ifup-down integration would be good) so a fixed number of days
is not ideal (if the system goes online only every week). But at the same time daily retries on a resource that is no longer
available forever is equally bad. So maybe a combination of both? Say 14 days or 10 retries (when there was a working
network connect)?

[Steve Langasek (vorlon)]

Requesting a FFe for this new feature. Code's not 100% ready, but as Michael has agreed with the overall direction, I think we should get some release team eyeballs on it.

[Kate Stewart (kate.stewart)]

Approved. Please try to land this between March 30 and April 5, so we can get as much testing on it as possible.
It would be good to get this sorted before the archive starts getting with the release.

[Launchpad Janitor (janitor)]

This bug was fixed in the package flashplugin-nonfree - 11.2.202.228ubuntu2

---------------
flashplugin-nonfree (11.2.202.228ubuntu2) precise; urgency=low

  * Depend on update-notifier-common (>= 0.119ubuntu2) and use a
    package-data-download hook instead of doing the download in the postinst;
    this way, network failures don't cause the package to fail to install
    and we can still let the user know about the failure and deal with it
    out of band. LP: #876298.
  * Change the package to use dh(1) instead of cdbs, with extreme prejudice.
  * Drop the debconf question asking whether to download; it's not a usefully
    worded question and it's never being shown anyway so is useless for
    purposes of asking users to agree to a license.
  * 'db_text' is not a debconf command, so this question was never being
    shown - the script was only succeeding because of the '|| true'
    afterwards. Change to db_input...
  * Further cleanup of the debconf .config script.
  * Have flashplugin-downloader depend on ${misc:Depends}; will probably
    never amount to anything, but shuts lintian up.
 -- Steve Langasek <email address hidden> Thu, 05 Apr 2012 22:18:02 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 0.119ubuntu2

---------------
update-notifier (0.119ubuntu2) precise; urgency=low

  * data/package-data-downloader, data/package-data-downloads-failed.in,
    data/package-data-downloads-failed-permanently.in,
    debian/update-notifier-common.{postinst,triggers}: add a new handler and
    dpkg trigger to let packages queue data for download after package
    install, so that network connectivity problems don't make installs
    and upgrades unreliable. LP: #876298.
  * debian/update-notifier-common.cron.daily: add a cronjob to periodically
    retry any failed downloads.
  * src/update-notifier.c: when there are new hooks, check them whether or
    not dpkg has run; this allows other packages to send us notifications
    by ways other than running a package maintainer script.
 -- Steve Langasek <email address hidden> Thu, 05 Apr 2012 22:04:05 -0700

[Steve Langasek (vorlon)]

I have now created https://wiki.ubuntu.com/Specs/UpdateNotifierPackageDataDownloader to document the interface between update-notifier and the consuming packages.

[Launchpad Janitor (janitor)]

This bug was fixed in the package msttcorefonts - 3.4ubuntu3

---------------
msttcorefonts (3.4ubuntu3) precise; urgency=low

  * Migrate to use an update-notifier hook, so that package data can be
    handled asynchronously and not cause install failures on network
    problems. LP: #876298.
  * Add missing pre-depends for use of debconf in preinst.
  * Convert to dh(1).
 -- Steve Langasek <email address hidden> Sun, 08 Apr 2012 22:36:25 -0700

[Laurent Dard (f-couperin)]

I try to install the flash plugin behind a firewall but there is a serious regression
with flashplugin-installer=11.2.202.235ubuntu0.12.04.1 (11.2.202.235ubuntu0.11.10.1 works fine, see below)

Logs:

# apt-get install flashplugin-installer=11.2.202.235ubuntu0.11.10.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  ttf-xfree86-nonfree xfs
The following packages will be DOWNGRADED:
  flashplugin-installer
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 0 B/8740 B of archives.
After this operation, 32.8 kB disk space will be freed.
Do you want to continue [Y/n]?
Preconfiguring packages ...
dpkg: warning: downgrading flashplugin-installer from 11.2.202.235ubuntu0.12.04.1 to 11.2.202.235ubuntu0.11.10.1.
(Reading database ... 683933 files and directories currently installed.)
Preparing to replace flashplugin-installer 11.2.202.235ubuntu0.12.04.1 (using .../flashplugin-installer_11.2.202.235ubuntu0.11.10.1_i386.deb) ...
Unpacking replacement flashplugin-installer ...
Processing triggers for update-notifier-common ...
Setting up flashplugin-installer (11.2.202.235ubuntu0.11.10.1) ...
Downloading...
--2012-05-15 12:35:38-- http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.235.orig.tar.gz
Connecting to 10.195.111.98:80... connected.
Proxy request sent, awaiting response... 200 OK
Length: 14131146 (13M) [application/x-gzip]
Saving to: `./adobe-flashplugin_11.2.202.235.orig.tar.gz'

     0K ........ ........ ........ ........ ........ ........ 22% 1.68M 6s
  3072K ........ ........ ........ ........ ........ ........ 44% 1.34M 5s
  6144K ........ ........ ........ ........ ........ ........ 66% 1.42M 3s
  9216K ........ ........ ........ ........ ........ ........ 89% 1.56M 1s
 12288K ........ ........ ....... 100% 1.59M=9.0s

2012-05-15 12:35:47 (1.50 MB/s) - `./adobe-flashplugin_11.2.202.235.orig.tar.gz' saved [14131146/14131146]

Download done.
Flash Plugin installed.

# apt-get install flashplugin-installer=11.2.202.235ubuntu0.12.04.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  ttf-xfree86-nonfree xfs
The following packages will be upgraded:
  flashplugin-installer
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/8346 B of archives.
After this operation, 32.8 kB of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 683919 files and directories currently installed.)
Preparing to replace flashplugin-installer 11.2.202.235ubuntu0.11.10.1 (using .../flashplugin-installer_11.2.202.235ubuntu0.12.04.1_i386.deb) ...
Unpacking replacement flashplugin-installer ...
Processing triggers for update-notifier-common ...
flashplugin-installer: downloading http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.235.orig.tar.gz
Traceback (most recent call last):
  File "/usr/lib/update-notifier/package-data-downloader", line 234, in process_download_requests
    dest_file = urllib.urlretrieve(files[i])[0]
  File "/usr/lib/python2.7/url...

[Antek (antmoszyns)]

thank you

2012/5/15 Laurent Dard <email address hidden>

> I try to install the flash plugin behind a firewall but there is a serious
> regression
> with flashplugin-installer=11.2.202.235ubuntu0.12.04.1
> (11.2.202.235ubuntu0.11.10.1 works fine, see below)
>
> Logs:
>
> # apt-get install flashplugin-installer=11.2.202.235ubuntu0.11.10.1
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Suggested packages:
> ttf-xfree86-nonfree xfs
> The following packages will be DOWNGRADED:
> flashplugin-installer
> 0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not
> upgraded.
> Need to get 0 B/8740 B of archives.
> After this operation, 32.8 kB disk space will be freed.
> Do you want to continue [Y/n]?
> Preconfiguring packages ...
> dpkg: warning: downgrading flashplugin-installer from
> 11.2.202.235ubuntu0.12.04.1 to 11.2.202.235ubuntu0.11.10.1.
> (Reading database ... 683933 files and directories currently installed.)
> Preparing to replace flashplugin-installer 11.2.202.235ubuntu0.12.04.1
> (using .../flashplugin-installer_11.2.202.235ubuntu0.11.10.1_i386.deb) ...
> Unpacking replacement flashplugin-installer ...
> Processing triggers for update-notifier-common ...
> Setting up flashplugin-installer (11.2.202.235ubuntu0.11.10.1) ...
> Downloading...
> --2012-05-15 12:35:38--
> http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.235.orig.tar.gz
> Connecting to 10.195.111.98:80... connected.
> Proxy request sent, awaiting response... 200 OK
> Length: 14131146 (13M) [application/x-gzip]
> Saving to: `./adobe-flashplugin_11.2.202.235.orig.tar.gz'
>
> 0K ........ ........ ........ ........ ........ ........ 22% 1.68M 6s
> 3072K ........ ........ ........ ........ ........ ........ 44% 1.34M 5s
> 6144K ........ ........ ........ ........ ........ ........ 66% 1.42M 3s
> 9216K ........ ........ ........ ........ ........ ........ 89% 1.56M 1s
> 12288K ........ ........ ....... 100%
> 1.59M=9.0s
>
> 2012-05-15 12:35:47 (1.50 MB/s) - `./adobe-
> flashplugin_11.2.202.235.orig.tar.gz' saved [14131146/14131146]
>
> Download done.
> Flash Plugin installed.
>
>
> # apt-get install flashplugin-installer=11.2.202.235ubuntu0.12.04.1
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Suggested packages:
> ttf-xfree86-nonfree xfs
> The following packages will be upgraded:
> flashplugin-installer
> 1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> Need to get 0 B/8346 B of archives.
> After this operation, 32.8 kB of additional disk space will be used.
> Preconfiguring packages ...
> (Reading database ... 683919 files and directories currently installed.)
> Preparing to replace flashplugin-installer 11.2.202.235ubuntu0.11.10.1
> (using .../flashplugin-installer_11.2.202.235ubuntu0.12.04.1_i386.deb) ...
> Unpacking replacement flashplugin-installer ...
> Processing triggers for update-notifier-common ...
> flashplugin-installer: downloading
> http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_11.2.202.235.orig.tar.gz
> Traceback (most recent...

[Steve Langasek (vorlon)]

This has nothing to do with the adobe-flash-plugin-tools project. Marking invalid.

The behavior you're seeing is by design. If you want flashplugin-nonfree to use a proxy for downloads, you need to set this via the http_proxy environment variable, and not just using the apt proxy settings. See bug #979477 for the justification for this behavior change.

If you're using the desktop proxy configuration dialog, this will be done automatically for you. Otherwise, you can manually set http_proxy in /etc/environment. Regardless, note that there is a bug in sudo that currently prevents /etc/environment from being applied to sudo sessions: bug #982684.

[Dom Spencer (domspencer11)]

Does this in anyway affect my duel boot system? ubuntu / Windows 7. Guess I'm about to find out

[Mark (makjegi)]

Uhmm... I;m new to this. could you please check the error out for me. thank you.

[Mark (makjegi)]

and this one to...

[Mark (makjegi)]

and here's the last one.