cdromupgrade from Lucid to Precise failed with unmet dependencies without network connection

[Jean-Baptiste Lallement]

RATIONALE
Unable to upgrade from lucid to precise using only the alternate media as a package source.

TEST CASE
1. Install Lucid and updates
2. Disconnect the network cable
3. Insert Precise Alternate CD
4. Start cdromupgrade from the root directory of the media

On top of these tests, a standard oneiric -> precise and lucid -> precise (with network) upgrade run should be performed to ensure no extra regression in the way the upgrade is resolved.

REGRESSION POTENTIAL
It's not impossible that these fixes further confuse apt in some upgrade scenario, the planned testing should be able to find any obvious case of that.

ACTUAL RESULT
Upgrader fails to calculate the upgrade.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: update-manager 1:0.134.12.1
ProcVersionSignature: Ubuntu 2.6.32-41.94-generic 2.6.32.59+drm33.24
Uname: Linux 2.6.32-41-generic x86_64
Architecture: amd64
Date: Thu Jul 26 18:04:23 2012
GConfNonDefault:
 /apps/update-manager/check_new_release_ignore=
 /apps/update-manager/first_run=false
 /apps/update-manager/show_details=false
 /apps/update-manager/show_versions=false
 /apps/update-manager/window_size=(600,520)
InstallationMedia: Ubuntu 10.04.3 LTS "Lucid Lynx" - Release amd64 (20110720.1)
PackageArchitecture: all
ProcEnviron:
 LANG=fr_FR.utf8
 SHELL=/bin/bash
SourcePackage: update-manager
VarLogDistupgradeTermlog:

[Stéphane Graber (stgraber)]

Doing some tests on this one, it looks like it only fails if your list aren't up to date before trying the update.
If you run apt-get update, then go offline and choose to run the upgrade in offline mode, it'll resolve the dependencies just fine.

[Stéphane Graber (stgraber)]

And then fail miserably half way through the fetching process :)

[Stéphane Graber (stgraber)]

http://paste.ubuntu.com/1135477 contains the list of packages that need to be downloaded from the internet to make the update succeed.

[Steve Langasek (vorlon)]

http://paste.ubuntu.com/1135477 seems to be the entire opposite of a CD-only upgrade. E.g.,

> Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mono/libmono-corlib2.0-cil_2.10.8.1-1ubuntu2.2_all.deb Could not resolve 'security.ubuntu.com'

So you've told apt about versions of packages from precise-security via apt-get update, then taken the machine offline? Not surprising that it then fails the upgrade.

This is meant to work including for systems that *can't* run apt-get update; and needs to be fixed by analyzing the upgrade errors in the apt upgrade log, and getting those packages included in the package pool on the alternate CD, or otherwise hinting packages out of the way that can't be installed.

Working back from the end of that log, the errors I see are:

    Installing liblaunchpad-integration-3.0-1 as Depends of gnome-system-log
      Installing liblaunchpad-integration-common as Depends of liblaunchpad-integration-3.0-1
[...]
Investigating (0) liblaunchpad-integration-common [ amd64 ] < none -> 0.1.56 > ( libs )
Broken liblaunchpad-integration-common:amd64 Breaks on liblaunchpad-integration1 [ amd64 ] < 0.1.35 > ( libs ) (< 0.1.41)
  Considering liblaunchpad-integration1:amd64 32 as a solution to liblaunchpad-integration-common:amd64 25
  Holding Back liblaunchpad-integration-common:amd64 rather than change liblaunchpad-integration1:amd64
[...]
Investigating (1) liblaunchpad-integration-3.0-1 [ amd64 ] < none -> 0.1.56 > ( libs )
Broken liblaunchpad-integration-3.0-1:amd64 Depends on liblaunchpad-integration-common [ amd64 ] < none -> 0.1.56 > ( libs )
  Considering liblaunchpad-integration-common:amd64 25 as a solution to liblaunchpad-integration-3.0-1:amd64 75
  Holding Back liblaunchpad-integration-3.0-1:amd64 rather than change liblaunchpad-integration-common:amd64
[...]
Broken libgdu-gtk0:amd64 Breaks on gnome-disk-utility [ amd64 ] < 2.30.1-1 -> 3.0.2-2ubuntu7 > ( admin ) (< 3.0.0)
  Considering gnome-disk-utility:amd64 1 as a solution to libgdu-gtk0:amd64 2
  Upgrading gnome-disk-utility:amd64 due to Breaks field in libgdu-gtk0:amd64
Investigating (9) gnome-disk-utility [ amd64 ] < 2.30.1-1 -> 3.0.2-2ubuntu7 > ( admin )
Broken gnome-disk-utility:amd64 Depends on liblaunchpad-integration-3.0-1 [ amd64 ] < none -> 0.1.56 > ( libs ) (>= 0.1.17)
  Considering liblaunchpad-integration-3.0-1:amd64 75 as a solution to gnome-disk-utility:amd64 1
  Holding Back gnome-disk-utility:amd64 rather than change liblaunchpad-integration-3.0-1:amd64

So something is preventing liblaunchpad-integration from being upgraded, and it needs to be hit with a bigger hammer.

[Steve Langasek (vorlon)]

$ grep liblaunchpad-integration ../simple/precise/ubuntu-12.04-alternate-amd64.list
/pool/main/l/launchpad-integration/liblaunchpad-integration-3.0-1_0.1.56_amd64.deb
/pool/main/l/launchpad-integration/liblaunchpad-integration-common_0.1.56_all.deb
$

liblaunchpad-integration1 isn't present on the alternate CD, so the version seen here is the one from lucid which has a Breaks: from liblaunchpad-integration-common.

Looking more closely at what wants liblaunchpad-integration1:

  Installing liblaunchpad-integration1 as Depends of libbrasero-media0
--> obsolete library.

apt later reconsiders libbrasero-media0 itself:

Investigating (0) libbrasero-media0 [ amd64 ] < 2.30.2-0ubuntu1.1 > ( libs )
Broken libbrasero-media0:amd64 Depends on brasero-common [ amd64 ] < 2.30.2-0ubuntu1.1 -> 3.4.1-0ubuntu1 > ( gnome ) (< 2.31)
  Considering brasero-common:amd64 10 as a solution to libbrasero-media0:amd64 -1
  Removing libbrasero-media0:amd64 rather than change brasero-common:amd64

But this happens too late to get liblaunchpad-integration-3.0-1 reconsidered.

An artificial Breaks: from liblaunchpad-integration-3.0-1 to libbrasero-media0 might be a sufficient hammer. I'd suggest testing with a mocked-up Packages file that introduces such a breaks, rather than going through an SRU cycle to test this.

[Stéphane Graber (stgraber)]

Thanks for the analysis of the log, I'll take a look at adding that Breaks.

I was using an up to date precise build, built 5 minutes before the test, so nothing in the archive should have been more recent that what was on the media, therefore the list I attached should still match what's missing from the media to make a clean successful upgrade. Any tweaks we do will just make a partial upgrade possible with a result quite different from what you'd get if you were online.

[Stéphane Graber (stgraber)]

Hmm, well, we have a problem:
stgraber@castiana:~/Desktop/sru$ rmadison libbrasero-media0
libbrasero-media0 | 2.30.0-0ubuntu1 | lucid | amd64, armel, i386, ia64, powerpc, sparc
libbrasero-media0 | 2.30.2-0ubuntu1.1 | lucid-updates | amd64, armel, i386, ia64, powerpc, sparc

[Stéphane Graber (stgraber)]

Right, so after some more debugging, thanks to Steve for parsing the apt output, the total list of problems is the following.

 - update-notifier-common attempts to run patch but doesn't depend on it, the failure is ignored so not causing a critical bug
 - the resolver fails to decide to remove librasero-media0 to make launchpad-integration installable
 - the resolver tries to keep the old evolution-plugins, breaking libnspr in the process
 - libgnome2 is required by the upgrade but not on the media
 - openoffice.org-core is needed by the upgrader but doesn't exist in the archive
 - some other openoffice.org packages are required by the upgrader but not on the media
 - the upgrader requires ".*screen.*" to be installed instead of "^screen$" making the upgrade fail if any package containing the work screen is removed by the upgrader

And the matching list of changes:
 - update-notifier-common needs to depend on patch
 - launchpad-integration needs to break on libbrasero-media0
 - libnspr4-0d needs to break on evolution-plugins (<< 3.2.0-0ubuntu2)
 - seed libgnome2 on alternate
 - add transitional package openoffice.org-core to openoffice.org
 - fix removal blacklist to use "^screen$"

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 0.120ubuntu2

---------------
update-notifier (0.120ubuntu2) quantal; urgency=low

  * Add dependency on patch to update-notifier-common as it's called
    from the cddistupgrader script when doing upgrades from an external
    media. (LP: #1029531)
 -- Stephane Graber <email address hidden> Wed, 08 Aug 2012 17:24:38 -0400

[Stéphane Graber (stgraber)]

Updated the seeds to include libgnome2-0 on the alternate media.

[Steve Langasek (vorlon)]

Hello Jean-Baptiste, or anyone else affected,

Accepted update-manager into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/update-manager/1:0.156.14.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Jean-Baptiste, or anyone else affected,

Accepted launchpad-integration into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/launchpad-integration/0.1.56.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Jean-Baptiste, or anyone else affected,

Accepted nspr into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nspr/4.8.9-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Jean-Baptiste, or anyone else affected,

Accepted update-notifier into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/update-notifier/0.119ubuntu8.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Jean-Baptiste, or anyone else affected,

Accepted openoffice.org into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/openoffice.org/1:3.3.0-7ubuntu7.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package launchpad-integration - 0.1.56.1

---------------
launchpad-integration (0.1.56.1) precise-proposed; urgency=low

  * Add Breaks: libbrasero-media0 to liblaunchpad-integration-3.0-1,
    required to get the dist-upgrader to resolve lucid -> precise.
    Unversioned breaks as the package last existed in lucid. (LP: #1029531)
 -- Stephane Graber <email address hidden> Wed, 08 Aug 2012 17:15:45 -0400

[Stéphane Graber (stgraber)]

I successfully upgraded an Ubuntu 10.04.4 desktop image to 12.04.1 using only the alternate media, so the resolver issues appear to be fixed. The test was done using an amd64 install.

However to consider this verification-done and have them copied to -updates (before the 7 days), we're still waiting on a no-regression confirmation from QA for automated 10.04 to 12.04 and 11.10 to 12.04 upgrades.

[Jean-Baptiste Lallement (jibel)]

I ran the following tests successfully:
- No network upgrade from Lucid up to date to 12.04.1 using alternate amd64
- No network upgrade from Lucid up to date to 12.04.1 using alternate i386

The auto upgrade tests (https://jenkins.qa.ubuntu.com/view/Precise/view/Upgrade%20Testing%20Dashboard/) ran successfully with -proposed enabled and no regression and no new failure have been identified.
I verified that proposed was enabled by checking the sources entries in main.log and comparing the version of the packages (update-manager, perl, libxml-sax-perl, launchpad-integration, update-notifier, libnspr4-0d, ...) in proposed with the version of the packages in term.log after upgrade.

Marking as verification-done

[Steve Langasek (vorlon)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 0.119ubuntu8.5

---------------
update-notifier (0.119ubuntu8.5) precise-proposed; urgency=low

  * Add dependency on patch to update-notifier-common as it's called
    from the cddistupgrader script when doing upgrades from an external
    media. (LP: #1029531)
 -- Stephane Graber <email address hidden> Wed, 08 Aug 2012 17:41:30 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package nspr - 4.8.9-1ubuntu2.1

---------------
nspr (4.8.9-1ubuntu2.1) precise-proposed; urgency=low

  * Add Breaks: evolution-plugins (<< 3.2.0-0ubuntu2) to libnspr4,
    required to get the dist-upgrader to resolve lucid -> precise.
    (LP: #1029531)
 -- Stephane Graber <email address hidden> Wed, 08 Aug 2012 17:17:49 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package launchpad-integration - 0.1.56.1

---------------
launchpad-integration (0.1.56.1) precise-proposed; urgency=low

  * Add Breaks: libbrasero-media0 to liblaunchpad-integration-3.0-1,
    required to get the dist-upgrader to resolve lucid -> precise.
    Unversioned breaks as the package last existed in lucid. (LP: #1029531)
 -- Stephane Graber <email address hidden> Wed, 08 Aug 2012 17:15:45 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-manager - 1:0.156.14.9

---------------
update-manager (1:0.156.14.9) precise-proposed; urgency=low

  * No change rebuild to pick up the fixed apt_clone.py

update-manager (1:0.156.14.8) precise-proposed; urgency=low

  * Fix removal_blacklist to blacklist "^screen$" instead of "screen".
    This fixes cases where the upgrade would fail because of a package
    containing "screen" being removed. (LP: #1029531)

update-manager (1:0.156.14.7) precise-proposed; urgency=low

  * DistUpgrade/DistUpgradeMain.py: call clone.save_state with
    scrub_sources set so that VarLogDistUpgradeAptclonesystemstate will be
    included in bug reports again (LP: #1029046)
  * DistUpgrade/DistUpgradeApport.py: check errormsg for the English version of
    the dependency problems error first (LP: #999890)
    - add apt-clone_system_state.tar.gz to white list of files to upload
  * In the apport source package hook collect apt-clone information if the bug
    report is about a distribution upgrade (LP: #1029046)
  * Don't throw exception on socket timeout when downloading metarelease file
    (LP: #818760)
 -- Stephane Graber <email address hidden> Thu, 09 Aug 2012 17:05:37 -0400

[Steve Langasek (vorlon)]

Further testing showed that the additional openoffice.org metapackage was neither necessary nor sufficient to fix the upgrade issues that we were seeing. Rescinding this SRU.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nspr - 4.8.9-1ubuntu3

---------------
nspr (4.8.9-1ubuntu3) quantal; urgency=low

  * Reupload to quantal.

nspr (4.8.9-1ubuntu2.3) precise-proposed; urgency=low

  * language-support-LANG was still around in 11.04 so the previous change
    is requiring a dist-upgrade for any user who upgraded from 11.04 to 11.10
    and then to 12.04, instead of being limited to machines that were
    originally installed with 9.04 as was planned.
    Removing language-support-LANG from the Breaks, which should resolve the
    issue for these users while still removing language-support-translations
    on systems. (LP: #1036794)

nspr (4.8.9-1ubuntu2.2) precise-proposed; urgency=low

  * Add additional Conflicts: on the obsolete evolution-documentation-*
    packages, which are held back on some users' systems despite being
    obsolete before 10.04; and also on the matching
    language-support-translations-* and language-support-* packages. This
    should ensure libnspr is getting upgraded, instead of evolution being
    held back. LP: #1036794.

nspr (4.8.9-1ubuntu2.1) precise-proposed; urgency=low

  * Add Breaks: evolution-plugins (<< 3.2.0-0ubuntu2) to libnspr4,
    required to get the dist-upgrader to resolve lucid -> precise.
    (LP: #1029531)
 -- Steve Langasek <email address hidden> Wed, 12 Sep 2012 08:13:48 -0700

[Beerslayer (beerslayer)]

I'm not an expert user, so there's probably a simple solution for this. Nonetheless, I'm still having (or I appear to be having) the exact problem described here. I'm trying to upgrade from lucid to precise using the alternate CD (and no network access), but the upgrade keeps failing, saying it "could not calculate the upgrade".

The precise alternate CD I'm using is one I just downloaded within the last few days (it's currently 20-Jan-2013) directly from the Ubuntu site. I would expect it to have all the fixes in precise described above, since they appear to have been released months ago, yet the upgrade still fails.

For what it's worth, I have enabled all of the standard lucid repositories (lucid, lucid-updates, lucid-backports) except for lucid-proposed, run apt-get update numerous times, and it doesn't help a bit.

I will be happy to provide any information I can retrieve for you, or open a separate bug report if necessary (and if I can figure out how), but please do not assume any expertise on my part - I can navigate OK and have no fear of the command line but I have only been using Linux (Ubuntu) regularly for a little over a year. If there are specific log files you need info from, you may need to tell me where they are - I haven't dug that deeply into the inner workings of Linux (yet).