[dist-upgrade] sources.list rewrite needs to be more robust

[Wesley Moore]

When the sources.list is rewritten it can happen that only parts are updated and vital bits like "deb http://$unknown_mirror/ubuntu breezy main" are commented out. This leads to random error messages like the ones below. After the sources.list rewrite and the apt-get update the upgrader should check if there are new candidate versions for various packages (apt, libc6) to heuristicly detect invalid rewrites.

The new aptsources has a concept of distro related sources. Perhaps we can reuse it here.

I am attempting to perform an upgrade of my i386 breezy install to dapper beta. I have set up the apt proxy by placing the following in /etc/apt/apt.conf.d/04proxy:
Acquire::http::Proxy "http://username:<email address hidden>:8080";

Steps:
1. run sudo update-manager -d
2. Click the Upgrade button
3. Click the Upgrade button on the release notes window
4. The Upgrading Ubuntu window shows and then reports the following when it reaches the "Downloading and installing the upgrades" step:

Could not calculate the upgrade

An unresolvable problem occured while calculating the upgrade. Please report this as a bug.

dist-upgrade-apt.log contains:
Starting
Starting 2
Done
MarkUpgrade() called on a non-upgrable pkg: 'ubuntu-desktop'
Starting
Starting 2
Investigating xchat-common
Package xchat-common has broken dep on xchat
  Considering xchat 10002 as a solution to xchat-common 0
Package xchat-common has broken dep on xchat-text
  Or group remove for xchat-common
Investigating ubuntu-desktop
Package ubuntu-desktop has broken dep on xchat
  Considering xchat 10002 as a solution to ubuntu-desktop 0
  Removing ubuntu-desktop rather than change xchat
Done

dist-upgrade.log contains:
2006-04-24 11:55:25,600 DEBUG Foreign:
2006-04-24 11:55:25,600 DEBUG Obsolete: alsa-modules-2.6.12-10-386 sun-j2sdk1.4
2006-04-24 11:55:25,917 DEBUG Marking 'ubuntu-desktop' for upgrade
2006-04-24 11:55:25,918 DEBUG Removing 'xchat' (ubuntu-desktop PostUpgradeRemove rule)
2006-04-24 11:55:25,923 DEBUG Removing 'xscreensaver' (ubuntu-desktop PostUpgradeRemove rule)
2006-04-24 11:55:25,924 DEBUG Purging 'xorg-common' (Distro PostUpgradePurge rule)
2006-04-24 11:55:25,950 DEBUG The package 'ubuntu-desktop' is marked for removal but it's in the removal blacklist
2006-04-24 11:55:46,939 ERROR Dist-upgrade failed: 'A essential package would have to be removed'

----
alsa-modules-2.6.12-10-386 was rebuilt from source to enable audio on this PC. sun-j2sdk1.4 was built using make-jpkg

[Adriaan Peeters (apeeters)]

Confirmed:

adpeeter@twiadria:/var/log$ cat dist-upgrade.log
2006-05-02 14:30:11,246 DEBUG Foreign: bzr libdvdcss2 w32codecs opera sun-j2sdk1.5 acroread bzr-doc java-package
2006-05-02 14:30:11,247 DEBUG Obsolete: network-manager-vpnc libexif10 kernel-image-2.6.14 libbind9-0 linux-image-2.6.10-5-386 libgnome-menu0 ubuntu-calendar libwvstreams3-base libdb4.2++ ubuntu-calendar-april libdevmapper1.00 libiw27 libiw28 dhcdbd libxaw8 music-applet libdns20 dnsutils wireless-tools libiw-dev libsoup2.2-7 network-manager-dev network-manager-openvpn icaclient libwnck16 ifrename sipcalc libecal1.2-2 computertemp libegroupwise1.2-5 libisccfg1 libisccc0 linux-restricted-modules-2.6.10-5-386 gnome-osd libisc7 libflac6 liblwres1 libdps1 libswig1.3.22 libisc9 libdns16
2006-05-02 14:30:12,009 DEBUG Marking 'ubuntu-desktop' for upgrade
2006-05-02 14:30:12,010 DEBUG Removing 'xchat' (ubuntu-desktop PostUpgradeRemove rule)
2006-05-02 14:30:12,014 DEBUG Removing 'xscreensaver' (ubuntu-desktop PostUpgradeRemove rule)
2006-05-02 14:30:12,014 DEBUG Purging 'xorg-common' (Distro PostUpgradePurge rule)
2006-05-02 14:30:12,034 DEBUG The package 'ubuntu-desktop' is marked for removal but it's in the removal blacklist

adpeeter@twiadria:/var/log$ cat dist-upgrade-apt.log
Starting
Starting 2
Done
MarkUpgrade() called on a non-upgrable pkg: 'ubuntu-desktop'
Starting
Starting 2
Investigating xchat-common
Package xchat-common has broken dep on xchat
  Considering xchat 10002 as a solution to xchat-common 0
  Removing xchat-common rather than change xchat
Investigating ubuntu-desktop
Package ubuntu-desktop has broken dep on xchat
  Considering xchat 10002 as a solution to ubuntu-desktop 0
  Removing ubuntu-desktop rather than change xchat
Done

[Michael Vogt (mvo)]

Thanks for this bugreport.

I got some reports about this, but I was never able to reproduce it myself. Could you please give it a new try in a 3-4h? I uploaded a new dist-ugprader that fixes some issues and it may fix this problem as well.

Thanks,
 Michael

[Michael Vogt (mvo)]

I reject this task because one task against update-manager is enough.

[Adriaan Peeters (apeeters)]

Now I get another message:
"Could not find release notes
The server might be overloaded."

A bit of debugging showed that there is a typo in the url for the annoucement:
ReleaseNotes: http://archive.ubuntu.com/ubuntu/dists/dapper/main/dist-upgrader-all/current/ReleaseAnouncement

That should be ReleaseAnnouncement (with double n).

[Adriaan Peeters (apeeters)]

The above is fixed now

[Adriaan Peeters (apeeters)]

I noticed I had xchat and xchat-common installed from backports, so I downgraded them to the breezy version.

adpeeter@twiadria:/var/log$ apt-cache policy xchat
xchat:
  Installed: 2.6.0-0ubuntu1~breezy1
  Candidate: 2.6.0-0ubuntu1~breezy1
  Version table:
 *** 2.6.0-0ubuntu1~breezy1 0
        500 http://archive.ubuntu.com breezy-backports/main Packages
        100 /var/lib/dpkg/status
     2.4.4-0ubuntu5 0
        500 http://ftp.belnet.be breezy/main Packages
adpeeter@twiadria:/var/log$ apt-cache policy xchat-common
xchat-common:
  Installed: 2.6.0-0ubuntu1~breezy1
  Candidate: 2.6.0-0ubuntu1~breezy1
  Version table:
 *** 2.6.0-0ubuntu1~breezy1 0
        500 http://archive.ubuntu.com breezy-backports/main Packages
        100 /var/lib/dpkg/status
     2.4.4-0ubuntu5 0
        500 http://ftp.belnet.be breezy/main Packages

I still get the same error message, attached is the dist-upgrade log. dist-upgrade-apt and dist-upgrade-term are empty.

[Adriaan Peeters (apeeters)]

I debugged a bit more and the upgrade seems to fail because of an unknown mirror:

2006-05-04 09:25:52,537 DEBUG entry '# deb http://ftp.belnet.be/packages/ubuntu/ubuntu breezy main restricted' was disabled (unknown mirror)

When I remove that line from my sources.list I get to the point where I get a list of to be changed packages. I did not continue (yet).

[Adriaan Peeters (apeeters)]

I do not know whether it is related, but in the same log file as the previous comment, I also see:

2006-05-04 09:25:57,693 DEBUG Marking 'ubuntu-desktop' for upgrade
2006-05-04 09:25:57,693 DEBUG no {ubuntu,edubuntu,kubuntu}-desktop pkg installed
2006-05-04 09:25:57,693 DEBUG guessing 'ubuntu-desktop' as missing meta-pkg

But ubuntu-desktop is installed:

$ dpkg -l ubuntu-desktop
ii ubuntu-desktop 0.80 The Ubuntu desktop system

[Adriaan Peeters (apeeters)]

Is there something else you want me to test before I do the upgrade?

[Adriaan Peeters (apeeters)]

Cannot test anymore, upgrade is done. But you might be able to reproduce this using the attached log.

[Wesley Moore (wjmoore)]

I still haven't successfully updated if there is anything I can do.

[Adriaan Peeters (apeeters)]

Try again and post your dist-upgrade.log (as attachment).

[Wesley Moore (wjmoore)]

I re-ran update-manager -d today and it successfully upgraded the system to Dapper beta. The log from the update is attached.

[Michael Vogt (mvo)]

@Wesley: Thanks for your logs, all looks good here. Has the your sources.list changed since the last attempt to upgrade?

@Adriaan: Thanks for your logs too, I added the belnet mirror to the list of valid mirros.

The problem is that the upgrader has problems detecting when the sources.list rewrite commted out the "main" repostiory. This needs to be improved for the final relese.

Cheers,
 Michael

[Wesley Moore (wjmoore)]

Michael,
No my sources.list hasn't changed since last time. Although the update manager said that it was rewriting the list this time, which it didn't do last time I ran it.

[Michael Vogt (mvo)]

#41090 seems to have the same problem. The only sources.list line that was detected was "breezy-security", the others are commented out.

[Manuel Aristarán (maristaran)]

Hi,

I'm getting the exact same error, and don't know how to fix it.

manuel@vicenta:/var/log$ cat dist-upgrade.log
2006-05-19 14:23:00,718 DEBUG Foreign:
2006-05-19 14:23:00,719 DEBUG Obsolete: tomboy libevms-2.5 ndiswrapper-utils ruby1.8 initng lsb-base ndiswrapper-modules-2.6.12-10-386 readline-common gaim-data totem libreadline5 libreadline4 fglrx-kernel-2.6.12-10-386 skype rhythmbox xorg-driver-fglrx libreadline-ruby1.8 postgresql-client-8.0 fglrx-kernel-source libpgsql-ruby1.8 evms-ncurses fglrx-control libssl0.9.8 libpq4 postgresql-common evms libtotem-plparser0 lighttpd ruby1.8-dev irb1.8 gaim libpq-dev libredcloth-ruby1.8 libruby1.8 serpentine totem-gstreamer eog postgresql-8.0 gthumb pmount
2006-05-19 14:23:00,720 DEBUG updateSourcesList()
2006-05-19 14:23:00,723 DEBUG rewriteSourcesList()
2006-05-19 14:23:00,725 DEBUG examining: 'deb http://archive.ubuntu.com/ubuntu breezy main restricted'
2006-05-19 14:23:00,725 DEBUG entry 'deb http://archive.ubuntu.com/ubuntu dapper main restricted' updated to new dist
2006-05-19 14:23:00,726 DEBUG examining: 'deb http://archive.ubuntu.com/ubuntu breezy-updates main restricted'
2006-05-19 14:23:00,726 DEBUG entry 'deb http://archive.ubuntu.com/ubuntu dapper-updates main restricted' updated to new dist2006-05-19 14:23:00,726 DEBUG examining: 'deb http://archive.ubuntu.com/ubuntu breezy universe multiverse'
2006-05-19 14:23:00,726 DEBUG entry 'deb http://archive.ubuntu.com/ubuntu dapper universe multiverse' updated to new dist
2006-05-19 14:23:00,726 DEBUG examining: 'deb http://security.ubuntu.com/ubuntu breezy-security main restricted'
2006-05-19 14:23:00,726 DEBUG entry 'deb http://security.ubuntu.com/ubuntu dapper-security main restricted' updated to new dist
2006-05-19 14:23:00,726 DEBUG examining: 'deb http://security.ubuntu.com/ubuntu breezy-security universe'
2006-05-19 14:23:00,727 DEBUG entry 'deb http://security.ubuntu.com/ubuntu dapper-security universe' updated to new dist
2006-05-19 14:26:03,152 DEBUG Marking 'ubuntu-desktop' for upgrade
2006-05-19 14:26:03,153 DEBUG Removing 'xchat' (ubuntu-desktop PostUpgradeRemove rule)
2006-05-19 14:26:03,240 DEBUG Removing 'xscreensaver' (ubuntu-desktop PostUpgradeRemove rule)
2006-05-19 14:26:03,241 DEBUG Purging 'xorg-common' (Distro PostUpgradePurge rule)
2006-05-19 14:26:03,529 DEBUG The package 'ubuntu-desktop' is marked for removal but it's in the removal blacklist
2006-05-19 14:26:14,518 ERROR Dist-upgrade failed: 'A essential package would have to be removed'
manuel@vicenta:/var/log$ cat dist-upgrade-apt.log
Starting
Starting 2
Done
MarkUpgrade() called on a non-upgrable pkg: 'ubuntu-desktop'
Starting
Starting 2
Investigating ubuntu-desktop
Package ubuntu-desktop has broken dep on xchat
  Considering xchat 10002 as a solution to ubuntu-desktop 0
  Removing ubuntu-desktop rather than change xchat
Done

[Hellkeepa (hellklone)]

HELLo!

I get the same error when trying to upgrade, plus a warning about "disabled sources". Reason I disabled them is because they only return 404, and the update wouldn't even get this far with them enabled.

After disabling the offending sources, and cleaning up to get enough free space; "Could not calculate size of update".
Attached the log of the update, and if you need any more files just let me know.

Happy codin'!

[Michael Vogt (mvo)]

The version of the dist-upgrader for dapper->edgy upgrades is much better in this regard. It will know about all registered mirrors and ensure that there is at least a working main sources.list entry and more sanity checking in general.

Cheers,
 Michael