mirrors only added to mirrors.cfg, never removed

[Seth Arnold]

/usr/share/ubuntu-release-upgrader/mirrors.cfg includes a mention of mirror.anl.gov; this service has been discontinued, see http://mirror.anl.gov/pub/ubuntu-iso/DVDs/ubuntu/ for details.

Thanks

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: ubuntu-release-upgrader-core 1:0.220.7
ProcVersionSignature: Ubuntu 3.13.0-46.79-generic 3.13.11-ckt15
Uname: Linux 3.13.0-46-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.11
Architecture: amd64
CrashDB: ubuntu
CurrentDesktop: Unity
Date: Tue Jun 9 23:22:17 2015
InstallationDate: Installed on 2012-10-18 (965 days ago)
InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1)
PackageArchitecture: all
SourcePackage: ubuntu-release-upgrader
UpgradeStatus: Upgraded to trusty on 2014-04-12 (424 days ago)
VarLogDistupgradeAptclonesystemstate.tar.gz: Error: command ['pkexec', 'cat', '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'] failed with exit code 126: Error executing command as another user: Request dismissed

[Brian Murray (brian-murray)]

Oh right, this is because update_mirrors.py only adds mirrors and doesn't remove them.

 13 d = feedparser.parse("https://launchpad.net/ubuntu/+archivemirrors-rss")
 14
 15 #import pprint
 16 #pp = pprint.PrettyPrinter(indent=4)
 17 #pp.pprint(d)
 18
 19 with open(sys.argv[1], "a") as outfile:
 20 for entry in d.entries:
 21 for link in entry.links:
 22 if link.href not in current_mirrors:
 23 outfile.write(link.href + "\n")

Having said that I'm not certain what ubuntu-release-upgrader would do if your mirror wasn't in that list.

[Brian Murray (brian-murray)]

I've marked an older bug as a duplicate of this one, just because.

[Brian Murray (brian-murray)]

Testing this on a Vivid system and setting the mirror to the anl.gov server caused the upgrade process to just quit. The release upgrader rewrote the entries from vivid to wily and then received 403s trying to contact the mirror, which then caused the failure.

, W:Failed to fetch http://mirror.mcs.anl.gov/pub/ubuntu/dists/wily-security/universe/binary-i386/Packages 403 Forbidden
, W:Failed to fetch http://mirror.mcs.anl.gov/pub/ubuntu/dists/wily-security/multiverse/binary-i386/Packages 403 Forbidden
, E:Some index files failed to download. They have been ignored, or old ones used instead.'. Retrying (currentRetry: 2)
2015-10-14 08:58:17,234 ERROR doUpdate() failed completely
2015-10-14 08:58:17,237 DEBUG abort called

There was no prompt to report a bug, so its hard to say how many people end up in this situation.

[Brian Murray (brian-murray)]

And then when I removed the mirror from mirrors.cfg my sources.list file was modified by the release upgrader to the following.

root@clean-vivid-amd64:/tmp/ubuntu-release-upgrader-7k_gvob6# cat /etc/apt/sources.list
# deb http://mirror.mcs.anl.gov/pub/ubuntu wily main restricted universe multiverse # disabled on upgrade to wily
# deb http://mirror.mcs.anl.gov/pub/ubuntu vivid-updates main restricted universe multiverse # disabled on upgrade to wily
# deb http://mirror.mcs.anl.gov/pub/ubuntu vivid-proposed main restricted universe multiverse
# deb http://mirror.mcs.anl.gov/pub/ubuntu vivid-security main restricted universe multiverse # disabled on upgrade to wily
# deb-src http://mirror.mcs.anl.gov/pub/ubuntu wily main restricted universe multiverse # disabled on upgrade to wily
# deb-src http://mirror.mcs.anl.gov/pub/ubuntu vivid-updates main restricted universe multiverse # disabled on upgrade to wily
# deb-src http://mirror.mcs.anl.gov/pub/ubuntu vivid-proposed main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu wily main
# deb-src http://mirror.mcs.anl.gov/pub/ubuntu vivid-security main restricted universe multiverse # disabled on upgrade to wily

[Brian Murray (brian-murray)]

I've added a branch that will also remove mirrors when updating the mirrors.cfg file, but that'll just put people in the situation where their sources.list files is modified to only only contain main which isn't good either.

[Brian Murray (brian-murray)]

I've reported the minimal sources.list issue in bug 1511783.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:16.04.3

---------------
ubuntu-release-upgrader (1:16.04.3) xenial; urgency=medium

  * DistUpgrade/DistUpgradeController.py: if a BaseMetaPkg has no origins,
    because an Ubuntu mirror has been deactivated, then create a useful
    sources.list file that parallels the one for the deactivated mirror.
    (LP: #1511783)
  * utils/update_mirrors.py: Instead of just appending new mirrors to the
    mirrors.cfg file also remove mirrors that are no longer valid.
    (LP: #1463680)

 -- Brian Murray <email address hidden> Wed, 18 Nov 2015 11:05:19 -0800

[Brian Murray (brian-murray)]

I've tested this fix by upgrading from wily to xenial with my mirror set to mirror.mcs.anl.gov. During the upgrade process my sources.list file was changed to the following:

# deb http://mirror.mcs.anl.gov/pub/ubuntu xenial main restricted universe multiverse # disabled on upgrade to xenial
# deb http://mirror.mcs.anl.gov/pub/ubuntu xenial-updates main restricted universe multiverse # disabled on upgrade to xenial
# deb http://mirror.mcs.anl.gov/pub/ubuntu wily-proposed main restricted universe multiverse
# deb http://mirror.mcs.anl.gov/pub/ubuntu xenial-security main restricted universe multiverse # disabled on upgrade to xenial
# deb-src http://mirror.mcs.anl.gov/pub/ubuntu xenial main restricted universe multiverse # disabled on upgrade to xenial
# deb-src http://mirror.mcs.anl.gov/pub/ubuntu xenial-updates main restricted universe multiverse # disabled on upgrade to xenial
# deb-src http://mirror.mcs.anl.gov/pub/ubuntu wily-proposed main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu xenial main
deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse # auto generated by ubuntu-release-upgrader
deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse # auto generated by ubuntu-release-upgrader
deb http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse # auto generated by ubuntu-release-upgrader
deb-src http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse # auto generated by ubuntu-release-upgrader
deb-src http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse # auto generated by ubuntu-release-upgrader
deb-src http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse # auto generated by ubuntu-release-upgrader
# deb-src http://mirror.mcs.anl.gov/pub/ubuntu xenial-security main restricted universe multiverse # disabled on upgrade to xenial

So this, and the fix for bug 1511783, looks good to me.