update-manager rewrites sources.list to only use official mirrors

[Ted Cabeen]

Binary package hint: update-manager

When upgrading from hardy to intrepid, update-manager ignores the local sources.list and downloads everything needed from archive.ubuntu.org. This is problematic for multiple reasons.

1. Some users may have a local or regional mirror that provides faster downloads than archive. Ignoring sources.list prevents them from using those local mirrors, possibly causing substantial bandwidth costs.2. During the initial release window for intrepid thousands of Ubuntu users are going to upgrade. Will archive be able to handle the demand?
3. When upgrading multiple machines in a laboratory or corporate environment, each machine will download all the packages they need from archive, rather than allowing a single download via the use of apt-proxy or other related proxy systems.

Please re-consider this policy decision.

WORKAROUND:

It is possible to disable this check - in DistUpgradeController we see:

        # skip mirror check if special environment is set
        # (useful for server admins with internal repos)
        if (self.config.getWithDefault("Sources","AllowThirdParty",False) or
            "RELEASE_UPRADER_ALLOW_THIRD_PARTY" in os.environ):
            logging.warning("mirror check skipped, *overriden* via config")
            mirror_check=False

[Brian Murray (brian-murray)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Could you please provide some information about what source was overwritten in your 'sources.list' file? (Ideally, the whole soruces.list file that you were using would be best.) This doesn't normally happen. Thanks in advance.

[Ted Cabeen (ted-cabeen)]

It wasn't that any data was overwritten, it's that the file was completely ignored. Although I have a local mirror and I usually download my updates from mirrors.kernel.org, the intrepid updater didn't recognize either of them and downloaded all of the updates I needed from archive.ubuntu.com. After the upgrade was complete, my sources.list file had all of my lines commented out, and the only active entry was the archive.ubuntu.com that the updater used.

According to a chat I had with nemo in #ubuntu+1, that's intended behavior. I think that this configuration is ill-advised with thousands of people expecting to upgrade to intrepid in the coming days. Here's the chat log from http://irclogs.ubuntu.com/2008/10/27/%23ubuntu+1.txt
[17:11] <TedC> The intrepid upgrader seems to ignore my sources.list and tries to download all of the packages it needs from archive.ubuntu.com. Is that how it's supposed to work?
[17:11] <nemo> TedC: updater does override your source list
[17:11] <nemo> TedC: you can reenable it after

[Brian Murray (brian-murray)]

Update-manager checks to see whether not the repositories you are using are official Ubuntu mirrors, as listed at https://launchpad.net/ubuntu/+archivemirrors, and disables any that are not. As mirrors come and go this functionality allows users to upgrade from release to release without having to modify their sources.list.

[Ted Cabeen (ted-cabeen)]

That's frustrating for those of us who want to use unofficial mirrors to upgrade. I can see disabling mirrors that aren't up to date or that don't have the files necessary to upgrade to intrepid, but a lot of people use "unofficial" yet well-known and up-to-date mirrors (mirrors.kernel.org, ftp.usf.edu, mirror.anl.gov, etc.) to get their ubuntu packages. I think unilaterally disabling these mirrors from their their sources.list is pretty aggressive and will cause some pretty serious bandwidth crunches on the official mirrors when 8.10 is released. It's also confusing, as there is no notification that this is going to happen, nor is there any option to use local or non-official mirrors.

Finally, there aren't any official US mirrors. The official US mirror points at archive.ubuntu.org, which is located in London. If you are going to do something like this, shouldn't we have an official mirror in the country with the largest number of Ubuntu users?

[Mark Schouten (mark-prevented)]

I've got two issues with this bug:
* Why choose archive.ubuntu.com and not CC.archive.ubuntu.com? You know where the system lives (it a question in the installer) so choose the nearest (official) mirror.
* Why change this silently? That's a bit rude. If someone was smart enough to change the sources.list, they will also understand it if you tell them that they might have broken it.

[Ted Cabeen (ted-cabeen)]

I just re-read your comment, and I realized I was missing something.

I just tested a hardy->intrepid upgrade with mirrors.kernel.org as the package source in sources.list and it worked properly. It still doesn't work with local mirrors, but that's not as bad as not supporting the official list. I'll close this bug.

[Ted Cabeen (ted-cabeen)]

Changing to Invalid, as it works properly with US mirrors.

[Ted Cabeen (ted-cabeen)]

Setting back to incomplete for Mark's questions.

[Jean-Baptiste Lallement (jibel)]

This is also an issue for non-networked machines that points only to an internal mirror. Thanks for taking the time to make Ubuntu better!

[David Monro (davidm-ub)]

Please provide an option to use a custom sources.list; I have a bandwidth cap, but my ISP maintains a mirror for its customers which doesn't count towards the cap. I'm sure I'm not the only person in that situation!

[Davertron (david-davis)]

Please forgive me if this doesn't seem to be exactly on topic, but I was re-directed here after posting a separate bug. My problem isn't so much that I wasn't allowed to use my own sources.list; while I would like to do that, it's not the end of the world if my machine had to go out and pull down the updates from archive.ubuntu.com. My problem was that I had a local mirror that didn't have the Intrepid packages, and my machine tried to use the mirror to do the dist-upgrade anyway, and then, after it obviously didn't find any updated packages, told me that I needed to remove all of my installed packages because they were obsolete. While I realize it's my own fault for not verifying that the Intrepid files existed on my local mirror, it seems it shouldn't be too complicated for Ubuntu to make sure the files exist for the upgrade its attempting before it tries to upgrade.

[Nathaniel W. Turner (nturner)]

Bug 773823 was marked as a duplicate of this bug. I am not sure the use cases are really the same. For one thing, in the former case, the symptom is that the upgrade *fails*. In the latter case (this bug), the reported symptom is that the upgrade succeeds, but forces one to use official (external) mirrors. (Also, 773823 deals with upgrades *from* lucid, while this deals with upgrades *to* lucid. I suspect the root cause is different between these cases.)

[Nathaniel W. Turner (nturner)]

Ok, the upgrade does appear to get off the ground (it's still running as I write this) if I create a /etc/update-manager/release-upgrades.d/DistUpgrade.cfg containing the following 2 lines:

[Sources]
AllowThirdParty = true

[Nathaniel W. Turner (nturner)]

Apologies for the spam. Comment 14 was meant for bug 992657.

[Nathaniel W. Turner (nturner)]

In comment #13 I said that creating /etc/update-manager/release-upgrades.d/DistUpgrade.cfg allowed me to work around the problem. It looks like this approach has at least one major problem:

On a system that has a variety of PPAs configured, if I do the aforementioned "AllowThirdParty = true" hack, these PPA sources all get rewritten to use precise instead of lucid. If any PPA no longer exists in Precise, the upgrade fails.

The old behavior was smart enough to rewrite just the sources for the local mirror, and disable the rest.