when rewriting sources.list ubuntu-release-upgrader doesn't check to see if new release is available

[Brian Murray]

[Impact]
When rewriting sources.list entries the release-upgrader doesn't check to see if the distribution to which the system is being upgraded is supported by the archive / PPA. This results in the following error message:

A problem occurred during the update. This is usually some sort of
network problem, please check your network connection and retry.

E:The repository 'http://ppa.launchpad.net/team-xbmc/ppa/ubuntu
cosmic Release' does not have a Release file.

If the release-upgrader were to actually check if the release were supported then that specific sources entry could be disabled and the upgrade may be able to proceed.

[Test Case]
The test case involves adding a PPA which doesn't have a release file for the release to which you are upgrading for upgrading to cosmic we can use the team-xbmc ppa.

1) sudo add-apt-repository ppa:team-xbmc/ppa
2) do-release-upgrade --frontend DistUpgradeViewText
3) Observe the error message re "does not have a Release file" and the upgrade cancel.

With the version of the release upgrader from -proposed you will instead see the following message: "Some third party entries in your sources.list were disabled.". Then the release upgrade will proceed to the calculation phase, you can quit the upgrade after the calculation and check /var/log/dist-upgrade/main.log for the following:

DEBUG entry '# deb http://ppa.launchpad.net/team-xbmc/ppa/ubuntu bionic main' was disabled (no Release file)

[Regression Potential]
It seems like an oversight that this part of the release-upgrader never did check for Release files and that this is the right thing to do. Its possible that people who were not allowed to upgrade, because they had PPAs without support for the release to which they wanted to upgrade, are now allowed to upgrade and somehow run in to an upgrade failure e.g. a failure to calculate the upgrade. That seems better and more informative than not allowing them to upgrade at all though.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:19.04.7

---------------
ubuntu-release-upgrader (1:19.04.7) disco; urgency=medium

  [ Brian Murray ]
  * DistUpgrade/DistUpgradeCache.py: When calculating free space needed for
    mount points don't use a negative number as the buffer. (LP: #1786484)
  * fix more typos re RELEASE_UPRADER_ALLOW_THIRD_PARTY
  * DistUpgrade/DistUpgradeCache.py: When calculating free space needed for
    mount points don't use a negative number as the buffer. (LP: #1786484)
  * fix more typos re RELEASE_UPRADER_ALLOW_THIRD_PARTY
  * DistUpgrade/DistUpgradeController.py: When rewriting sources.list entries
    check to see if the source provides packages for the release to which the
    upgrade is occurring. If it doesn't the entry is disabled thereby
    improving upgrades with PPAs. (LP: #1807043)
  * do-release-upgrade: add a parameter to allow third party mirrors and
    repositories, additionally pass along the environmental variable
    RELEASE_UPGRADER_ALLOW_THIRD_PARTY via pkexec and sudo. (LP: #1807032)
  * DistUpgrade/DistUpgradeCache.py: improve messaging regarding failures to
    calculate the upgrade and suggest using ppa-purge if PPAs are an issue.
  * DistUpgrade/DistUpgradeCache.py: in the event there is a failure to
    calculate the upgrade provide information about the log files in
    /var/log/dist-upgrade. (LP: #1771387)

  [ Jeremy Bicha ]
  * autopkgtest: Update dependency from gir1.2-webkit-3.0 to gir1.2-webkit2-4.0

 -- Brian Murray <email address hidden> Tue, 11 Dec 2018 15:35:34 -0800

[Łukasz Zemczak (sil2100)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.10.11.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

I tested this using the release-upgrader version '18.10.11.3' from -proposed and saw the following error message:

"Third party sources disabled

Some third party entries in your sources.list were disabled. You can
re-enable them after the upgrade with the 'software-properties' tool
or your package manager."

Setting this to verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.10.11.3

---------------
ubuntu-release-upgrader (1:18.10.11.3) cosmic; urgency=medium

  * DistUpgrade/DistUpgradeCache.py: When calculating free space needed for
    mount points don't use a negative number as the buffer. (LP: #1786484)
  * fix more typos re RELEASE_UPRADER_ALLOW_THIRD_PARTY.
  * DistUpgrade/DistUpgradeController.py: When rewriting sources.list entries
    check to see if the source provides packages for the release to which the
    upgrade is occurring. If it doesn't the entry is disabled thereby
    improving upgrades with PPAs. (LP: #1807043)
  * do-release-upgrade: add a parameter to allow third party mirrors and
    repositories, additionally pass along the environmental variable
    RELEASE_UPGRADER_ALLOW_THIRD_PARTY via pkexec and sudo. (LP: #1807032)
  * DistUpgrade/DistUpgradeCache.py: improve messaging regarding failures to
    calculate the upgrade and suggest using ppa-purge if PPAs are an issue.
  * DistUpgrade/DistUpgradeCache.py: in the event there is a failure to
    calculate the upgrade provide information about the log files in
    /var/log/dist-upgrade. (LP: #1771387)
  * debian/source_ubuntu-release-upgrader.py: Include xorg_fixup.log if it
    exists.
  * DistUpgrade/xorg_fix_proprietary.py: modify how the system is checked to
    see if nvidia is being used, drop fglrx check since it has been
    deprecated. (LP: #1773637)

 -- Brian Murray <email address hidden> Thu, 20 Dec 2018 12:31:37 -0800

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.30 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

After adding the PPA from the test case and using the ubuntu-release-upgrader packages from -proposed I received the following message when the upgrade was being calculated:

"Some third party entries in your sources.list were disabled. You can
re-enable them after the upgrade with the 'software-properties' tool
or your package manager."

bdmurray@clean-bionic-amd64:~$ apt-cache policy ubuntu-release-upgrader-core
ubuntu-release-upgrader-core:
  Installed: 1:18.04.30
  Candidate: 1:18.04.30

[Brian Murray (brian-murray)]

Actually the thing I wanted to test was an upgrade from Xenial to Bionic with the dist-upgrader from -proposed. I've done that now and we can see the following messages in /var/log/dist-upgrade/main.log:

2019-01-23 13:53:38,179 DEBUG entry 'deb-src http://ubuntu.osuosl.org/ubuntu bionic-security main restricted universe multiverse' updated to new dist
2019-01-23 13:53:38,180 DEBUG examining: 'deb http://ppa.launchpad.net/bryce/builds/ubuntu xenial main'
2019-01-23 13:53:38,180 DEBUG verifySourcesListEntry: deb http://ppa.launchpad.net/bryce/builds/ubuntu bionic main
2019-01-23 13:53:38,181 DEBUG url_downloadable: http://ppa.launchpad.net/bryce/builds/ubuntu/dists/bionic/Release
2019-01-23 13:53:38,181 DEBUG s='http' n='ppa.launchpad.net' p='/bryce/builds/ubuntu/dists/bionic/Release' q='' f=''
2019-01-23 13:53:38,489 DEBUG error from httplib: 'HTTP Error 404: Not Found'
2019-01-23 13:53:38,490 DEBUG entry '# deb http://ppa.launchpad.net/bryce/builds/ubuntu xenial main' was disabled (no Release file)

I had to use a different PPA since xbmc one exists for Bionic and Xenial.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.30

---------------
ubuntu-release-upgrader (1:18.04.30) bionic; urgency=medium

  * data/release-upgrades: Clarify documentation regarding the behavior for
    different Prompt settings. (LP: #1795024)
  * do-release-upgrade: Utilize information regarding what Prompt is set to so
    that a more informative error message can be displayed.
    (LP: #1798618, LP: #1795024)
  * DistUpgrade/DistUpgradeCache.py: When calculating free space needed for
    mount points don't use a negative number as the buffer. (LP: #1786484)
  * fix more typos re RELEASE_UPRADER_ALLOW_THIRD_PARTY
  * DistUpgrade/DistUpgradeController.py: When rewriting sources.list entries
    check to see if the source provides packages for the release to which the
    upgrade is occurring. If it doesn't the entry is disabled thereby
    improving upgrades with PPAs. (LP: #1807043)
  * do-release-upgrade: add a parameter to allow third party mirrors and
    repositories, additionally pass along the environmental variable
    RELEASE_UPGRADER_ALLOW_THIRD_PARTY via pkexec and sudo. (LP: #1807032)
  * DistUpgrade/DistUpgradeCache.py: improve messaging regarding failures to
    calculate the upgrade and suggest using ppa-purge if PPAs are an issue.
  * DistUpgrade/DistUpgradeCache.py: in the event there is a failure to
    calculate the upgrade provide information about the log files in
    /var/log/dist-upgrade. (LP: #1771387)
  * debian/source_ubuntu-release-upgrader.py: Include xorg_fixup.log if it
    exists.
  * DistUpgrade/xorg_fix_proprietary.py: modify how the system is checked to
    see if nvidia is being used, drop fglrx check since it has been
    deprecated. (LP: #1773637)

 -- Brian Murray <email address hidden> Wed, 16 Jan 2019 15:53:58 -0800

[Stephane Chazelas (stephane-chazelas)]

That fix breaks the upgrade (in my case from xenial to bionic) for users that use a local mirror over HTTPS as the _sourcesListEntryDownloadable->url_downloadable doesn't support HTTPS.

It comments out the entries and adds:

      2019-02-19 15:13:51,808 DEBUG entry '# deb https://localmirror.example.com/example/security.ubuntu.com/ubuntu xenial-security multiverse' was disabled (no Release file)

The "Release file" is there but "url_downloadable" failed to download it because of the "https" scheme (note: the apt-transport-https package is installed).

[Brian Murray (brian-murray)]

I reported bug 1823410 regarding the lack of https support.