utils.py's url_downloadable doesn't support https

Bug #1823410 reported by Brian Murray on 2019-04-05
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-release-upgrader (Ubuntu)
High
Brian Murray
Bionic
High
Unassigned
Cosmic
High
Unassigned
update-manager (Ubuntu)
High
Brian Murray
Bionic
High
Brian Murray
Cosmic
High
Brian Murray

Bug Description

[Impact]
In the interest of improving release upgrades for users of PPAs a check was added to ubuntu-release-upgrader to see if the release to which the system is being upgraded is provided by a sources list entry and if it is not that entry is disabled. This check uses the function url_downloadable which is provided by update-manager's utils.py. Come to find out the url_downloadable check only supports ftp and http, not https. This can cause a situation where upgrades to the next release of Ubuntu fail if the system's mirror of the Ubuntu archive is using https. This is mentioned here - https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1807043/comments/10.

While the bug manifests itself in ubuntu-release-upgrader it needs fixing in update-manager and will require a rebuild of ubuntu-release-upgrader which has a symlink to utils.py which is shipped in the dist-upgrader tarball.

[Test Case]
update-manager - only ubuntu-release-upgrader uses url_downloadable but I added a test to update-manager to ensure that https works, the test may be skipped though if the https test site is not available so keep an eye out for that.

ubuntu-release-upgrader
1) On a xenial or bionic system edit your sources.list file so that you are using an https mirror of the archive.
2) try to upgrade to the next release of ubuntu using 'do-release-upgrade'

With the version of ubuntu-release-upgrader in the archive you will not be allowed to upgrade because ubuntu-minimal will not be found. With the version in -proposed, accessed by using 'do-release-upgrade -p' you should be able to upgrade to the next release of Ubuntu.

[Regression Potential]
There isn't much chance of a regression as we are just adding https to the list of schemes that we will test with urlopen.

Changed in update-manager (Ubuntu):
importance: Undecided → High
status: New → Triaged
Changed in update-manager (Ubuntu Bionic):
status: New → Triaged
Changed in update-manager (Ubuntu Cosmic):
status: New → Triaged
Changed in update-manager (Ubuntu Bionic):
importance: Undecided → High
Changed in update-manager (Ubuntu Cosmic):
importance: Undecided → High
Changed in update-manager (Ubuntu):
status: Triaged → In Progress
assignee: nobody → Brian Murray (brian-murray)
description: updated
Changed in ubuntu-release-upgrader (Ubuntu):
status: New → In Progress
assignee: nobody → Brian Murray (brian-murray)
importance: Undecided → High
Changed in ubuntu-release-upgrader (Ubuntu Bionic):
status: New → Triaged
Changed in ubuntu-release-upgrader (Ubuntu Cosmic):
status: New → Triaged
Changed in ubuntu-release-upgrader (Ubuntu Bionic):
importance: Undecided → High
Changed in ubuntu-release-upgrader (Ubuntu Cosmic):
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntu-release-upgrader - 1:19.04.15

---------------
ubuntu-release-upgrader (1:19.04.15) disco; urgency=medium

  * debian/control: build depend on the new version of update-manager which
    supports checking to see if an https url is downloadable. (LP: #1823410)
  * tests/test_sources.list.py: add a test for https support.
  * Update of demotions and mirrors data.
  * DistUpgrade/DistUpgradeQuirks.py: update the deb2snap quirk for disco,
    bump the version of gnome snap, and include the core18 snap.

 -- Brian Murray <email address hidden> Mon, 08 Apr 2019 14:19:29 -0700

Changed in ubuntu-release-upgrader (Ubuntu):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package update-manager - 1:19.04.5

---------------
update-manager (1:19.04.5) disco; urgency=medium

  * UpdateManager/Core/utils.py: when testing to see if a url is downloadable
    support https in addition to http and ftp. (LP: #1823410)
  * debian/tests/control: really use pyflakes3 instead of pyflakes,
    additionally resolve pyflakes3 errors.

 -- Brian Murray <email address hidden> Mon, 08 Apr 2019 10:00:34 -0700

Changed in update-manager (Ubuntu):
status: In Progress → Fix Released
description: updated
Changed in update-manager (Ubuntu Bionic):
status: Triaged → In Progress
Changed in update-manager (Ubuntu Cosmic):
status: Triaged → In Progress
Changed in update-manager (Ubuntu Bionic):
assignee: nobody → Brian Murray (brian-murray)
Changed in update-manager (Ubuntu Cosmic):
assignee: nobody → Brian Murray (brian-murray)
tags: added: id-5ca3d1e6bc6a797ad8a39c67

Hello Brian, or anyone else affected,

Accepted update-manager into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-manager/1:18.10.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in update-manager (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Łukasz Zemczak (sil2100) wrote :

Hello Brian, or anyone else affected,

Accepted update-manager into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-manager/1:18.04.11.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in update-manager (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Łukasz Zemczak (sil2100) wrote :

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.10.11.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ubuntu-release-upgrader (Ubuntu Cosmic):
status: Triaged → Fix Committed
Łukasz Zemczak (sil2100) wrote :

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.31 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ubuntu-release-upgrader (Ubuntu Bionic):
status: Triaged → Fix Committed
Brian Murray (brian-murray) wrote :

I tested a release upgrade from Bionic to Cosmic with https entries in my sources.list file. With the version of the release upgrader in the -updates pocket I was not allowed to upgrade however when I tried to upgrade using 'do-release-upgrade -p' the upgrade was successfully calculated. Here's an excerpt from /var/log/dist-upgrade/main.log:

2019-04-15 09:15:10,252 DEBUG examining: 'deb https://ubuntu.osuosl.org/ubuntu bionic main restricted universe multiverse'
2019-04-15 09:15:10,253 DEBUG found apt-cacher/apt-torrent style uri https://ubuntu.osuosl.org/ubuntu
2019-04-15 09:15:10,253 DEBUG verifySourcesListEntry: deb https://ubuntu.osuosl.org/ubuntu cosmic main restricted universe multiverse
2019-04-15 09:15:10,253 DEBUG url_downloadable: https://ubuntu.osuosl.org/ubuntu/dists/cosmic/Release
2019-04-15 09:15:10,253 DEBUG s='https' n='ubuntu.osuosl.org' p='/ubuntu/dists/cosmic/Release' q='' f=''
2019-04-15 09:15:10,623 DEBUG entry 'deb https://ubuntu.osuosl.org/ubuntu cosmic main restricted universe multiverse' updated to new dist

The "apt-cacher/apt-torrent" message is a bit strange but it worked.

Brian Murray (brian-murray) wrote :

I tested a release upgrade from Xenial to Bionic with https entries in my sources.list file. With the version of the release upgrader in the -updates pocket I was not allowed to upgrade however when I tried to upgrade using 'do-release-upgrade -p' the upgrade was successfully calculated. Here's an excerpt from /var/log/dist-upgrade/main.log:

2019-04-15 09:39:47,429 DEBUG examining: 'deb https://ubuntu.osuosl.org/ubuntu xenial main restricted universe multiverse'
2019-04-15 09:39:47,429 DEBUG found apt-cacher/apt-torrent style uri https://ubuntu.osuosl.org/ubuntu
2019-04-15 09:39:47,429 DEBUG verifySourcesListEntry: deb https://ubuntu.osuosl.org/ubuntu bionic main restricted universe multiverse
2019-04-15 09:39:47,429 DEBUG url_downloadable: https://ubuntu.osuosl.org/ubuntu/dists/bionic/Release
2019-04-15 09:39:47,429 DEBUG s='https' n='ubuntu.osuosl.org' p='/ubuntu/dists/bionic/Release' q='' f=''
2019-04-15 09:39:47,613 DEBUG entry 'deb https://ubuntu.osuosl.org/ubuntu bionic main restricted universe multiverse' updated to new dist

Setting to verification-done.

tags: added: verification-done-bionic verification-done-cosmic
removed: verification-needed-bionic verification-needed-cosmic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntu-release-upgrader - 1:18.10.11.6

---------------
ubuntu-release-upgrader (1:18.10.11.6) cosmic; urgency=medium

  * DistUpgradeController.py: Instead of checking every sources.list entry to
    see if they are accessible keep track of the uris and only test once per
    uri thereby preventing situations where entries for the same server are
    enabled and disabled. (LP: #1822886)
  * debian/control: build depend on the new version of update-manager which
    supports checking to see if an https url is downloadable. (LP: #1823410)
  * tests/test_sources.list.py: add a test for https support.

 -- Brian Murray <email address hidden> Tue, 09 Apr 2019 17:10:21 -0700

Changed in ubuntu-release-upgrader (Ubuntu Cosmic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package update-manager - 1:18.10.10.2

---------------
update-manager (1:18.10.10.2) cosmic; urgency=medium

  * UpdateManager/Core/utils.py: when testing to see if a url is downloadable
    support https in addition to http and ftp. (LP: #1823410)

 -- Brian Murray <email address hidden> Tue, 09 Apr 2019 16:19:41 -0700

Changed in update-manager (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package update-manager - 1:18.04.11.10

---------------
update-manager (1:18.04.11.10) bionic; urgency=medium

  * UpdateManager/Core/utils.py: when testing to see if a url is downloadable
    support https in addition to http and ftp. (LP: #1823410)

 -- Brian Murray <email address hidden> Tue, 09 Apr 2019 16:03:46 -0700

Changed in update-manager (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.31

---------------
ubuntu-release-upgrader (1:18.04.31) bionic; urgency=medium

  * DistUpgradeController.py: Instead of checking every sources.list entry to
    see if they are accessible keep track of the uris and only test once per
    uri thereby preventing situations where entries for the same server are
    enabled and disabled. (LP: #1822886)
  * debian/control: build depend on the new version of update-manager which
    supports checking to see if an https url is downloadable. (LP: #1823410)
  * tests/test_sources.list.py: add a test for https support.

 -- Brian Murray <email address hidden> Tue, 09 Apr 2019 18:05:01 -0700

Changed in ubuntu-release-upgrader (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers