changelogs.ubuntu.com should be using HTTPS

[TJ]

[Impact]
Although the packages listed in meta-release files on changelogs.ubuntu.com are signature-checked there doesn't appear to be any way to verify the meta-release files are valid so a man-in-the-middle could maliciously supply an alternate meta-release.

meta-release files should be signed with the archive GPG key and/or delivered over HTTPS.

[Test case]
Block port 80 access to changelogs.ubuntu.com and check that do-release-upgrade still works

[Regression potential]
This breaks any clients behind a proxy where HTTPS (CONNECT on the proxy) is not allowed.

[Dimitri John Ledkov (xnox)]

1) probably needs an RT to https-ify the subdomain
2) probably needs an RT/trello planning for inline gpg signed meta-releases
3) probably needs a trello planning for code changes to use the inline gpg signed meta-releases and/or https

[Julian Andres Klode (juliank)]

I think we'll turn on https for now, and defer GPG to a later time. There are essentially two ways we could go for that:

(1) implement GPG verification in UpdateManager. gpg is hard to use, so I'd expect us to mess up somewhere. Also should have rollback and starving prevention (date/valid-until).

(2) generate an InRelease file for the meta-release files, and re-use APT for the fetching and validation. This means we get security features automagically.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-manager - 1:18.04.6

---------------
update-manager (1:18.04.6) bionic; urgency=medium

  * Use HTTPS for changelogs.ubuntu.com (LP: #1744318)

 -- Julian Andres Klode <email address hidden> Thu, 15 Mar 2018 14:19:24 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.12

---------------
ubuntu-release-upgrader (1:18.04.12) bionic; urgency=medium

  [ Simon Quigley ]
  * Port away from kdesudo.

  [ Brian Murray ]
  * Increase the size of the buffer used when calculating the free space to
    estimate for the initramfs.

 -- Brian Murray <email address hidden> Wed, 21 Mar 2018 16:32:15 -0700

[Julian Andres Klode (juliank)]

I think I vaguely recall some issues that occured after this SRU in bionic, but I'm not sure anymore. It certainly means that tools stop working for people behind proxies in quite a few cases (e.g. various apt proxies not allowing https connect; or access to changelogs.ubuntu.com).

So we need to consider whether the benefits of backporting this to xenial outweight the risks.

[Julian Andres Klode (juliank)]

I think the issue I remembered was bug 1771914

[Brian Murray (brian-murray)]

Hello TJ, or anyone else affected,

Accepted update-manager into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-manager/1:16.04.16 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello TJ, or anyone else affected,

Accepted ubuntu-release-upgrader into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:16.04.27 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-release-upgrader (1:16.04.27) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-release-upgrader/1:16.04.27 (armhf)
update-manager/1:16.04.15 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#ubuntu-release-upgrader

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Julian Andres Klode (juliank)]

FTR: All autopkgtests have passed now.

[Julian Andres Klode (juliank)]

Blocking access:

ufw deny out to 2001:67c:1560:8008::11 port 80
ufw deny out to 91.189.95.15 port 80

[Julian Andres Klode (juliank)]

old version tries to connect to port 80, hangs because I blocked it.

[pid 1035] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("91.189.95.15")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 1035] poll([{fd=3, events=POLLOUT|POLLERR}], 1, 20000

afterwards it uses HTTPS:

# strace -e connect -f do-release-upgrade
[...]
[pid 8264] connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("91.189.95.15")}, 16) = 0
[pid 8264] connect(3, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "2001:67c:1560:8008::11", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
[pid 8264] connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("91.189.95.15")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 8264] +++ exited with 0 +++
strace: Process 8265 attached
[pid 8265] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=8265, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
strace: Process 8266 attached
[pid 8266] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=8266, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
Please install all available updates for your release before upgrading.
+++ exited with 1 +++

I can't get this container to actually show me there's an update availagble, but it works fine in another, maybe I broke some cache file?

[Julian Andres Klode (juliank)]

Oh silly me, I had https proxy set to non-existing hostname, this works fine after removing it!

[Julian Andres Klode (juliank)]

FWIW, This was for the correct versions.

# dpkg -l ubuntu-release-upgrader-core update-manager-core
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================================================================-=====================================-=====================================-=======================================================================================================================================
ii ubuntu-release-upgrader-core 1:16.04.27 all manage release upgrades
ii update-manager-core 1:16.04.16 all manage release upgrades

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for update-manager has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:16.04.27

---------------
ubuntu-release-upgrader (1:16.04.27) xenial; urgency=medium

  * Use HTTPS for changelogs.ubuntu.com (LP: #1744318)
  * Run pre-build script to update mirror list, hold apt-btrfs-snapshot changes

 -- Julian Andres Klode <email address hidden> Wed, 02 Oct 2019 16:08:25 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-manager - 1:16.04.16

---------------
update-manager (1:16.04.16) xenial; urgency=medium

  * Use HTTPS for changelogs.ubuntu.com (LP: #1744318)
  * Add support for HTTPS proxies; this breaks UpdateManager.Core.utils.init_proxy()
    API - the return value is now a dict, rather than a string (LP: #1771914).

 -- Julian Andres Klode <email address hidden> Tue, 20 Aug 2019 13:59:31 +0200

[Sebastien Bacher (seb128)]

The change seem to be buggy, at least it created bug #1849004 which changelog fetchning stopped working in case without proxy in use