do-release-upgrade fails with TLS inspecting proxy (if CA is not installed system wide)

[DaJo]

Problem: do-release-upgrade fails with TLS inspecting proxy (if CA is not installed system wide)
Solution: patch provided below (at least for the detection)
          additional error reporting: see below for demonstration of added error messages.

Related bug that might be fixed by the attached patch:
https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/1821034

Affected package (Bionic and package main branch also does not fix it):
Package: python3-update-manager
Architecture: all
Version: 1:18.04.11.13
Priority: standard
Section: python
Source: update-manager
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug

Problem description details:
  Company environment with HTTP-proxy required to connect to the internet.
  The proxy inspects HTTPS traffic and changes HTTPS server's certificate.
  The proxy's CA is not installed/trusted system wide.
  APT is configured to use additional CA certificate file via "Acquire::https::CAInfo "/etc/ssl/company/proxyCA.pem" to trust the proxy when downloading updates via HTTPS.

  After I created the patch I learned about:
```
/usr/lib/apt/apt-helper auto-detect-proxy "https://www.ubuntu.com"
Using proxy '' for URL 'https://www.ubuntu.com/'
```
  but that command does not output proxy or CA information for me. This might be another bug?

  I've provided a patch that applies to python3-update-manager AND python3-distupgrade (with changed paths - not sure why there are redundant copies).
  And allows do-release-upgrade to detect / use the correct certificate while not breaking existing setups (as far as I can tell).

System details:
1) lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic
2) apt-cache policy python3-update-manager
 python3-update-manager:
  Installed: 1:20.04.10.1
  Candidate: 1:20.04.10.1
  Version table:
 *** 1:20.04.10.1 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:20.04.9 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
3) Expectation: do-release-upgrade works and picks up working apt configuration
4) Actual: do-release-upgrade reports no updates (stuck on Bionic)

Solution:
Note that an apt miss-configuration does only output an error/warning, but then the attempt to check for upgrades is continued (without TLS-config or proxy).
So the proxy and default trust store is used to access the https URL.

Additionally if that fails due to certificate mismatch that error is now reported.
As well as timeouts or BadStatusLine errors - just to understand the root cause of the problem.
```
$ do-release-upgrade
Error failed to read '/etc/ssl/company/company_proxy.pem2' from apt conf: [Errno 2] No such file or directory
Checking for a new Ubuntu release
Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings
        Reason: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852) There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.
```

Example /etc/apt/apt.conf.d/proxy.conf:
```
Acquire::http::Proxy "http://proxy.example.org:8080";
Acquire::https::Proxy "http://proxy.example.org:8080";
Acquire::https::CAInfo "/etc/ssl/company/proxyCA.pem";
```

The successful update with debug information now looks like this (while it failed before):
```
$ DEBUG_UPDATE_MANAGER="yes" do-release-upgrade
Checking for a new Ubuntu release
MetaRelease.__init__() useDevel=False useProposed=False
/etc/update-manager/meta-release: https://changelogs.ubuntu.com/meta-release
/etc/update-manager/meta-release: https://changelogs.ubuntu.com/meta-release-lts
/etc/update-manager/meta-release: -development
/etc/update-manager/meta-release: -proposed
metarelease-uri: https://changelogs.ubuntu.com/meta-release-lts
MetaRelease.download()
have self.metarelease_information
MetaRelease.parse()
current dist name: 'bionic'
found distro name: 'dapper'
found distro name: 'hardy'
found distro name: 'lucid'
found distro name: 'precise'
found distro name: 'trusty'
found distro name: 'xenial'
found distro name: 'bionic'
found distro name: 'focal'
new dist: <UpdateManager.Core.MetaRelease.Dist object at 0x7f5ac7022e80>
Please install all available updates for your release before upgrading.
```

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "fix-ubuntu_python3-update-manager_add-https-proxy-support.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Brian Murray (brian-murray)]

Thanks for working on this fix to update-manager. Could you please complete the Canonical contributor licence agreement so we can incorporate this change?

https://ubuntu.com/legal/contributors

[DaJo (xyxyxy)]

The legal clearance is in the works.
I expect this to be ready in the next weeks. Sorry for the delays.

[DaJo (xyxyxy)]

I've mailed the code contribution document.

[DaJo (xyxyxy)]

I've provided a signed Canonical contribution license agreement via email.

[DaJo (xyxyxy)]

Any update or priority when this patch will be included?