fetch-url does not use --no-check-certificate on HTTP to HTTPS redirects

[Mauricio Faria de Oliveira]

[Impact]

 * fetch-url fails to download files from URL with HTTP to HTTPS
   redirect if server has invalid/cannot be verified certificate.

 * Install fails in case a preseed/other files use an HTTP URL
   that redirects to an HTTPS URL with an invalid certificate.

 * Servers/URLs that started using HTTP to HTTPS redirect and
   have their URLs already spread over scripts/infrastructure
   start to cause install/deployment failures.

 * This fix checks for debian-installer/allow_unauthenticated_ssl
   in the HTTP protocol as well (to enable --no-check-certificate),
   which is OK as that option must be explicitly enabled by users,
   indicating awareness of the SSL/HTTPS context and certificates
   that may not be verified.

[Test Case]

 * Setup web-server with HTTP to HTTPS redirect and an invalid/
   self-signed certificate, and put a file (eg, preseed) on it.

 * Boot with preseed option 'url=http://<server>/preseed' and
   the install will fail in the 'network-preseed' stage, with
   syslog errors about invalid/cannot be verified certificates,
   suggesting the 'wget --no-check-certificate' option.

 * Other files downloaded by the installer can hit same error,
   if using HTTP URLs from that server.

 * In the installer shell, run:
   ~ # fetch-url http://<server>/<file>

[Regression Potential]

 * Low risk of regression, this only expands the check from HTTPS-only
   to HTTPS or HTTP, to *then* check for d-i/allow_unauthenticated_ssl.

 * The theoretical case is that a HTTP URL with no redirect to HTTPS
   may use --no-check-certificate, thus without actually needing it,
   (it should not cause problems at all, the option should be ignored)
   but anyway, since the user acknowledged that sort of behavior with
   the d-i/allow_unauthenticated_ssl, that should not be a concern.

[Other Info]

 * Debian Bug #913740.

[Problem Description]

In fetch-url the --no-check-certificate option is conditioned to HTTPS.
In case of HTTP to HTTPS redirect, that option is not enabled, and may
cause fetch-url to fail if the certificate cannot be verified.

Since that option/functionality must be explicitly requested with the
debian-installer/allow_unauthenticated_ssl preseed option (i.e., user
is aware of SSL/HTTPS context and agrees w/ non-verified certificates)
we can just check for this in the HTTP protocol too, and assume HTTPS
may potentially be used, as the user specified this option.

An alternative/obvious solution in the _user_ side is to specify HTTPS
URLs upfront, but there are cases when an user does not know for sure
whether the server uses/supports that, or the server might change its
behavior and start HTTP to HTTPS redirect after URLs have spread over
(e.g., scripts and infrastructure) - thus a fix in the installer side
is a simpler and more complete approach.

[Mauricio Faria de Oliveira (mfo)]

Testing performed with Disco, Cosmic, Bionic, Xenial, and Trusty.

The output is similar if not identical, so pasting just one test, from Disco.

(Web Server, HTTP/HTTPS redirect, setup not included, ping me if interested.)

$ wget http://archive.ubuntu.com/ubuntu/dists/disco/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/{linux,initrd.gz}

$ GUEST=disco

$ virt-install \
  --name $GUEST \
  --vcpus 2 \
  --memory 1024 \
  --disk $GUEST.qcow2,bus=virtio,format=qcow2,size=8 \
  --network bridge=virbr0,model=virtio \
  --graphics none \
  --import \
  --boot \
kernel=linux,\
initrd=initrd.gz,\
kernel_args='console=ttyS0 url=http://192.168.122.1/preseed debian-installer/allow_unauthenticated_ssl=true'

The installer hits an error when trying to get the preseed file:

     ┌──────────┤ [!!] Download debconf preconfiguration file ├──────────┐
     │ │
     │ Failed to retrieve the preconfiguration file │
     │ The file needed for preconfiguration could not be retrieved from │
     │ http://192.168.122.1/preseed. The installation will proceed in │
     │ non-automated mode. │
     │ │
     │ <Continue> │
     │ │
     └───────────────────────────────────────────────────────────────────┘

The synthetic tests with fetch-url:
===

~ # cat /proc/cmdline
console=ttyS0 url=http://192.168.122.1/preseed debian-installer/allow_unauthenticated_ssl=true

~ # cat /etc/default-release
disco

Without patch:
---

~ # fetch-url http://192.168.122.1/preseed preseed
ERROR: cannot verify 192.168.122.1's certificate, ...
...
To connect to 192.168.122.1 insecurely, use `--no-check-certificate'.

~ # echo $?
1

With patch:
---

~ # wget --no-check-certificate http://192.168.122.1/di-utils_1.124ubuntu2_amd64.udeb
~ # udpkg -i di-utils_1.124ubuntu2_amd64.udeb

~ # fetch-url http://192.168.122.1/preseed preseed
WARNING: cannot verify 192.168.122.1's certificate, ...
...
2018-11-14 13:17:03 URL:https://192.168.122.1//preseed [11/11] -> "./_fetch-url_preseed.1467" [1]

~ # echo $?
0

With patch and Without d-i/allow_unauthenticated_ssl=true: No Change
---

~ # cat /proc/cmdline
console=ttyS0 url=http://192.168.122.1/preseed

~ # wget --no-check-certificate http://192.168.122.1/di-utils_1.124ubuntu2_amd64.udeb
~ # udpkg -i di-utils_1.124ubuntu2_amd64.udeb

~ # fetch-url http://192.168.122.1/preseed preseed
ERROR: cannot verify 192.168.122.1's certificate, ...
...
To connect to 192.168.122.1 insecurely, use `--no-check-certificate'.

~ # echo $?
1

[Mauricio Faria de Oliveira (mfo)]

Oops, I'll add the Debina/Ubuntu-Bug: DEP-3 tags in the debdiffs.

[Eric Desrochers (slashd)]

Sponsor note:

* Substitute (LP: #NNNNNNN) by (LP: #1803385)
* Re-visit version for cosmic and bionic
  - In this case, if Disco is 1.124ubuntu2 | cosmic need to be 1.124ubuntu1.18.10.1 and bionic 1.124ubuntu1.18.04.1. Please double-check with dpkg --compare-version to make sure I didn't do typos.

- Eric

[Mauricio Faria de Oliveira (mfo)]

Er, I mean, the right LP bug numbers in changelog,
and fix the version suffixes for Bionic and Cosmic,
as reported by slashd in IRC (thanks!)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "disco_di-utils_httpsredir.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Mauricio Faria de Oliveira (mfo)]

Eric,

Just updated the attached debdiffs w/ the fixes for the points listed.

Thanks for reviewing; nice catch in the Cosmic/Bionic version suffixes
(I have verified them with dpkg --compare-versions, all good this time).

cheers,
Mauricio

[Mauricio Faria de Oliveira (mfo)]

Closing this bug as Invalid.
The real solution is fix-released in LP#1807023.

This bug was a workaround for not having ca-certificates in d-i and use an HTTP mirror that redirected to HTTPS
(the resulting certificate validation error couldn't be ignored due to HTTP protocol not using the wget option.)

But this is no longer required with the ca-certificates shipped in debian-installer.

Sorry, I had lost track of this bug.
Mauricio

[Dan Streetman (ddstreet)]

this was fixed a different way, in d-i and ca-certificates.

bug 1807023