fetch-url does not use --no-check-certificate on HTTP to HTTPS redirects

Bug #1803385 reported by Mauricio Faria de Oliveira on 2018-11-14
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
debian-installer-utils (Debian)
New
Unknown
debian-installer-utils (Ubuntu)
Medium
Mauricio Faria de Oliveira
Trusty
Medium
Mauricio Faria de Oliveira
Xenial
Medium
Mauricio Faria de Oliveira
Bionic
Medium
Mauricio Faria de Oliveira
Cosmic
Medium
Mauricio Faria de Oliveira

Bug Description

[Impact]

 * fetch-url fails to download files from URL with HTTP to HTTPS
   redirect if server has invalid/cannot be verified certificate.

 * Install fails in case a preseed/other files use an HTTP URL
   that redirects to an HTTPS URL with an invalid certificate.

 * Servers/URLs that started using HTTP to HTTPS redirect and
   have their URLs already spread over scripts/infrastructure
   start to cause install/deployment failures.

 * This fix checks for debian-installer/allow_unauthenticated_ssl
   in the HTTP protocol as well (to enable --no-check-certificate),
   which is OK as that option must be explicitly enabled by users,
   indicating awareness of the SSL/HTTPS context and certificates
   that may not be verified.

[Test Case]

 * Setup web-server with HTTP to HTTPS redirect and an invalid/
   self-signed certificate, and put a file (eg, preseed) on it.

 * Boot with preseed option 'url=http://<server>/preseed' and
   the install will fail in the 'network-preseed' stage, with
   syslog errors about invalid/cannot be verified certificates,
   suggesting the 'wget --no-check-certificate' option.

 * Other files downloaded by the installer can hit same error,
   if using HTTP URLs from that server.

 * In the installer shell, run:
   ~ # fetch-url http://<server>/<file>

[Regression Potential]

 * Low risk of regression, this only expands the check from HTTPS-only
   to HTTPS or HTTP, to *then* check for d-i/allow_unauthenticated_ssl.

 * The theoretical case is that a HTTP URL with no redirect to HTTPS
   may use --no-check-certificate, thus without actually needing it,
   (it should not cause problems at all, the option should be ignored)
   but anyway, since the user acknowledged that sort of behavior with
   the d-i/allow_unauthenticated_ssl, that should not be a concern.

[Other Info]

 * Debian Bug #913740.

[Problem Description]

In fetch-url the --no-check-certificate option is conditioned to HTTPS.
In case of HTTP to HTTPS redirect, that option is not enabled, and may
cause fetch-url to fail if the certificate cannot be verified.

Since that option/functionality must be explicitly requested with the
debian-installer/allow_unauthenticated_ssl preseed option (i.e., user
is aware of SSL/HTTPS context and agrees w/ non-verified certificates)
we can just check for this in the HTTP protocol too, and assume HTTPS
may potentially be used, as the user specified this option.

An alternative/obvious solution in the _user_ side is to specify HTTPS
URLs upfront, but there are cases when an user does not know for sure
whether the server uses/supports that, or the server might change its
behavior and start HTTP to HTTPS redirect after URLs have spread over
(e.g., scripts and infrastructure) - thus a fix in the installer side
is a simpler and more complete approach.

Changed in debian-installer-utils (Ubuntu):
status: New → Confirmed
assignee: nobody → Mauricio Faria de Oliveira (mfo)

Testing performed with Disco, Cosmic, Bionic, Xenial, and Trusty.

The output is similar if not identical, so pasting just one test, from Disco.

(Web Server, HTTP/HTTPS redirect, setup not included, ping me if interested.)

$ wget http://archive.ubuntu.com/ubuntu/dists/disco/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/{linux,initrd.gz}

$ GUEST=disco

$ virt-install \
  --name $GUEST \
  --vcpus 2 \
  --memory 1024 \
  --disk $GUEST.qcow2,bus=virtio,format=qcow2,size=8 \
  --network bridge=virbr0,model=virtio \
  --graphics none \
  --import \
  --boot \
kernel=linux,\
initrd=initrd.gz,\
kernel_args='console=ttyS0 url=http://192.168.122.1/preseed debian-installer/allow_unauthenticated_ssl=true'

The installer hits an error when trying to get the preseed file:

     ┌──────────┤ [!!] Download debconf preconfiguration file ├──────────┐
     │ │
     │ Failed to retrieve the preconfiguration file │
     │ The file needed for preconfiguration could not be retrieved from │
     │ http://192.168.122.1/preseed. The installation will proceed in │
     │ non-automated mode. │
     │ │
     │ <Continue> │
     │ │
     └───────────────────────────────────────────────────────────────────┘

The synthetic tests with fetch-url:
===

~ # cat /proc/cmdline
console=ttyS0 url=http://192.168.122.1/preseed debian-installer/allow_unauthenticated_ssl=true

~ # cat /etc/default-release
disco

Without patch:
---

~ # fetch-url http://192.168.122.1/preseed preseed
ERROR: cannot verify 192.168.122.1's certificate, ...
...
To connect to 192.168.122.1 insecurely, use `--no-check-certificate'.

~ # echo $?
1

With patch:
---

~ # wget --no-check-certificate http://192.168.122.1/di-utils_1.124ubuntu2_amd64.udeb
~ # udpkg -i di-utils_1.124ubuntu2_amd64.udeb

~ # fetch-url http://192.168.122.1/preseed preseed
WARNING: cannot verify 192.168.122.1's certificate, ...
...
2018-11-14 13:17:03 URL:https://192.168.122.1//preseed [11/11] -> "./_fetch-url_preseed.1467" [1]

~ # echo $?
0

With patch and Without d-i/allow_unauthenticated_ssl=true: No Change
---

~ # cat /proc/cmdline
console=ttyS0 url=http://192.168.122.1/preseed

~ # wget --no-check-certificate http://192.168.122.1/di-utils_1.124ubuntu2_amd64.udeb
~ # udpkg -i di-utils_1.124ubuntu2_amd64.udeb

~ # fetch-url http://192.168.122.1/preseed preseed
ERROR: cannot verify 192.168.122.1's certificate, ...
...
To connect to 192.168.122.1 insecurely, use `--no-check-certificate'.

~ # echo $?
1

Oops, I'll add the Debina/Ubuntu-Bug: DEP-3 tags in the debdiffs.

Eric Desrochers (slashd) wrote :

Sponsor note:

* Substitute (LP: #NNNNNNN) by (LP: #1803385)
* Re-visit version for cosmic and bionic
  - In this case, if Disco is 1.124ubuntu2 | cosmic need to be 1.124ubuntu1.18.10.1 and bionic 1.124ubuntu1.18.04.1. Please double-check with dpkg --compare-version to make sure I didn't do typos.

- Eric

Changed in debian-installer-utils (Ubuntu Trusty):
importance: Undecided → Medium
Changed in debian-installer-utils (Ubuntu Xenial):
importance: Undecided → Medium
Changed in debian-installer-utils (Ubuntu Bionic):
importance: Undecided → Medium
Changed in debian-installer-utils (Ubuntu Cosmic):
importance: Undecided → Medium
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in debian-installer-utils (Ubuntu Bionic):
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in debian-installer-utils (Ubuntu Xenial):
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in debian-installer-utils (Ubuntu Trusty):
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in debian-installer-utils (Ubuntu Cosmic):
status: New → In Progress
Changed in debian-installer-utils (Ubuntu Bionic):
status: New → In Progress
Changed in debian-installer-utils (Ubuntu Xenial):
status: New → In Progress
Changed in debian-installer-utils (Ubuntu Trusty):
status: New → In Progress
Changed in debian-installer-utils (Ubuntu):
status: Confirmed → In Progress

Er, I mean, the right LP bug numbers in changelog,
and fix the version suffixes for Bionic and Cosmic,
as reported by slashd in IRC (thanks!)

The attachment "disco_di-utils_httpsredir.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch

Eric,

Just updated the attached debdiffs w/ the fixes for the points listed.

Thanks for reviewing; nice catch in the Cosmic/Bionic version suffixes
(I have verified them with dpkg --compare-versions, all good this time).

cheers,
Mauricio

Changed in debian-installer-utils (Debian):
status: Unknown → New
Changed in debian-installer-utils (Ubuntu):
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.