Ubuntu ISOs downloaded insecurely, over HTTP rather than HTTPS

[Matthew Paul Thomas]

1. Go to <https://www.ubuntu.com/>.
2. Follow the most obvious route to download the recommended version of Ubuntu for PC.

What happens: You end up downloading Ubuntu over HTTP.

What should happen: The download is over HTTPS.

An attacker with sufficient savvy and bandwidth could MITM your local Ubuntu mirror, serving you an ISO of something that looked and worked like Ubuntu but did all kinds of nefarious things.

The equivalent for software updates is bug 1186793.

Discussed on ubuntu-devel-discuss@ in September 2015. <https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2015-September/thread.html#15819>

[Originally reported by Tony Webster of "HTTP Shaming". <http://httpshaming.tumblr.com/post/95277096082/problem-1-the-iso-for-ubuntu-is-downloaded-via>]

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1359836/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Jamie Strandboge (jdstrand)]

I'm not sure if ubuntu-website-content is the right project-- feel free to reassign.

[Peter Mahnke (peterm-ubuntu)]

This is not for the webteam to decide, it is an IS issue and you will need to raise it, via an RT with them. If IS can do https, then the webteam can make the www.ubuntu.com site point to those mirrors.

[Matthew Paul Thomas (mpt)]

Sooooo, ubuntu-website-content wasn't the right project.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

After downloading the ISO image one should verify
1) checksums
2) GPG signatures on those checksums

These are available from http://releases.ubuntu.com/

e.g. For trusty
http://releases.ubuntu.com/trusty/SHA256SUMS
http://releases.ubuntu.com/trusty/SHA256SUMS.gpg

And the keys used to sign these, are in the global strongly connected GnuPG web of trust set. And one can establish a trust path to said key, via real humans. (Most ubuntu and debian developers).

However, I do take that this is slightly obscure. However, we will not rely on TLS (SSL) as that opens us up to all the TLS/SSL server and client side vulnerabilities as well as rogue CA, whilst limiting our ability to utilise CDN, mirror network and country specific mirrors.

An easier way to grab and validate checksums and GPG signatures would be nice. Imho web browsers should be able to find and validate those.

[Matthew Paul Thomas (mpt)]

Of all the weird and wonderful excuses I've seen for Web sites and downloads being insecure, I don't think I've ever seen someone claim that using TLS "opens us up to the TLS/SSL server and client side vulnerabilities". Opens us up compared to what, exactly? If you mean that an attacker could take advantage of a briefly-known TLS library vulnerability (like Goto Fail) to MITM the HTTPS download, remember that *they can already do that right now all the time* with HTTP downloads.

As far as I know Ubuntu isn't served using a CDN, and even if it was, plenty of CDNs provide HTTPS. And I'm well aware that requiring HTTPS would make mirroring more difficult, but in the equivalent RT I suggested that Let's Encrypt could be a solution to that. <https://letsencrypt.org/>

GPG-signed checksums might have been relevant in the first few months of Ubuntu's existence, when you could reasonably expect that a large proportion of downloaders would (a) bother to check them at all and (b) have the faintest idea what a "GnuPG web of trust" was. But neither of those has been remotely true for over a decade.

"Incomplete" is for bug reports that lack enough information to reproduce them. If that applies to this report, please let me know.

[Matthew Paul Thomas (mpt)]

This attack is now much less theoretical. Yesterday, someone really did backdoor the ISOs for a Linux OS, specifically Linux Mint, and also altered its Web site to point to the backdoored ISOs. <http://news.softpedia.com/news/linux-mint-website-hack-a-timeline-of-events-500719.shtml>

Unfortunately the developers were still using MD5 checksums, but even if they had been using a more secure checksum, the attacker altered the checksums on the Web site too. <http://blog.linuxmint.com/?p=2994#comment-125064>

This particular attack apparently happened via a WordPress installation. Switching to HTTPS would not have prevented someone from attacking WordPress. But it would prevent someone from achieving the same result via MITM without having to alter the Ubuntu Web site at all.

[F R (fuzzyl0gik)]

We have have a serious problem with the abuse of our networks by both state and civilian actors. Cyberwarfare and psyops is a big part of the massive war campaign in the USA.

It's really a serious shame that the main install repository from the most popular distribution for machine learning (it is part of the default stack for Google's TensorFlow, along with Python/Nvidia) doesn't have basic system integrity.

I can understand that it might be more expensive to bother with an HTTPS server for the bulk download and that the certicate authorities have been compromised in the past.

Nonetheless, not having the MD5 and SH5 hashes on a verified HTTPS connection is just recklessly, lazy security. I know Canonical has deep roots in security, and I'm just an ignoramus.

But seriously, it doesn't make sense that the initial install is the perfect target for any abusive cyberops Freedom Center. I know you guys in South Africa must feel like, But we ARE the establishment, what do WE have to fear. There is no punishment for dissent.

In the USA, things are more controversial, and numerous factions tend to abuse power much like in communist states.

Could we please have secure install images?

[Seth Arnold (seth-arnold)]

Clicking The Obvious things from https://www.ubuntu.com to get a download leads to:

https://www.ubuntu.com/download/desktop/thank-you?country=US&version=16.04.3&architecture=amd64

which has an https link to https://tutorials.ubuntu.com/tutorial/tutorial-how-to-verify-ubuntu

If you trust the CA system far enough, then this provides an easy way to get instructions how to validate the download.

Thanks

[Thomas Mayer (thomas303)]

@seth-arnold.

https://tutorials.ubuntu.com/tutorial/tutorial-how-to-verify-ubuntu?backURL=%2F#1

takes me to

http://releases.ubuntu.com/16.04/

How stupid is that?

[Chad Miller (cmiller)]

Ubuntu security is still based on Debian-Developers' complacency. A DD can check the image integrity because he visited a Debian keysigning party and has the GPG web of trust to know the image signing keys are good. That's why it's okay to have instructions and validation text in an insecure scheme. You're a DD.

If you aren't a DD, fuck off.

[Chad Miller (cmiller)]

I'm sorry. I'm frustrated that it's possible at all to be uncertain about the provenance of the root of your trust of a machine.

[Thomas Mayer (thomas303)]

@sabdfl In 2018, I would not download ubuntu in Turkey and Egypt.

https://www.bleepingcomputer.com/news/security/turkish-isp-swapped-downloads-of-popular-software-with-spyware-infected-apps/

[Matthew Paul Thomas (mpt)]

Since I reported this bug, Web browsers have started encouraging HTTPS adoption in several ways, one of which is flagging as “Not secure” more and more uses of HTTP: pages with password fields or payment card number fields, pages with any input fields at all, pages loaded in private/incognito windows, and eventually all HTTP pages.

So far, I’ve not seen these plans mentioning HTTP files downloaded from HTTPS pages. But if attacks like the one Thomas linked above increase, that may change.
Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1303739
Chromium/Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=739090

Of course, Ubuntu downloads shouldn’t be secure just to avoid embarrassment from browsers; they should be secure to help keep users safe.

[TJ (tj)]

Followed up on this today. No joy. IRC log from #ubuntu-hardened:

12:37 <TJ-> Could we revisit Bug #1359836 ? At the very least the checksum files should require HTTPS because most new users have no idea (nor sometimes, facility) to verify using GPG - think Windows users coming to Ubuntu. This affects both {releases,cdimage}.ubuntu.com
12:37 <ubot5> bug 1359836 in Ubuntu "Ubuntu ISOs downloaded insecurely, over HTTP rather than HTTPS" [Undecided,Confirmed] https://launchpad.net/bugs/1359836
12:40 <maswan> our mirror is happy to serve them over https, as long as the name ftp.acc.umu.se and not se.releases.ubuntu.com is used
12:41 <TJ-> maswan: yes, several mirrors are using HTTPS but I'd think most users will download them via the ubuntu.com domain
12:44 <maswan> yeah
12:45 <maswan> my favourite suggestion is to run a central mirrorbits instance that serves checksum files directly over https and then redirects to http[s] sources
12:46 <maswan> but it is not me running the services, so it is just a suggestion
12:47 <TJ-> Last time I discussed this, maybe in canonical-sysadmin, the reply was bascially "our infrastructure is complicated and HTTPS would require expensive resources"
12:47 <TJ-> My view is it is a CODB (cost of doing business) if Canonical wants to be taken seriously
12:52 <maswan> Yeah. My mirrorbits solution would around to encourage more use of [verified good] mirrors to offset the direct costs a bit
12:54 <maswan> Something like https://download.lineageos.org/star2lte
12:55 <amurray> TJ-: that is still the case as far as I understand it - TLS overhead is non-trivial CPU time and cost-wise, plus the existing mirror system of xx.archive.ubuntu.com pointing to 3rd-party hosted mirrors would mean Canonical has to give certs to 3rd parties which are trusted (to allow to use xx.archive.canonical.com) which is difficult
12:55 <mdeslaur> TJ-: we brought this up again a few months ago and we got updates quotes, and it was still cost prohibitive
12:56 <TJ-> mdeslaur: can we get that added to the bug report then?
12:56 <mdeslaur> I'm just the messenger, and am not involved at all in that
12:57 <TJ-> mdeslaur: right, but whoever is responsible, since it is certainly a security issue
12:58 <mdeslaur> I can ask next time

[TJ (tj)]

Additional, more positive, discussion.

12:59 <maswan> or just relax the policy and let us get LE certs outselves, just like for all the other names
13:00 <mdeslaur> maybe we could do that if we registered a new domain name
13:01 <mdeslaur> I'll bring it up again soon
13:03 <mdeslaur> the problem is we round robin mirrors under country-specific domains...not sure how we would handle that
13:04 <mdeslaur> (not that I've thought about it, the people who are responsible for it have)
13:04 <maswan> yeah, that's why I like the mirrorbits solution for image downloads, it makes the mirror name much less important and you can probably drop *.releases.u.c. archive is another matter and will be more complicated, especially since those are in long-term config files
13:05 <mdeslaur> hrm, good point, may be worth separating those two and just doing the image downloads somehow first
13:07 <mdeslaur> there's also the issue that we don't trust half the mirrors in the list, so just using ssl doesn't do much to improve security
13:08 <maswan> yeah, that's another reason why I like mirrorbits serving up the checksum in the redirector, not relying on downloading the checksum file from the mirror
13:10 <mdeslaur> I need to look into that

[Thomas Mayer (thomas303)]

In respect to the above discussion TJ posted:

Please be aware that https-securing mirrors does in fact not necessarily increase the trustworthyness of the download. Reason:

An attacker could compromise a mirror's downloads (e.g. via stolen credentials or via MITM while the mirror downloads via http) or the attacker could create a new mirror. The mirror would be perfectly secured via https. Accordingly, compromised checksums would be provided via e.g. the mirror as well. And the attacker's malware would just be downloaded via https, even the https-downloaded checksums of the mirror would seem right. On the side of the victim, this can lead to a false sense of security.

Instead, the _producer_ of the software must give the commitment which checksum is right, not some random guy (e.g. a mirror). Consequently, the downloading person must check checksums against the checksums the _producer_ provided - and not only check checksums but also check the trustworthyness of the _origin_ of these checksums.

Consequently, the checksums should be provided via a https-secured domain or at least subdomain of the _producer_ (e.g. https://www.ubuntu.com/...).

[Thomas Mayer (thomas303)]

Here's a perfect illustration how NOT to protect against MITM: http://www.system-rescue-cd.org/Download/

Assumed, the attacker _can_ attack via MITM, then

1. the attacker can let the download link point somewhere else (e.g. to a compromised download).
2. the attacker can _also_ show a checksum which was calculated for the compromised download but would be wrong for the original download.

The victim successfully checks compromised download against compromised checksum and everything seems fine.

For the scenario described in my previous comment, replace MITM attacker with "mirror" and trust is gone pretty much in the same manner.

[Thomas Mayer (thomas303)]

I'd like to sum it up like this: Users should _download_ from a mirror but they should neither _trust_ the download of the mirror nor the checksums a mirror provides.

It's even the other way round: Having mirrors in the game makes it _even more_ important that checksums are provided by Canonical and that the user can verify both integrity _and_ origin (=> Canonical's domain) of the _checksums_. That's what TLS provides (besides encryption) when done right.

[Seth Arnold (seth-arnold)]

On Wed, Jun 05, 2019 at 12:13:54AM -0000, Thomas Mayer wrote:
> I'd like to sum it up like this: Users should _download_ from a mirror
> but they should neither _trust_ the download of the mirror nor the
> checksums a mirror provides.

Users can trust checksums provided by mirrors because we publish
signatures on the SHA256SUMS files.

If the user has a copy of GnuPG that they trust, they can use it to verify
the SHA256SUMS file. We've published instructions on how to do this at:
https://tutorials.ubuntu.com/tutorial/tutorial-how-to-verify-ubuntu#3

> It's even the other way round: Having mirrors in the game makes it _even
> more_ important that checksums are provided by Canonical and that the
> user can verify both integrity _and_ origin (=> Canonical's domain) of
> the _checksums_. That's what TLS provides (besides encryption) when done
> right.

Because the SHA256SUMS files are generated and signed by us, they're going
to be identical across all mirrors -- at least among the mirrors that keep
up to date.

It doesn't matter where the checksums and signatures are retrieved from,
so long as they are fresh. It is difficult to determine what exactly
"fresh" means, but GnuPG will report the time that a signature was
created:

$ gpg --verify SHA256SUMS.gpg SHA256SUMS
gpg: Signature made Fri 15 Feb 2019 08:32:38 AM PST
gpg: using DSA key 46181433FBB75451
gpg: Good signature from "Ubuntu CD Image Automatic Signing Key <email address hidden>" [full]
gpg: Signature made Fri 15 Feb 2019 08:32:38 AM PST
gpg: using RSA key D94AA3F0EFE21092
gpg: Good signature from "Ubuntu CD Image Automatic Signing Key (2012) <email address hidden>" [full]

Double-checking the desired image against e.g.
https://wiki.ubuntu.com/Releases to find out when the signatures should
have been created is about the only way to address the freshness problem.
That is a slight wrinkle of using mirrors rather than Ubuntu's own
infrastructure. Someone using our archives directly could skip this check.

Thanks

[Thomas Mayer (thomas303)]

@seth-arnold

There we go and let an imaginary grandma (she's a non-DD) verify an ubuntu ISO image via gpg. Of course, she will know by herself which DSA key IDs are trusted and not just extract the (MITM-compromised) IDs from the (MITM-compromised) SHA256SUMS.gpg as described in https://tutorials.ubuntu.com/tutorial/tutorial-how-to-verify-ubuntu#3 The attacker then can't trick our grandma in verifying the (MITM-compromised) ubuntu ISO. Bravo.

I cite: "This is actually a really useful message, as it tells us which key or keys were used to generate the signature file. [...] Knowing these ID numbers [...], means we can request them from the Ubuntu key server.".

No way, it's NOT useful at all. We don't want to see the MITM's DSA key IDs when processing a MITM-compromised SHA256SUMS.gpg file. We don't want to download the attackers keys via hkp and never ever do we want to verify the MITM-compromised our ISO using the attacker's keys.

I was just revisiting the download dialogue. For me, this whole discussion turned into an academic one because when starting the download, the user _now_ gets shown an ubuntu-domain-https-secured checksum for the download. Hooray. It's that simple. That is pretty much what I was waiting for for years - and something we already had a couple of years ago.

https://www.ubuntu.com/download/desktop/thank-you?country=DE&version=18.04.2&architecture=amd64

For the downloads themselves, https only makes downloads from ubuntu domain trustworthy. For mirrors, verification via trustworthy checksums is still needed, be it with http or https downloads from a (non-trusted) mirror. Still, having https for mirrors protects from MITM attacks between user and mirror. When users leave out the verification (out of lazyness or lack of knowledge), this can still mitigate a portion of the attacks which would have been successful otherwhise. But there's no guarantee any more.

[Matthew Paul Thomas (mpt)]

Credit to Peter Mahnke and others on Canonical’s Web & Design team for converting ubuntu.com to HTTPS, and separately for embedding the exact verification terminal command — with checksum included, and even a Copy button! — on the (now-HTTPS) “Thank you” page when downloading standard LTS-desktop, latest-desktop, LTS-server, and latest-server images. (I’m now on that team, but I wasn’t involved in that work, and opinions here are my own.)

The instructions as written don’t work on macOS or other BSDs, Chrome OS, or (without great effort) Windows, but those are fixable. And releases.ubuntu.com, cloud-images.ubuntu.com, old-releases.ubuntu.com, and mirror file listings are still HTTP with no verification instructions, though people using those sites are perhaps more willing and able to work it out for themselves.

However, the current verification system can’t protect, *in a way that HTTPS would*:
* people not willing/bothering to run the verification command (probably the biggest category)
* Windows users not knowing/willing/able to install either Microsoft File Checksum Integrity Verifier or, as suggested in the linked tutorial, Ubuntu for Windows (download a second copy of Ubuntu just to verify your download of the first copy? really?)
* Windows S Mode users not knowing/willing to switch out of S Mode.

Of course, as Thomas wrote, “the attacker could create a new mirror … perfectly secured via https”. (Though if they did, at the very least, we’d remove them from the list of mirrors!) Or they could attack your browser’s key store such that it trusts shonky certificates, or engage in corporate sabotage, or find an SSL zero-day like Dimitri mentioned, or, or, or … The point here is not to protect against every possible attack. The point is to reduce the attack surface.

[Thomas Mayer (thomas303)]

@mpt There's also non-public mirrors in the field which have never been on the list of mirrors. And never will be.

For public mirrors on the list, how would Canonical know about a compromised mirror _before_ a victim downloads from it?

I'm still very happy with having https'ed mirrors, because it secures the download between mirror and user at least - against e.g. MITM. That should be by far the largest attack vector these days.

It's just that a download which did not come via https from ubuntu.com still has to be verified. And now that Canonical (aka the _producer_) presents a https-secured checksum from a trustworthy domain (e.g. ubuntu.com) everything is in place.

[Thomas Mayer (thomas303)]

For the record in terms of Google's Chrome browser: Google announced that they plan to block http downloads when the user comes from a https page.

Source: https://www.xda-developers.com/google-chrome-block-insecure-downloads-https-pages/

[Joel Sing (jsing)]

As a heads up, releases is now available over HTTPS:

  https://releases.ubuntu.com/

Additionally, the CD mirrors list on Launchpad now also includes https entries where supported:

  https://launchpad.net/ubuntu/+cdmirrors

[Joel Sing (jsing)]

Download links from ubuntu.com have been redirecting to HTTPS (for sometime now).

Additionally, cdimage is now also available over HTTPS:

  https://cdimage.ubuntu.com/

I believe this bug can now be marked as fixed.

[Joel Sing (jsing)]

A correction to my previous comment - download links to releases.ubuntu.com are HTTPS, however download links to cdimage.ubuntu.com still need to be updated. Once this is complete the bug can be marked as fixed.

[Brian Foster (blfoster)]

Please also see bug https://bugs.launchpad.net/ubuntu/+bug/1186793 ("Updating is over insecure connection"), updating an installation using HTTPS rather HTTP seems possible, but perhaps not completely? And doesn't seem to be the default!

Also, as per comment https://bugs.launchpad.net/ubuntu/+bug/1186793/comments/14 in that issue ("Attacks against GPG signed APT repositories", https://blog.packagecloud.io/attacks-against-gpg-signed-apt-repositories/), there have been concerns raised by outside(?)/independent(?) analysis of Canonical using HTTP rather than HTTPS.

[Junien F (axino)]

As far as I can see, all links to download Ubuntu are now HTTPS, so I'm resolving this bug.

Please let us know otherwise.