Arbitrary code execution in centos template

[Gianni Tedesco]

All the RPM's which get installed in an lxc-centos container are downloaded insecurely over http and then installed with yum --nogpgcheck. A man in the middle attacker, web proxy admin, or whoever can use this to install arbitrary code in to the container which will then get executed as root.

The GPG keys should probably be shipped as part of the package where they are covered by the root of trust for the host distro. Or at the very least, https should be used to fetch the RPMs.

[Seth Arnold (seth-arnold)]

Thanks for the report Gianni, I subscribed the LXC team for their feedback.

[Stéphane Graber (stgraber)]

Yeah, looks like a bunch of our template authors didn't exactly have MITM attacks in mind when they wrote their template scripts...

The current list of templates is:
lxc-alpine.in => Okay, contains hashes of the APK keys
lxc-altlinux.in => Okay, seems to rely on apt-get for the first level of bootstrapping
lxc-archlinux.in => Calls pacstrap from the distro which seems to be doing
lxc-busybox.in => Copies local files from the distro
lxc-centos.in => Bad, runs yum with --nogpgcheck
lxc-cirros.in => Bad, downloads over plain http without any check
lxc-debian.in => Okay, uses GPG validation with keyring downloaded over https if missing
lxc-download.in => Okay, uses both https and gpg
lxc-fedora.in => Bad, downloads a base system over plain rsync and then uses yum from it with gpg validation turned off
lxc-gentoo.in => Bad, downloads over plain http with no gpg or checksum checks anywhere
lxc-openmandriva.in => Bad, uses --no-verify-rpm which I can only assume turns of gpg validation
lxc-opensuse.in => Okay, uses the distro zypper command which seems to be keeping a keyring
lxc-oracle.in => Bad, runs yum with --nogpgcheck
lxc-plamo.in => Bad, unless DLSCHEME is modified, downloads over http with no gpg check
lxc-pld.in => Okay, seems to bootstrap using "poldek" from the distro with no flag suggesting package validation is disabled
lxc-slackware.in => Okay, seems to bootstrap using "slackpkg" from the distro with gpg turned on in configuration
lxc-sparclinux.in => Bad, runs yum with --nogpgcheck
lxc-sshd.in => Okay, uses local distro binaries
lxc-ubuntu-cloud.in => Okay, uses ubuntu-cloudimg-query which handles gpg
lxc-ubuntu.in => Okay, uses the distro debootstrap which uses the distro keyring
lxc-voidlinux.in => Bad, no clear indication that xbps does gpg in any way and it's not using an https mirror

[Stéphane Graber (stgraber)]

So that's not so great, looks like we'd need fixes from template maintainers for:
 - centos
 - cirros
 - fedora (a new template is being worked on, will have to check if also affected)
 - gentoo
 - openmandriva
 - oracle
 - plamo
 - sparclinux
 - voidlinux

Some of those look trivial to fix, some of the others, no so much. Unfortunately those template scripts are community contributed and the LXC core team only actively maintains:
 - download
 - ubuntu
 - ubuntu-cloud

A few of the templates above can be trivially fixed and I'll be preparing patches for those, for the others we'll need to reach out to the various maintainers and start piling up patches for this.

[Stéphane Graber (stgraber)]

Commented in the pull request of the new fedora template, to make sure we don't merge a new template that suffers from this: https://github.com/lxc/lxc/pull/1371

[Stéphane Graber (stgraber)]

Subscribed Scott Moser as he's behind Cirros and the lxc-cirros template.

Scott: Any chance you can have https turned on for http://download.cirros-cloud.net/ ?
Looks like this is hosted by Dreamhost and they apparently offer let's encrypt for free SSL, so it may be just a knob to turn on somewhere, at which point we can just have the template use that.

If that's not an option, then we'd need to ship with the GPG key in the template and use simplestreams with gpg validation which would make the template a fair bit more complex.

[Stéphane Graber (stgraber)]

download.cirros-cloud.net is now https enabled, will provide a patch to make use of that

[Gianni Tedesco (scara)]

Are we ready to make this public?

I'd like to make some public mention of the issue in a blog post about container security.

[Stéphane Graber (stgraber)]

Yeah, we were originally considering fixing all of the individual templates but frankly it was just too much of a mess of bad patterns from a variety of different authors with no real consistency.

Instead what we came up with is distrobuilder (https://github.com/lxc/distrobuilder) which has now taken over image building duties for all the images we produce (https://images.linuxcontainers.org) and does have proper https and gpg support from the start.

All images we produce are built using public YAML definitions that can be found in https://github.com/lxc/lxc-ci and all of those either rely on https for the download of the base tarball (which then contains what's needed for the package manager to safely fetch packages) or directly contain a custom GPG keyring that's exposed to the image build.

The rest of the story is effectively the same as before, all builds happen on our infrastructure (https://jenkins.linuxcontainers.org), images are then pulled, validated and signed by a separate system which then pushes them to the image server. All artifacts are available through both valid https and gpg signed using the key that's baked into the lxc-download script.

Back in LXC 3.0 we moved the legacy template scripts to their own repository at https://github.com/lxc/lxc-templates and they are now community maintained without security/lts commitments on them on our side. Ubuntu still ships lxc-templates but it does so in universe rather than main, matching the upstream commitment.

[Stéphane Graber (stgraber)]

We're marking this issue as "Fix Released" for LXC due to the removal of all those scripts from the standard LXC distribution, instead relying on distrobuilder for our users to generate custom LXC images (which can then be consumed by the lxc-local template).

[Gianni Tedesco (scara)]

Great, thanks again for all the hard work!