ubuntu core cloud-init allows infinite creation of sudo users

[Ian Johnson]

On Ubuntu Core 18, cloud-init is configured to run w/o any restrictions. What this means is that you can craft cloud-init user-data/meta-data which creates users at any point in time during a device's lifetime by just attaching a flash drive and rebooting a device. I'm considering this a security bug as it totally bypasses any full disk encryption that we might have on Ubuntu Core 18 customers and is generally unexpected with Ubuntu Core 18.

I reproduced this with a flash drive formatted as vfat with label CIDATA, with the following files on it:

/user-data:
#cloud-config
users:
  - default
  - name: user1
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    passwd: $6$rounds=4096$ftDwPSSVP0Jq9$4hXIcusbcZMxbSnfv8D/vp/bdzgVAds9qGcFjeBvv1Ths9mLiNPKAxW8/1zOtGLPKsEcorUOzl16hn9jxswDz0

/meta-data:
instance-id: iid-local01
local-hostname: cloudimg

(the passwd hash is just "passw0rd")

With this flash drive, I can insert it and reboot a UC18 device at any point in it's lifetime, i.e. on first boot during seeding, or after seeding has completed and the device is connected to a brand store, etc. and upon booting up cloud-init will happily import this data and create the user as a sudoer, giving full root access to the device. Additionally, I can change the user-data / meta-data and reboot again (change user1 to user2 and instance-id to iid-local002, etc.) and cloud-init will happily re-import the data again. Arguably this is undesired behavior because it's expected to be able to use cloud-init to configure a device initially with a user/pw for various use-cases, but it's _not_ expected that after a device is seeded/setup/managed that cloud-init will run again creating new users.

I tested this with the released UC18 image on cdimage for Raspberry Pi 4 with the following snaps:

core18 20200427 1753 latest/stable canonical✓ base
pi 18-1 34 18-pi/stable canonical✓ gadget
pi-kernel 5.3.0-1023.25~18.04.1 135 18-pi/stable canonical✓ kernel
snapd 2.44.3 7267 latest/stable canonical✓ snapd

It's unclear how to fix this, but after discussing with Samuele and Michael, we probably want to just limit cloud-init to only run once on first-boot and then never again. This will allow current UC18 use cases for cloud-init such as testing or automated provisioning with MAAS to work, but will block attackers from adding new users at any point in time afterwards.

[Ian Johnson (anonymouse67)]

For reference, I tested this with a manually modified version of the core18 snap to not include the etc/cloud/cloud.cfg.d/90_dpkg.cfg file (which specifies to cloud-init it can use all possible data sources) and I could still reproduce this bug with Ubuntu Core 18.

[Ian Johnson (anonymouse67)]

Also verified that this bug affects Ubuntu Core 16 as well in the same way, tested with a flash drive labeled CIDATA like the bug description for Ubuntu Core 18. This time tested on a Raspberry Pi 2 running the released Ubuntu Core 16 images however since we don't have UC16 images for Raspberry Pi 4.

[Seth Arnold (seth-arnold)]

Please use CVE-2020-11933 for this issue. Thanks.

[Jamie Strandboge (jdstrand)]

Thanks Seth. To the snapd team, we'll be discussing disclosure/etc later today and I'll circle back to you with details after that.

[Samuele Pedroni (pedronis)]

After some thinking the algorithm to disable cloud-int on seeded devices, is also the algorithm we want for UC20 in the few cases were we enable it explicitly there.

This is our thinking about how to lock down cloud-init after seeding, as designed there should be negligible risk for anything up and running and deployed, this might break some unusual factory or deployment processes, but there we should be able to deal with them on a case-by-case after the fact:

https://docs.google.com/document/d/1JZaxGYerYljEG2IJX0v60LRpK-tVUhnkl4zi0o4T67M/edit#heading=h.9r3qixmb1z3y

[Jamie Strandboge (jdstrand)]

As per the email thread, adding Loic and Donald to this bug.

[Tim Chen (timchen119)]

We have checked on our UC FDE and classic platforms and would like to make sure we verify the cloud-init issue with the right way,

On machines we have (we have checked Rigardo UC16, ABB UC18, Interactive Strength UC18 and Classic),
With command line on running systems, `cloud-init status` shows it's been disabled.

fuser@localhost:~$ cloud-init status
status: disabled

However we also found the cloud-init service is enabled, while on running system it's inactive (dead).

fuser@localhost:~$ sudo systemctl status cloud-init.service
● cloud-init.service - Initial cloud-init job (metadata service crawler)
Loaded: loaded (/lib/systemd/system/cloud-init.service; enabled; vendor preset: enabled)
Active: inactive (dead)

[Jamie Strandboge (jdstrand)]

Added Dan Watkins as a subscriber to advise on cloud-init.

[Dan Watkins (oddbloke)]

Could I get a cloud-init.log from an Ubuntu Core system that has been booted with a config drive attached, please?

[Ian Johnson (anonymouse67)]

Hi Dan, by config drive do you mean a USB formatted with label CIDATA to work with NoCloud? I have not tested with the ConfigDrive datasource yet on Ubuntu Core.

[Ian Johnson (anonymouse67)]

With this user-data and meta-data like this:

#cloud-config
users:
  - default
  - name: user1
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    passwd: $6$rounds=4096$ftDwPSSVP0Jq9$4hXIcusbcZMxbSnfv8D/vp/bdzgVAds9qGcFjeBvv1Ths9mLiNPKAxW8/1zOtGLPKsEcorUOzl16hn9jxswDz0

instance-id: iid-local01
local-hostname: cloudimg

I get the first log after inserting the USB drive and rebooting the device. Then if I change the user-data and meta-data on the drive to this:

#cloud-config
users:
  - default
  - name: user2
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    passwd: $6$rounds=4096$ftDwPSSVP0Jq9$4hXIcusbcZMxbSnfv8D/vp/bdzgVAds9qGcFjeBvv1Ths9mLiNPKAxW8/1zOtGLPKsEcorUOzl16hn9jxswDz0

instance-id: iid-local02
local-hostname: cloudimg

and reboot again I get the second log. These are both from /var/log/cloud-init.log on the system.

[Ian Johnson (anonymouse67)]

Here's the second log.

[Dan Watkins (oddbloke)]

Yes, apologies, I did mean "a drive containing NoCloud config", not a ConfigDrive; poor choice of words! Thanks for those logs, I'll take a look at them now.

[Dan Watkins (oddbloke)]

OK, here's my analysis of why this is happening:

At a high level, the issue is that cloud-init will mount and use an
available NoCloud drive on any boot[0]. If that drive includes a
changed instance ID in its metadata, then cloud-init will treat this as
a first instance boot and re-run every module: this means that all
configuration that is specified in the new NoCloud drive will be
applied. This is bad!

[0] This isn't _exactly_ true, as I'll discuss below.

More specifically, here is the sequence which causes this to happen:

* cloud-init restores the NoCloud datasource instance from the previous
  boot from its pickled form on disk:
  https://github.com/canonical/cloud-init/blob/f3bd42659efeed4b092ffcdfd5df7f24813f2d3e/cloudinit/stages.py#L215
* to ensure that this unpickled instance is still applicable for this
  boot (an example of when it might not be applicable: if this instance
  has been launched from an image that was captured from a
  previously-booted instance), we call `ds.check_instance_id`:
  https://github.com/canonical/cloud-init/blob/f3bd42659efeed4b092ffcdfd5df7f24813f2d3e/cloudinit/stages.py#L230-L234
* we can see in the provided logs that the `else` is taken (search for
  "cache invalid in datasource"), which means that cloud-init has
  determined that the cached datasource object is no longer applicable
* we can see why this happens by looking at NoCloud's implementation of
  `check_instance_id`
  (https://github.com/canonical/cloud-init/blob/f3bd42659efeed4b092ffcdfd5df7f24813f2d3e/cloudinit/sources/DataSourceNoCloud.py#L209-L222 );
  we only check the kernel command line and seed directories (i.e.
  something in the instance's filesystem at one of
  /var/lib/cloud/seed/nocloud{,-net}/); we will never detect a cached
  instance ID if a NoCloud drive was the source of configuration
* because the cache is invalid, cloud-init does its datasource
  discovery process from scratch; it discovers the new NoCloud drive,
  determines that the instance ID from this new drive does not match
  the instance ID from the previous boot, and so assumes that this is
  the first boot of a new instance. This means that it reads all the
  configuration and performs all actions that aren't "per-once" (i.e.
  it performs ~all of cloud-init's actions.)

subiquity installations are not vulnerable by default because the
installer writes the NoCloud configuration into /var/lib/cloud/seed,
cloud-init _can_ determine that the instance ID of the pickled
datasource is correct, and so it doesn't re-attempt datasource
discovery, and so doesn't even go looking for the attached NoCloud
drive. If you remove /var/lib/cloud/seed, then a subiquity
installation _does_ become vulnerable. (I have tested this.)

(I haven't tested this, but I assume that if the metadata is provided
on the kernel command line then, similarly, we won't run into the
issue.)

[Jamie Strandboge (jdstrand)]

AIUI, 20.04 LTS Raspberry Pi images are similarly affected but do not include FDE. I've added a new Ubuntu cloud-init task and assigning Foundations.

@Foundations - there is a hardening opportunity for the 20.04 LTS Raspberry Pi images (ie, the CVE does not affect that product but we may want to update it and future products in a similar manner). I wasn't sure what package to assign this to so picked cloud-init. Please adjust as necessary. Also, we'll be making this bug public in a little while to give Ubuntu Core devices a chance to refresh.

[Jamie Strandboge (jdstrand)]

The snapd portion of this was fixed in 2.45.2 with a description in https://ubuntu.com/security/notices/USN-4424-1.

[Paride Legovini (paride)]

Hi,

Is there anything that still should or could be done from the cloud-init side? AIUI cloud-init behaves just as expected, but this doesn't imply that there's no room from improvement. Setting the task to Incomplete for the moment.

Paride

[Ian Johnson (anonymouse67)]

Hi Paride, I commented on https://bugs.launchpad.net/cloud-init/+bug/1883122 which I think is where cloud-init could be improved.

[James Falcon (falcojr)]

Hey Ian. I may not have the full context here, but do you think that if we address what's mentioned in 1883122 that we'd fully address what's mentioned in this bug? Could we close the cloud-init aspect of this bug in favor of 1883122?

[Richard Harding (rharding)]

I'm going to go ahead then and mark this as won't fix for cloud-init as it's behaving as expected and we have the other associated bug to work on improving the experience. I'll note this bug in a comment over there.