ubuntu core cloud-init allows infinite creation of sudo users

Bug #1879530 reported by Ian Johnson
278
This bug affects 1 person
Affects Status Importance Assigned to Milestone
snapd
High
Ian Johnson
cloud-init (Ubuntu)
Undecided
Unassigned

Bug Description

On Ubuntu Core 18, cloud-init is configured to run w/o any restrictions. What this means is that you can craft cloud-init user-data/meta-data which creates users at any point in time during a device's lifetime by just attaching a flash drive and rebooting a device. I'm considering this a security bug as it totally bypasses any full disk encryption that we might have on Ubuntu Core 18 customers and is generally unexpected with Ubuntu Core 18.

I reproduced this with a flash drive formatted as vfat with label CIDATA, with the following files on it:

/user-data:
#cloud-config
users:
  - default
  - name: user1
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    passwd: $6$rounds=4096$ftDwPSSVP0Jq9$4hXIcusbcZMxbSnfv8D/vp/bdzgVAds9qGcFjeBvv1Ths9mLiNPKAxW8/1zOtGLPKsEcorUOzl16hn9jxswDz0

/meta-data:
instance-id: iid-local01
local-hostname: cloudimg

(the passwd hash is just "passw0rd")

With this flash drive, I can insert it and reboot a UC18 device at any point in it's lifetime, i.e. on first boot during seeding, or after seeding has completed and the device is connected to a brand store, etc. and upon booting up cloud-init will happily import this data and create the user as a sudoer, giving full root access to the device. Additionally, I can change the user-data / meta-data and reboot again (change user1 to user2 and instance-id to iid-local002, etc.) and cloud-init will happily re-import the data again. Arguably this is undesired behavior because it's expected to be able to use cloud-init to configure a device initially with a user/pw for various use-cases, but it's _not_ expected that after a device is seeded/setup/managed that cloud-init will run again creating new users.

I tested this with the released UC18 image on cdimage for Raspberry Pi 4 with the following snaps:

core18 20200427 1753 latest/stable canonical✓ base
pi 18-1 34 18-pi/stable canonical✓ gadget
pi-kernel 5.3.0-1023.25~18.04.1 135 18-pi/stable canonical✓ kernel
snapd 2.44.3 7267 latest/stable canonical✓ snapd

It's unclear how to fix this, but after discussing with Samuele and Michael, we probably want to just limit cloud-init to only run once on first-boot and then never again. This will allow current UC18 use cases for cloud-init such as testing or automated provisioning with MAAS to work, but will block attackers from adding new users at any point in time afterwards.

CVE References

Revision history for this message
Ian Johnson (anonymouse67) wrote :

For reference, I tested this with a manually modified version of the core18 snap to not include the etc/cloud/cloud.cfg.d/90_dpkg.cfg file (which specifies to cloud-init it can use all possible data sources) and I could still reproduce this bug with Ubuntu Core 18.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Also verified that this bug affects Ubuntu Core 16 as well in the same way, tested with a flash drive labeled CIDATA like the bug description for Ubuntu Core 18. This time tested on a Raspberry Pi 2 running the released Ubuntu Core 16 images however since we don't have UC16 images for Raspberry Pi 4.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Please use CVE-2020-11933 for this issue. Thanks.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Thanks Seth. To the snapd team, we'll be discussing disclosure/etc later today and I'll circle back to you with details after that.

Changed in snapd:
assignee: nobody → Ian Johnson (anonymouse67)
status: New → Confirmed
importance: Undecided → High
Changed in snapd:
status: Confirmed → In Progress
summary: - uc18 cloud-init allows infinite creation of sudo users
+ ubuntu core cloud-init allows infinite creation of sudo users
Revision history for this message
Samuele Pedroni (pedronis) wrote :

After some thinking the algorithm to disable cloud-int on seeded devices, is also the algorithm we want for UC20 in the few cases were we enable it explicitly there.

This is our thinking about how to lock down cloud-init after seeding, as designed there should be negligible risk for anything up and running and deployed, this might break some unusual factory or deployment processes, but there we should be able to deal with them on a case-by-case after the fact:

https://docs.google.com/document/d/1JZaxGYerYljEG2IJX0v60LRpK-tVUhnkl4zi0o4T67M/edit#heading=h.9r3qixmb1z3y

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

As per the email thread, adding Loic and Donald to this bug.

Revision history for this message
Tim Chen (timchen119) wrote :

We have checked on our UC FDE and classic platforms and would like to make sure we verify the cloud-init issue with the right way,

On machines we have (we have checked Rigardo UC16, ABB UC18, Interactive Strength UC18 and Classic),
With command line on running systems, `cloud-init status` shows it's been disabled.

fuser@localhost:~$ cloud-init status
status: disabled

However we also found the cloud-init service is enabled, while on running system it's inactive (dead).

fuser@localhost:~$ sudo systemctl status cloud-init.service
● cloud-init.service - Initial cloud-init job (metadata service crawler)
Loaded: loaded (/lib/systemd/system/cloud-init.service; enabled; vendor preset: enabled)
Active: inactive (dead)

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Added Dan Watkins as a subscriber to advise on cloud-init.

Revision history for this message
Dan Watkins (oddbloke) wrote :

Could I get a cloud-init.log from an Ubuntu Core system that has been booted with a config drive attached, please?

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Hi Dan, by config drive do you mean a USB formatted with label CIDATA to work with NoCloud? I have not tested with the ConfigDrive datasource yet on Ubuntu Core.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

With this user-data and meta-data like this:

#cloud-config
users:
  - default
  - name: user1
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    passwd: $6$rounds=4096$ftDwPSSVP0Jq9$4hXIcusbcZMxbSnfv8D/vp/bdzgVAds9qGcFjeBvv1Ths9mLiNPKAxW8/1zOtGLPKsEcorUOzl16hn9jxswDz0

instance-id: iid-local01
local-hostname: cloudimg

I get the first log after inserting the USB drive and rebooting the device. Then if I change the user-data and meta-data on the drive to this:

#cloud-config
users:
  - default
  - name: user2
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    passwd: $6$rounds=4096$ftDwPSSVP0Jq9$4hXIcusbcZMxbSnfv8D/vp/bdzgVAds9qGcFjeBvv1Ths9mLiNPKAxW8/1zOtGLPKsEcorUOzl16hn9jxswDz0

instance-id: iid-local02
local-hostname: cloudimg

and reboot again I get the second log. These are both from /var/log/cloud-init.log on the system.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Here's the second log.

Revision history for this message
Dan Watkins (oddbloke) wrote :

Yes, apologies, I did mean "a drive containing NoCloud config", not a ConfigDrive; poor choice of words! Thanks for those logs, I'll take a look at them now.

Revision history for this message
Dan Watkins (oddbloke) wrote :

OK, here's my analysis of why this is happening:

At a high level, the issue is that cloud-init will mount and use an
available NoCloud drive on any boot[0]. If that drive includes a
changed instance ID in its metadata, then cloud-init will treat this as
a first instance boot and re-run every module: this means that all
configuration that is specified in the new NoCloud drive will be
applied. This is bad!

[0] This isn't _exactly_ true, as I'll discuss below.

More specifically, here is the sequence which causes this to happen:

* cloud-init restores the NoCloud datasource instance from the previous
  boot from its pickled form on disk:
  https://github.com/canonical/cloud-init/blob/f3bd42659efeed4b092ffcdfd5df7f24813f2d3e/cloudinit/stages.py#L215
* to ensure that this unpickled instance is still applicable for this
  boot (an example of when it might not be applicable: if this instance
  has been launched from an image that was captured from a
  previously-booted instance), we call `ds.check_instance_id`:
  https://github.com/canonical/cloud-init/blob/f3bd42659efeed4b092ffcdfd5df7f24813f2d3e/cloudinit/stages.py#L230-L234
* we can see in the provided logs that the `else` is taken (search for
  "cache invalid in datasource"), which means that cloud-init has
  determined that the cached datasource object is no longer applicable
* we can see why this happens by looking at NoCloud's implementation of
  `check_instance_id`
  (https://github.com/canonical/cloud-init/blob/f3bd42659efeed4b092ffcdfd5df7f24813f2d3e/cloudinit/sources/DataSourceNoCloud.py#L209-L222 );
  we only check the kernel command line and seed directories (i.e.
  something in the instance's filesystem at one of
  /var/lib/cloud/seed/nocloud{,-net}/); we will never detect a cached
  instance ID if a NoCloud drive was the source of configuration
* because the cache is invalid, cloud-init does its datasource
  discovery process from scratch; it discovers the new NoCloud drive,
  determines that the instance ID from this new drive does not match
  the instance ID from the previous boot, and so assumes that this is
  the first boot of a new instance. This means that it reads all the
  configuration and performs all actions that aren't "per-once" (i.e.
  it performs ~all of cloud-init's actions.)

subiquity installations are not vulnerable by default because the
installer writes the NoCloud configuration into /var/lib/cloud/seed,
cloud-init _can_ determine that the instance ID of the pickled
datasource is correct, and so it doesn't re-attempt datasource
discovery, and so doesn't even go looking for the attached NoCloud
drive. If you remove /var/lib/cloud/seed, then a subiquity
installation _does_ become vulnerable. (I have tested this.)

(I haven't tested this, but I assume that if the metadata is provided
on the kernel command line then, similarly, we won't run into the
issue.)

Changed in snapd:
status: In Progress → Fix Committed
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

AIUI, 20.04 LTS Raspberry Pi images are similarly affected but do not include FDE. I've added a new Ubuntu cloud-init task and assigning Foundations.

@Foundations - there is a hardening opportunity for the 20.04 LTS Raspberry Pi images (ie, the CVE does not affect that product but we may want to update it and future products in a similar manner). I wasn't sure what package to assign this to so picked cloud-init. Please adjust as necessary. Also, we'll be making this bug public in a little while to give Ubuntu Core devices a chance to refresh.

Changed in cloud-init (Ubuntu):
assignee: nobody → Ubuntu Foundations Team (ubuntu-foundations-team)
Changed in snapd:
milestone: none → 2.45.2
status: Fix Committed → Fix Released
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

The snapd portion of this was fixed in 2.45.2 with a description in https://ubuntu.com/security/notices/USN-4424-1.

information type: Private Security → Public Security
Revision history for this message
Paride Legovini (paride) wrote :

Hi,

Is there anything that still should or could be done from the cloud-init side? AIUI cloud-init behaves just as expected, but this doesn't imply that there's no room from improvement. Setting the task to Incomplete for the moment.

Paride

Changed in cloud-init (Ubuntu):
status: New → Incomplete
Revision history for this message
Ian Johnson (anonymouse67) wrote :

Hi Paride, I commented on https://bugs.launchpad.net/cloud-init/+bug/1883122 which I think is where cloud-init could be improved.

Changed in cloud-init (Ubuntu):
status: Incomplete → New
Revision history for this message
James Falcon (falcojr) wrote :

Hey Ian. I may not have the full context here, but do you think that if we address what's mentioned in 1883122 that we'd fully address what's mentioned in this bug? Could we close the cloud-init aspect of this bug in favor of 1883122?

Revision history for this message
Richard Harding (rharding) wrote :

I'm going to go ahead then and mark this as won't fix for cloud-init as it's behaving as expected and we have the other associated bug to work on improving the experience. I'll note this bug in a comment over there.

Changed in cloud-init (Ubuntu):
status: New → Won't Fix
assignee: Ubuntu Foundations Team (ubuntu-foundations-team) → nobody
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers