ubiquity should support encryption by default with zfsroot, with users able to opt in to running change-key after install

[Steve Langasek]

zfs supports built-in encryption support, but the decision of whether a pool is encrypted or not must be made at pool creation time; it is possible to add encrypted datasets on top of an unencrypted pool but it is not possible to do an online change of a dataset (or a whole pool) to toggle encryption.

We should therefore always install with encryption enabled on zfs systems, with a non-secret key by default, and allow the user to use 'zfs change-key -o keylocation=prompt' after install to take ownership of the encryption and upgrade the security.

This is also the simplest way to allow users to avoid having to choose between the security of full-disk encryption, and the advanced filesystem features of zfs since it requires no additional UX work in ubiquity.

We should make sure that https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1857040 is fixed first in the kernel so that enabling zfs encryption does not impose an unreasonable performance penalty.

[Steve Langasek (vorlon)]

Here is a proof-of-concept patch for the zfs-initramfs side of such a feature.

[Steve Langasek (vorlon)]

ubiquity side of the equation: https://code.launchpad.net/~vorlon/ubiquity/+git/ubiquity/+ref/lp.1857398

[Richard Laager (rlaager)]

This is an interesting approach. I figured the installer should prompt for encryption, and it probably still should, but if the performance impact is minimal, this does have the nice property of allowing for enabling encryption post-install.

It might be worthwhile (after merging the SIMD fixes) to benchmark aes256-ccm (the default) vs encryption=aes-256-gcm. I think GCM seems to be preferred, security wise, in various places (though I don't immediately have references) and may be faster. There's also an upstream PR in progress that significantly improves AES-GCM: https://github.com/zfsonlinux/zfs/pull/9749

[Richard Laager (rlaager)]

Here are some quick performance comparisons:
https://github.com/zfsonlinux/zfs/pull/9749#issuecomment-569132997

In summary, "the GCM run is approximately 1.15 times faster than the CCM run. Please also note that this PR doesn't improve AES-CCM performance, so if this gets merged, the speed difference will be much larger."

I would recommend setting encryption=aes256-gcm instead of encryption=on (which is aes256-ccm).

[Richard Laager (rlaager)]

I have come up with a potential security flaw with this design:

The user installs Ubuntu with this fixed passphrase. This is used to derive the "user key", which is used to encrypt the "master key", which is used to encrypt their data. The encrypted version of the master key is obviously written to disk.

Later, the user changes their passphrase. This rewraps the master key with a new user key (derived from the new/real passphrase). It writes that to disk. But, I presume that does NOT overwrite the old wrapped key in place on disk. I don't actually know this, but I am assuming so based on the general design of ZFS being copy-on-write. As far as I know, only uberblocks are rewritten in place.

Therefore, it is possible for some indeterminate amount of time to read the old wrapped master key off the disk, which can be decrypted using the known passphrase. This gives the master key, which can then be used to decrypt the _existing_ data.

If the master key is not rotated when using zfs change-key, then _new_ data can also be read for some indefinite period of time. I'm not 100% sure whether change-key changes the master key or only the user key. From the man page, it sounds like it does change the master key. It says, "...use zfs change-key to break an existing relationship, creating a new encryption root..."

I'll try to get a more clueful answer on these points.

[Richard Laager (rlaager)]

I put these questions to Tom Caputi, who wrote the ZFS encryption. The quoted text below is what I asked him, and the unquoted text is his response:

> 1. Does ZFS rewrite the wrapped/encrypted master key in place? If
> not, the old master key could be retrieved off disk, decrypted
> with the known passphrase, and used to decrypt at least
> _existing_ data.

1) No. This is definitely an attack vector (although a very minor
   one). At the time we had said that we would revisit the idea of
   overwriting old keys when TRIM was added. That was several years ago
   and TRIM is now in. I will talk to Brian about it after I am back
   from the holiday.

> 2. Does a "zfs change-key" create a new master key? If not, the old
> master key could be used to decrypt _new_ data as well, at least
> until the master key is rotated.

2) zfs change-key does not create a new master key. It simply re-wraps
   the existing master key. The master keys are never rotated. The key
   rotation is done by using the master keys to generate new keys.

[Didier Roche-Tolomelli (didrocks)]

Thanks Richard for digging in, the performance comparison and the valuable upstream feedback and pointers.
Good catch about retrieving the master key written in old blocks with the previous (fix) passphrase even if changed later on. It seems that trimming could help. Do you think that we should base on work going that direction (overwriting old keys) and keep the current approach?

On a more general side, the approach seems to be forward-compatible with per user dataset encryption (zfs change-key <dataset>), which creates a new encryption root.

Steve:
* the only comment I have on the ubiquity part of the equation is based on Richard's feedback. Otherwise, looks good to me. I think we should wait on the above feedback before taking a finale decision on the approach though.
* the zfs-linux initramfs POC looks good (not tested though, currently travelling, but I didn't spot any issues). It should be easily pluggable later on once the user set it to "prompt" with their own passphrase and use the plymouth prompt codepath. (Not tested yet either).
Just a nitpick: Colin asked for our patches in zfs-linux to be numbered (hence the 4XXX- namespace), as the debian ones. It seems that it's not reliably the case since the 0.8.2 merge with debian, so need double checking (order seems as well messy after this merge right now).

Anyway, we need to wait on the kernel patches first.

[Didier Roche-Tolomelli (didrocks)]

One last thing: I think we should test this on rotational disk and assess the performance impacts before pushing it as a default. This will give us a good baseline to decide if this should be pushed or if we need to add even more warnings on the ZFS install option.

[Richard Laager (rlaager)]

I've given this a lot of thought. For what it's worth, if it were my decision, I would first put your time into making a small change to the installer to get the "encryption on" case perfect, rather than the proposal in this bug.

The installer currently has:

 O Erase disk an install Ubuntu
   Warning: ...
   [] Encrypt the new Ubuntu installation for security
      You will choose a security key in the next step.
   [] Use LVM with the new Ubuntu installation
      This will set up Logical Volume Management. It allows taking
      snapshots and easier partition resizing.
 O EXPERIMENTAL: Erase disk and use ZFS
   Warning: This will delete all your files on all operating systems.
   This is experimental and may cause data loss. Do not use on
   production systems.
 O Something else
   ...

I would move the ZFS option to be a peer of / alternative to LVM instead:

 O Erase disk and install Ubuntu
   Warning: ...
   [] Encrypt the new Ubuntu installation for security
      You will choose a security key in the next step.
   Volume Managment:
     O None (Fixed Partitions)
     O Logical Volume Manager (LVM)
        LVM allows taking snapshots and easier partition resizing.
     O EXPERIMENTAL: ZFS
        ZFS allows taking snapshots and dynamically shares space between
        filesystems.
        Warning: This is experimental and may cause data loss. Do not use
        on production systems.
 O Something else
   ...

This is a very straightforward UI change. The only new combination introduced with this UI is encryption on + ZFS, which is what we want. In that scenario, run the same passphrase prompting screen that is used now for LUKS. Then pass the passphrase to `zpool create` (and use encryption=aes-256-gcm for the reasons already discussed).

If the "always enable encryption" feature is to future-proof for people who would otherwise choose "no encryption", that's worth considering, but if it's an alternative to prompting them in the installer, I'm personally opposed.

However, we do need to consider why they're turning off encryption. Are they saying, "I don't want encryption ever (e.g. because of the performance penalty)." or "I don't care about encryption right now." If you always enable encryption, you are forcing encryption on them, which has real performance impacts on older hardware. For example, I just yesterday upgraded my personal server to use ZFS encryption, but made a media dataset that is unencrypted. Sustained writes to the media dataset are _at least_ twice as fast. With encryption, I was CPU bound. With it off, I was not, so I suspect I could have written even faster. This system is older and does not have AES-NI.

You mentioned spinning disks. Perhaps I misunderstood, but I don't know why you'd be asking about spinning disks in particular. They are slower than SSDs, so encryption is less likely to be a concern there, not more. My server scenario involved spinning disks.

If the old wrapped master key were overwritten when changed _and_ the system has AES-NI instructions, then I think it would be reasonable to make "no encryption" turn on encryption anyway with a fixed passphrase. This would achieve the goal of allowing...

[Jean-Baptiste Lallement (jibel)]

I agree with Richard and we had the same discussion yesterday with Didier.
We should expose zfs encryption in Ubiquity and align is on LVM. This would leave the decision to the user to use or not ZFS encryption and not force him to use it if he selects ZFS.

Secondly, we don't have any sort of feedback and the measure of the impact of zfs encryption on our users is very limited.

It seems to me very risky to enable it by default without any way to disable it, especially for an LTS and on a desktop where users are not necessarily familiar with the CLI.

@Steve could you add a section with your vision of ZFS encryption to the "ZFS on root" specification so we can discuss and refine the approach there?

Thanks.

[Garrett Fields (fields-g)]

FWIW: Running Ubiquity 20.04 with a modified "zsys-setup" configuration file that manually incorporates a password and encryption pool properties works great.

echo <password> | zpool create -f \
  -O encryption=aes-256-gcm \
  -O keylocation=prompt \
  -O keyformat=passphrase \
  ......
  -O mountpoint=/ -R "${target}" rpool "${partrpool}"

This works especially well now that the "plymouth ask-for-password" is working.

Though a known password file would allow an autounlock mechanism until the change-key is done, I believe it would be rather trivial to have Ubiquity collect a password from the user, use "-O keylocation=prompt" and to expect the user to provide the password every reboot.

The performance penalty and the potential for a misguided perception of security from encrypting everything yet "leaving the key in the handle until you rekey" seems to be a much.

[Steve Langasek (vorlon)]

> I believe it would be rather trivial to have Ubiquity collect
> a password from the user, use "-O keylocation=prompt" and to
> expect the user to provide the password every reboot.

It is not appropriate to require the user to type a password on every boot by default; this must be opt-in.

The purpose of this change is to make it possible to defer the question of whether to encrypt, without requiring the user to reinstall. This is similar to what other operating systems do nowadays, and provided the performance penalty is reasonable, is still my recommendation.

[Steve Langasek (vorlon)]

> I would recommend setting encryption=aes256-gcm instead of
> encryption=on (which is aes256-ccm).

I think the right way to handle this is to change the behavior of zfs-linux so that encryption=on defaults to the recommended algorithm - rather than hard-coding the algorithm selection in ubiquity, which is generally speaking a good recipe for bit rot.

[Richard Laager (rlaager)]

> It is not appropriate to require the user to type a password on every
> boot by default; this must be opt-in.

Agreed.

The installer should prompt (with a checkbox) for whether the user wants encryption. It should default to off. If the user selects the checkbox, prompt them for a passphrase. Setup encryption using that passphrase. This is exactly how the installer behaves today for non-ZFS (e.g. using LUKS). I'm proposing to extend that existing behavior to ZFS. Thi should be trivial to implement; I'm not sure if we still have time for 20.04, but I'd really love to see at least this much implemented now.

What should happen if the user leaves the encryption box unchecked? Currently, they get no encryption, and that's what I'm proposing initially. You'd like to improve that so that the user can later set a passphrase without having to reformat their disk. I agree that's a reasonable goal.

I think the blockers / potential blockers are:

1) `zfs change-key` does not overwrite the old wrapped master key on disk, so it is accessible to forensic analysis. Given that the old wrapping key is a known passphrase ("ubuntuzfs"), another way of looking at this is that the master key is still on disk in what is, security-wise, effectively plaintext. I (and other upstream ZFS developers) are concerned about giving the user a false sense of security in this situation. ZFS could overwrite the key on disk when changed. If/when someone adds that enhancement to `zfs change-key`, then I think this objection goes away. I don't see this being implemented in time for 20.04.

2) Is the performance acceptable? On older systems without AES-NI, there is a noticeable impact, which I've seen myself. I recommended using AES-NI support as the deciding factor here... if they have AES-NI, then encrypt (with a known passphrase) even if the user didn't opt-in; if they don't have AES-NI, then not opting-in means encryption is really off. If that inconsistency is a problem, then ultimately Ubuntu just has to decide one way or the other. Personally, I'm a big fan of encryption, so I'm not going to be upset if the decision is that the performance impact on older hardware is just something to accept.

> > I would recommend setting encryption=aes256-gcm instead of
> > encryption=on (which is aes256-ccm).
>
> I think the right way to handle this is to change the behavior of
> zfs-linux so that encryption=on defaults to the recommended algorithm -

Agreed. I proposed this at the last OpenZFS Leadership meeting and there is general agreement to do so. It does need a bit more discussion and then implementation (which should be trivial).

> rather than hard-coding the algorithm selection in ubiquity, which is
> generally speaking a good recipe for bit rot.

Given that I'd like to see encryption land in 20.04, I think it would be reasonable to set -o encryption=aes-256-gcm today and then change it (e.g. for 20.10) to "on" once the default changes in OpenZFS.

[Matthew Ahrens (matthew.ahrens)]

I agree with what Richard said above in comment #14, especially:

> The installer should prompt (with a checkbox) for whether the user wants encryption. It should default to off. If the user selects the checkbox, prompt them for a passphrase. Setup encryption using that passphrase.

[Garrett Fields (fields-g)]

> It is not appropriate to require the user to type a password on every
> boot by default; this must be opt-in.

My suggestion was not to make EVERYONE use encrypted ZFS with a passphrase, only those who selected an encryption option within Ubiquity. (Perhaps my code was misunderstood as a implementation suggestion instead of a proof of concept for developers.) Having users opt for encryption and password would leave ZFS in parity with other filesystems installed within Ubiquity. I understand the original idea is to transparently force encryption on everyone but not bother them. Not only is there a performance issue, but you could have some really annoyed people if they ever needed to livecd/mount that filesystem and discover the hidden encryption they have to figure out how to work around.

[Mike Gerdts (mgerdts)]

This is a very interesting idea, but care should be applied when combined with cloud (disk) images. In particular, I worry that if an encrypted disk image is distributed that the well-known passphrase could be used to get the master key from any pristine copy of the disk image. In effect, the master key becomes an open secret. Thus, any instance derived from this disk image would be fairly easy to read regardless of knowledge of the wrapping key.

[Richard Laager (rlaager)]

We discussed this at the January 7th OpenZFS Leadership meeting. The notes and video recording are now available.

The meeting notes are in the running document here (see page 2 right now, or search for this Launchpad bug number):
https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit

The video recording is here; the link starts you at 15:45 when we start discussing this:
https://youtu.be/x9-wua_mzt0?t=945

[Richard Laager (rlaager)]

The AES-GCM performance improvements patch has been merged to master. This also included the changes to make encryption=on mean aes-256-gcm:
https://github.com/zfsonlinux/zfs/commit/31b160f0a6c673c8f926233af2ed6d5354808393

[Colin Ian King (colin-king)]

Backporting that commit is rather non-trivial as it has some other change dependencies and it's quite a large change for the normal SRU process.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Colin Ian King (colin-king)]

FYI: The AES-GCM patches as referenced in comment #19 were applied to zfs 0.8.3-1ubuntu12.1 in 6 Jul 2020.