The passphrase used to encrypt or decrypt volumes was mangled prior to Newton

[Lee Yarwood]

Description
===========

tl;dr hex(x) previously stripped leading 0's from individual hex numbers while encoding the passphrase back to a hex string before use to encrypt/decrypt a luks volume.

Prior to Newton the following method was used to encode passphrases when attempting to use or create a luks volume :

    def _get_passphrase(self, key):
        """Convert raw key to string."""
        return ''.join(hex(x).replace('0x', '') for x in key)

https://github.com/openstack/nova/blob/82190bdd283dda37f7517fd9a268b5e55183f06c/nova/volume/encryptors/cryptsetup.py#L92-L94

This was replaced in Newton with the move to Castellan in the following change that altered both the decoding and encoding steps :

Replace key manager with Castellan
https://review.openstack.org/#/c/309614/

The original method used the built-in hex() call to convert individual unsigned ints back to hex. This would strip the leading 0 from each hex digit pair, altering the eventual passphrase used to encrypt or decrypt the volume.

For example, the following one liner represents both the initial decode step preformed by ConfKeyManager and the step above to encode the passphrase in the LuksEncryptor class :

>>> ''.join(hex(x).replace('0x', '') for x in array.array('B', '752523eb50c3bf2ba3ff639c250405805fd4e779894ef5360e15e081696a'.decode('hex')).tolist())
'752523eb50c3bf2ba3ff639c2545805fd4e779894ef536e15e081696a'

Original string: 752523eb50c3bf2ba3ff639c250405805fd4e779894ef5360e15e081696a
New string : 752523eb50c3bf2ba3ff639c25 4 5805fd4e779894ef536 e15e081696a

The returned string is missing various 0's that have been stripped by the hex() call :

>>> hex(14)
'0xe'

>>> int(0x0e)
14

>>> int(0xe)
14

>>> hex(4)
'0x4'

>>> int(0x04)
4

>>> int(0x4)
4

The following one liner represents the current decode and encode steps, producing the same string as is entered :

>>> import binascii
>>> binascii.hexlify(bytes(binascii.unhexlify('752523eb50c3bf2ba3ff639c250405805fd4e779894ef5360e15e081696a'))).decode('utf-8')
u'752523eb50c3bf2ba3ff639c250405805fd4e779894ef5360e15e081696a'

Original string: 752523eb50c3bf2ba3ff639c250405805fd4e779894ef5360e15e081696a
New string : 752523eb50c3bf2ba3ff639c250405805fd4e779894ef5360e15e081696a

IMHO the way to handle this is to add a simple retry in master and stable/newton when we fail due to a bad passphrase using the mangled passphrase.

We should also improve the testing in this area as it appears all previous testing used zero based passphrases, missing this issue when it landed in Newton.

More notes available downstream in the following bug :

Nova encryption alters the key used
https://bugzilla.redhat.com/show_bug.cgi?id=1382415

Steps to reproduce
==================
- Encrypt a volume in Mitaka or earlier.
- Upgrade to Newton or later.
- Attempt to use the volume.

Expected result
===============
Volume is decrypted and usable.

Actual result
=============
Unable to decrypt the volume due to the use of an modified passphrase during initial formatting and use prior to Newton.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/
   Newton and later.

2. Which hypervisor did you use?
   Libvirt

2. Which storage type did you use?
   N/A

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)
   N/A

Logs & Configs
==============
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/386670

[Lee Yarwood (lyarwood)]

This also impacts os-brick that has recently copied the encryptor codebase with change :

Copy encryptors from Nova to os-brick
https://review.openstack.org/#/c/247372/

We should really try to remove the Nova encryptors this cycle I guess....

[Lee Yarwood (lyarwood)]

I've been asked to document a manual attempt at verifying the current workaround for LuksEncryptor :

- Using a fixed key of 010203040506 (123456 when mangled) :

$ grep fixed_key ../logs/n-cpu.log
2016-11-10 08:22:44.334 DEBUG oslo_service.service [req-09c9dd4c-ca17-48fb-99b7-21211241e84e None None] key_manager.fixed_key = 010203040506 from (pid=15466) log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2689

- Create a LUKS volume-type and create a single bootable volume :

$ cinder type-create LUKS
$ cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
$ cinder create --volume-type LUKS 1
$ cinder set-bootable c93a9d66-08d7-4ad0-babb-87a2c6fe5cf6 true

- Boot from the volume to ensure it is formatted by Nova :

$ nova boot --boot-volume c93a9d66-08d7-4ad0-babb-87a2c6fe5cf6 --flavor 1 test
$ nova delete test

- Remove the correct passphrase and replace it with the mangled version used prior to Newton to trigger the workaround :

$ sudo cryptsetup luksAddKey /dev/mapper/stack--volumes--lvmdriver--1-volume--c93a9d66--08d7--4ad0--babb--87a2c6fe5cf6
Enter any existing passphrase: 010203040506
Enter new passphrase for key slot: 123456
Verify passphrase: 123456
$ sudo cryptsetup luksRemoveKey /dev/mapper/stack--volumes--lvmdriver--1-volume--c93a9d66--08d7--4ad0--babb--87a2c6fe5cf6
Enter passphrase to be deleted: 010203040506
$ sudo cryptsetup luksDump /dev/mapper/stack--volumes--lvmdriver--1-volume--c93a9d66--08d7--4ad0--babb--87a2c6fe5cf6
LUKS header information for /dev/mapper/stack--volumes--lvmdriver--1-volume--c93a9d66--08d7--4ad0--babb--87a2c6fe5cf6

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512
MK digest: 04 09 bc a9 cb ab fc 3f 65 b2 dd e5 a8 2d 32 b4 37 ec b9 80
MK salt: 39 13 31 5a ab 1e ce 63 5e 96 bb d4 26 7d 19 a4
                8c 6a 39 79 1a c5 60 3c 2f 16 a2 a4 36 cb 23 64
MK iterations: 100500
UUID: a3b047cf-4a6a-46bb-bf2c-c38ea7c16fc3

Key Slot 0: DISABLED
Key Slot 1: ENABLED
        Iterations: 872230
        Salt: ea c6 19 21 ed 45 ce ce 96 51 08 90 a6 b0 e2 7e
                                dc 5a 94 f9 c6 f8 d8 90 d2 38 79 fa 21 f0 b7 e0
        Key material offset: 512
        AF stripes: 4000
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

- Boot an instance again using the volume :

$ nova boot --boot-volume c93a9d66-08d7-4ad0-babb-87a2c6fe5cf6 --flavor 1 test
$ nova delete test

- Reviewing n-cpu.log we can see the WARNING marker highlighting that we are going to attempt to use a mangled passphrase :

n-cpu.log

5123 2016-11-10 08:36:10.666 WARNING nova.volume.encryptors.luks [req-ae0c9bce-54fd-4691-939d-d68808bc81bb admin admin] /dev/sdb is not usable with the current passphrase, attempting to use a mangled passphrase to open the volume.
[..]
5143 2016-11-10 08:36:28.919 DEBUG nova.volume.encryptors.luks [req-ae0c9bce-54fd-4691-939d-d68808bc81bb admin admin] /dev/sdb m...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/396217

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/386670
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2010c13c0a229531a1ec3c84fb9c05e7310cf824
Submitter: Jenkins
Branch: master

commit 2010c13c0a229531a1ec3c84fb9c05e7310cf824
Author: Lee Yarwood <email address hidden>
Date: Fri Oct 14 16:41:53 2016 +0100

    encryptors: Workaround mangled passphrases

    Prior to Ib563b0ea the passphrase used by CryptsetupEncryptor and
    LuksEncryptor had any leading zeros per hexadecimal digit removed, for
    example 0x04 or 04 would turn into 0x4 or 4. As a result any volume
    encrypted prior to the release of Newton used a modified passphrase that
    was different to that stored by the key manager being used in the
    environment.

    To correct this for LuksEncryptor volumes permission denied errors are
    now caught when attempting to open a volume. A second attempt to open
    the volume is then made using a mangled passphrase. If successful the
    correct passphrase is then added to the volume before the mangled
    passphrase is finally removed. This workaround can be removed in a
    future release once it is safe to assume that all LuksEncryptor volumes
    have had any mangled passphrases replaced in this way.

    This isn't possible for CryptsetupEncryptor volumes as the plain mode
    used by cryptsetup does not provide a way for adding and removing keys.
    As such on a permission denied error a second attempt is made to open
    the volume using a mangled passphrase. Unlike the above workaround this
    cannot be removed in a future release.

    Change-Id: I7096463c5eba951dd6322ee6965435e877ca0371
    Partial-bug: #1633518

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/396288

[Lee Yarwood (lyarwood)]

The following change has also been pushed for os-brick to correct this :

encryptors: Workaround mangled passphrases
https://review.openstack.org/#/c/397934/

As with the Nova change I've now manually verified this. To do so I also had to use the following change moving Nova to use the os-brick supplied encryptors :

DNM - encryptors: Switch to os-brick encryptor classes
https://review.openstack.org/#/c/391597/

$ cinder type-create LUKS
$ cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
$ cinder create --volume-type LUKS 1
$ cinder set-bootable b9567416-f463-44ec-be70-c25977124614 true
$ nova boot --boot-volume b9567416-f463-44ec-be70-c25977124614 --flavor 1 test
$ nova delete test
$ sudo cryptsetup luksAddKey /dev/mapper/stack--volumes--lvmdriver--1-volume--b9567416--f463--44ec--be70--c25977124614
Enter any existing passphrase: 010203040506
Enter new passphrase for key slot: 123456
Verify passphrase: 123456
$ sudo cryptsetup luksRemoveKey /dev/mapper/stack--volumes--lvmdriver--1-volume--b9567416--f463--44ec--be70--c25977124614
Enter passphrase to be deleted: 010203040506
$ sudo cryptsetup luksDump /dev/mapper/stack--volumes--lvmdriver--1-volume--b9567416--f463--44ec--be70--c25977124614
LUKS header information for /dev/mapper/stack--volumes--lvmdriver--1-volume--b9567416--f463--44ec--be70--c25977124614

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512
MK digest: e1 48 9b 16 0f 2c 12 b2 e3 0f 59 5b a0 69 cd 2b 77 a8 e5 06
MK salt: ad b8 a8 ad ac c1 99 d8 55 59 7f 16 77 e1 1d 80
                0f 14 81 e7 59 ed 39 f0 ce 44 ae f8 11 9a 1f c2
MK iterations: 148250
UUID: 2d841497-ce78-4a67-a778-efeb7d6e17ad

Key Slot 0: DISABLED
Key Slot 1: ENABLED
        Iterations: 860503
        Salt: 1b a2 86 6a db 7a 99 7d d0 cc 73 bd fe c1 57 04
                                22 ec 25 79 18 22 12 68 6a 08 03 22 76 69 18 c2
        Key material offset: 512
        AF stripes: 4000
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

$ nova boot --boot-volume b9567416-f463-44ec-be70-c25977124614 --flavor 1 test
$ nova delete test
$ sudo cryptsetup luksOpen /dev/mapper/stack--volumes--lvmdriver--1-volume--b9567416--f463--44ec--be70--c25977124614 test
Enter passphrase for /dev/mapper/stack--volumes--lvmdriver--1-volume--b9567416--f463--44ec--be70--c25977124614: 010203040506
$ sudo cryptsetup luksClose test
$ sudo cryptsetup luksDump /dev/mapper/stack--volumes--lvmdriver--1-volume--b9567416--f463--44ec--be70--c25977124614
LUKS header information for /dev/mapper/stack--volumes--lvmdriver--1-volume--b9567416--f463--44ec--be70--c25977124614

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512
MK digest: e1 48 9b 16 0f 2c 12 b2 e3 0f 59...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/396288
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d0358255f943adc77eb4030f4632cd29f6d04803
Submitter: Jenkins
Branch: stable/newton

commit d0358255f943adc77eb4030f4632cd29f6d04803
Author: Lee Yarwood <email address hidden>
Date: Fri Oct 14 16:41:53 2016 +0100

    encryptors: Workaround mangled passphrases

    Prior to Ib563b0ea the passphrase used by CryptsetupEncryptor and
    LuksEncryptor had any leading zeros per hexadecimal digit removed, for
    example 0x04 or 04 would turn into 0x4 or 4. As a result any volume
    encrypted prior to the release of Newton used a modified passphrase that
    was different to that stored by the key manager being used in the
    environment.

    To correct this for LuksEncryptor volumes permission denied errors are
    now caught when attempting to open a volume. A second attempt to open
    the volume is then made using a mangled passphrase. If successful the
    correct passphrase is then added to the volume before the mangled
    passphrase is finally removed. This workaround can be removed in a
    future release once it is safe to assume that all LuksEncryptor volumes
    have had any mangled passphrases replaced in this way.

    This isn't possible for CryptsetupEncryptor volumes as the plain mode
    used by cryptsetup does not provide a way for adding and removing keys.
    As such on a permission denied error a second attempt is made to open
    the volume using a mangled passphrase. Unlike the above workaround this
    cannot be removed in a future release.

    Change-Id: I7096463c5eba951dd6322ee6965435e877ca0371
    Partial-bug: #1633518
    (cherry picked from commit 2010c13c0a229531a1ec3c84fb9c05e7310cf824)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/397934
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=7e33521a39d5b7478008e5d6ba4c754857edbee9
Submitter: Jenkins
Branch: master

commit 7e33521a39d5b7478008e5d6ba4c754857edbee9
Author: Lee Yarwood <email address hidden>
Date: Mon Nov 14 14:29:17 2016 +0000

    encryptors: Workaround mangled passphrases

    Prior to Ib563b0ea the passphrase used by CryptsetupEncryptor and
    LuksEncryptor had any leading zeros per hexadecimal digit removed, for
    example 0x04 or 04 would turn into 0x4 or 4. As a result any volume
    encrypted prior to the release of Newton used a modified passphrase that
    was different to that stored by the key manager being used in the
    environment.

    To correct this for LuksEncryptor volumes permission denied errors are
    now caught when attempting to open a volume. A second attempt to open
    the volume is then made using a mangled passphrase. If successful the
    correct passphrase is then added to the volume before the mangled
    passphrase is finally removed. This workaround can be removed in a
    future release once it is safe to assume that all LuksEncryptor volumes
    have had any mangled passphrases replaced in this way.

    This isn't possible for CryptsetupEncryptor volumes as the plain mode
    used by cryptsetup does not provide a way for adding and removing keys.
    As such on a permission denied error a second attempt is made to open
    the volume using a mangled passphrase. Unlike the above workaround this
    cannot be removed in a future release.

    Change-Id: I7096463c5eba951dd6322ee6965435e877ca0371
    Partial-bug: #1633518

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/406068

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/396217
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7d77fe09e86ef286ab3419282dc7afbb77d5ffd2
Submitter: Jenkins
Branch: master

commit 7d77fe09e86ef286ab3419282dc7afbb77d5ffd2
Author: Lee Yarwood <email address hidden>
Date: Thu Nov 10 14:03:07 2016 +0000

    Add a releasenote for bug#1633518

    The code relating to this patch can be seen in:
    I7096463c5eba951dd6322ee6965435e877ca0371

    Partial-bug: #1633518
    Change-Id: I297e138aca52ae10fbfaa50c79cf001fadaf557e

[Maciej Szankin (mszankin)]

lyarwood - AFAIK we do not use ``Fix COmmited`` status, just ``Fix Released``.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/406068
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=8c401a68bbe52685cbe9122723ff6b7dc921706f
Submitter: Jenkins
Branch: stable/newton

commit 8c401a68bbe52685cbe9122723ff6b7dc921706f
Author: Lee Yarwood <email address hidden>
Date: Mon Nov 14 14:29:17 2016 +0000

    encryptors: Workaround mangled passphrases

    Prior to Ib563b0ea the passphrase used by CryptsetupEncryptor and
    LuksEncryptor had any leading zeros per hexadecimal digit removed, for
    example 0x04 or 04 would turn into 0x4 or 4. As a result any volume
    encrypted prior to the release of Newton used a modified passphrase that
    was different to that stored by the key manager being used in the
    environment.

    To correct this for LuksEncryptor volumes permission denied errors are
    now caught when attempting to open a volume. A second attempt to open
    the volume is then made using a mangled passphrase. If successful the
    correct passphrase is then added to the volume before the mangled
    passphrase is finally removed. This workaround can be removed in a
    future release once it is safe to assume that all LuksEncryptor volumes
    have had any mangled passphrases replaced in this way.

    This isn't possible for CryptsetupEncryptor volumes as the plain mode
    used by cryptsetup does not provide a way for adding and removing keys.
    As such on a permission denied error a second attempt is made to open
    the volume using a mangled passphrase. Unlike the above workaround this
    cannot be removed in a future release.

    Change-Id: I7096463c5eba951dd6322ee6965435e877ca0371
    Partial-bug: #1633518
    (cherry picked from commit 7e33521a39d5b7478008e5d6ba4c754857edbee9)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/462674

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/791270
Committed: https://opendev.org/openstack/os-brick/commit/4c654d94540dfd0469e703b205d44b92bd2482ad
Submitter: "Zuul (22348)"
Branch: master

commit 4c654d94540dfd0469e703b205d44b92bd2482ad
Author: Stephen Finucane <email address hidden>
Date: Fri May 14 10:13:31 2021 +0100

    encryptors: Remove workaround for bug #1633518

    This has been in place since Newton and was targeted for removal in
    Rocky. That has long since passed but we can ease things by removing it
    now.

    Despite what change I7096463c5eba951dd6322ee6965435e877ca0371 said about
    not removing this workaround from 'CryptsetupEncryptor', we do so anyway
    since we're going to remove this encryptor shortly.

    Change-Id: Id8d96c1eda2beab1aebd8ecc054d8511ccefca3e
    Signed-off-by: Stephen Finucane <email address hidden>
    Related-bug: #1633518