[SRU] Fix inconsistent encoding secret encoding

[Corey Bryant]

[Impact]
This SRU corresponds with the following story for upstream barbican
https://storyboard.openstack.org/#!/story/2008335.

The problem is some secrets were stored in plaintext and some were stored encoded. This resulted in the inability to decode some secrets.

This is fixed by always storing secrets in plaintext and decoding inconsistently stored data as needed when getting secrets.

[Test Case]
  * deploy Openstack with Barbican using Vault as a backend
  * openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LUKS
  * openstack volume create --size 1 --type LUKS luks_vol1
  * ensure volume created successfully
  * openstack volume show luks_vol1
  * create vm and attach volume
  * mkfs and mount then test can read/write

[Where things could go wrong]
If things were to go wrong it would probably be in the get_secret() method which calls _ensure_legacy_base64(). _ensure_legacy_base64() assumes that anything that is not a key was stored base64 encoded. Presumably this is correct, but there was a path added to catch a UnicodeDecodeError exception to handle unexpected non-base64-encoded secrets.

[Corey Bryant (corey.bryant)]

A new version of barbican with this fix has been uploaded to the hirsute unapproved queue:
https://launchpad.net/ubuntu/hirsute/+queue?queue_state=1&queue_text=barbican

[Brian Murray (brian-murray)]

Hello Corey, or anyone else affected,

Accepted barbican into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/barbican/2:12.0.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Corey, or anyone else affected,

Accepted barbican into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Liam Young (gnuoy)]

Tested successfully on hirsute using 2:12.0.0-0ubuntu2 . I created an encrypted volume and attached it to a VM.

cinder type-create LUKS
cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
cinder create --volume-type LUKS --poll --name testvol 1
openstack keypair show guests || openstack keypair create --public-key ~/.ssh/id_rsa_guests.pub guests
openstack flavor create --id 8 --ram 1024 --disk 8 --vcpus 1 --public m1.ly
openstack server create --image bionic --flavor m1.ly --network private --key-name guests --wait test3
openstack floating ip create ext_net
openstack server add floating ip test3 172.20.0.207
openstack server add volume --device /dev/vdb test3 testvol
cinder list
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+
| 67564b48-54b7-47bf-ac95-d701b455cb7d | in-use | testvol | 1 | LUKS | false | 6c43fed1-a195-47d8-b5a9-dc7fd166bf58 |
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+

cinder show testvol
+--------------------------------+------------------------------------------+
| Property | Value |
+--------------------------------+------------------------------------------+
| attached_servers | ['6c43fed1-a195-47d8-b5a9-dc7fd166bf58'] |
| attachment_ids | ['f0c3ed24-2973-407a-b6f6-afcef999ed43'] |
| availability_zone | nova |
| bootable | false |
| cluster_name | None |
| consistencygroup_id | None |
| created_at | 2021-11-01T16:38:32.000000 |
| description | None |
| encrypted | True |
| encryption_key_id | c6079e38-fe86-4e16-aee0-09d07fdfc719 |
| group_id | None |
| id | 67564b48-54b7-47bf-ac95-d701b455cb7d |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | testvol |
| os-vol-host-attr:host | juju-86a900-zaza-c440171f601b-11@LVM#LVM |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:te...

[Liam Young (gnuoy)]

Tested successfully on focal wallaby using 2:12.0.0-0ubuntu2~cloud0 . I created an encrypted volume and attached it to a VM.

cinder type-create LUKS
cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
cinder create --volume-type LUKS --poll --name testvol 1
openstack keypair show guests || openstack keypair create --public-key ~/.ssh/id_rsa_guests.pub guests
openstack flavor create --id 8 --ram 1024 --disk 8 --vcpus 1 --public m1.ly
openstack server create --image bionic --flavor m1.ly --network private --key-name guests --wait test3
openstack floating ip create ext_net
openstack server add floating ip test3 172.20.0.207
openstack server add volume --device /dev/vdb test3 testvol
cinder list
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+
| ebf6c7d9-aac4-440e-b29f-c4ddd6a3e544 | in-use | testvol | 1 | LUKS | false | 6c47befa-4b32-4d87-9a03-c23e26ed9255 |
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+

cinder show testvol

+--------------------------------+------------------------------------------+
| Property | Value |
+--------------------------------+------------------------------------------+
| attached_servers | ['6c47befa-4b32-4d87-9a03-c23e26ed9255'] |
| attachment_ids | ['c6653494-c23e-4312-a441-f86eba08794f'] |
| availability_zone | nova |
| bootable | false |
| cluster_name | None |
| consistencygroup_id | None |
| created_at | 2021-11-01T18:15:41.000000 |
| description | None |
| encrypted | True |
| encryption_key_id | dde779f5-ad06-45e8-979c-37dd3cea8505 |
| group_id | None |
| id | ebf6c7d9-aac4-440e-b29f-c4ddd6a3e544 |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | testvol |
| os-vol-host-attr:host | juju-9ce866-zaza-17f25c1dd768-11@LVM#LVM |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-t...

[Launchpad Janitor (janitor)]

This bug was fixed in the package barbican - 2:12.0.0-0ubuntu2

---------------
barbican (2:12.0.0-0ubuntu2) hirsute; urgency=medium

  * d/gbp.conf: Create stable/wallaby branch.
  * d/p/fix-castellan-secret-store-encoding.patch: Fix inconsistent encoding
    of SecretDTO objects (LP: #1946787).

 -- Corey Bryant <email address hidden> Tue, 12 Oct 2021 14:17:00 -0400

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for barbican has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for barbican has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package barbican - 2:12.0.0-0ubuntu2~cloud0
---------------

 barbican (2:12.0.0-0ubuntu2~cloud0) focal-wallaby; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 barbican (2:12.0.0-0ubuntu2) hirsute; urgency=medium
 .
   * d/gbp.conf: Create stable/wallaby branch.
   * d/p/fix-castellan-secret-store-encoding.patch: Fix inconsistent encoding
     of SecretDTO objects (LP: #1946787).

[Corey Bryant (corey.bryant)]

Hello Corey, or anyone else affected,

Accepted barbican into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

I've uploaded a new version of barbican with this fix to the bionic unapproved queue:
https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=barbican

[Liam Young (gnuoy)]

Tested successfully on focal victoria using 1:11.0.0-0ubuntu1~cloud1 . I created an encrypted volume and attached it to a VM.

cinder type-create LUKS
cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
cinder create --volume-type LUKS --poll --name testvol 1
openstack keypair show guests || openstack keypair create --public-key ~/.ssh/id_rsa_guests.pub guests
openstack flavor create --id 8 --ram 1024 --disk 8 --vcpus 1 --public m1.ly
openstack server create --image bionic --flavor m1.ly --network private --key-name guests --wait test3
openstack floating ip create ext_net
openstack server add floating ip test3 172.20.0.235
openstack server add volume --device /dev/vdb test3 testvol

cinder list
WARNING:cinderclient.shell:API version 3.64 requested,
WARNING:cinderclient.shell:downgrading to 3.62 based on server support.
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+
| 7ea1296e-a478-4aea-ade0-49f00034b58b | in-use | testvol | 1 | LUKS | false | e1b2c025-0ede-4330-9129-80f6c281ac4d |
+--------------------------------------+--------+---------+------+-------------+----------+--------------------------------------+

cinder show 7ea1296e-a478-4aea-ade0-49f00034b58b
WARNING:cinderclient.shell:API version 3.64 requested,
WARNING:cinderclient.shell:downgrading to 3.62 based on server support.
+--------------------------------+------------------------------------------+
| Property | Value |
+--------------------------------+------------------------------------------+
| attached_servers | ['e1b2c025-0ede-4330-9129-80f6c281ac4d'] |
| attachment_ids | ['c4410464-ff27-4234-9f5f-c5a7b094463b'] |
| availability_zone | nova |
| bootable | false |
| cluster_name | None |
| consistencygroup_id | None |
| created_at | 2021-11-02T11:23:28.000000 |
| description | None |
| encrypted | True |
| group_id | None |
| id | 7ea1296e-a478-4aea-ade0-49f00034b58b |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | testvol |
| os-vol-host-attr:host | ju...

[Steve Langasek (vorlon)]

Hello Corey, or anyone else affected,

Accepted barbican into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/barbican/1:10.1.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for barbican has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package barbican - 1:11.0.0-0ubuntu1~cloud1
---------------

 barbican (1:11.0.0-0ubuntu1~cloud1) focal-victoria; urgency=medium
 .
   * d/gbp.conf: Create stable/victoria branch.
   * d/p/fix-castellan-secret-store-encoding.patch: Fix inconsistent encoding
     of SecretDTO objects (LP: #1946787).

[Corey Bryant (corey.bryant)]

Hello Corey, or anyone else affected,

Accepted barbican into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Edward Hope-Morley (hopem)]

focal-ussuri-proposed verified using [Test Case] and output is:

# apt-cache policy python3-barbican
python3-barbican:
  Installed: 1:10.1.0-0ubuntu2
  Candidate: 1:10.1.0-0ubuntu2
  Version table:
 *** 1:10.1.0-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:10.1.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     1:10.0.0~b2~git2020020508.7b14d983-0ubuntu3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages

# virsh dumpxml instance-00000001| grep -A 10 "device='disk'"| grep encryption
      <encryption format='luks'>

$ sudo mkfs.ext4 /dev/vdb
$ sudo mount /dev/vdb /mnt/
$ echo "I'm feeling luksy"| sudo tee /mnt/secure
I'm feeling luksy
$ cat /mnt/secure
I'm feeling luksy

[Launchpad Janitor (janitor)]

This bug was fixed in the package barbican - 1:10.1.0-0ubuntu2

---------------
barbican (1:10.1.0-0ubuntu2) focal; urgency=medium

  * d/p/fix-castellan-secret-store-encoding.patch: Fix inconsistent encoding
    of SecretDTO objects (LP: #1946787).

 -- Corey Bryant <email address hidden> Mon, 01 Nov 2021 14:09:38 -0400

[Edward Hope-Morley (hopem)]

bionic-ussuri-proposed verified using [Test Case] and output is:

# apt-cache policy python3-barbican
python3-barbican:
  Installed: 1:10.1.0-0ubuntu2~cloud0
  Candidate: 1:10.1.0-0ubuntu2~cloud0
  Version table:
 *** 1:10.1.0-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status

# virsh dumpxml instance-00000001| grep -A 10 "device='disk'"| grep encryption
      <encryption format='luks'>

$ sudo mkfs.ext4 /dev/vdb
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: ac572bfb-074f-485c-8c3f-e2c97cb51d12
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

$ sudo mount /dev/vdb /mnt/
$ echo "I'm feeling luksy"| sudo tee /mnt/secure
I'm feeling luksy
$ cat /mnt/secure
I'm feeling luksy

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for barbican has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package barbican - 1:10.1.0-0ubuntu2~cloud0
---------------

 barbican (1:10.1.0-0ubuntu2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 barbican (1:10.1.0-0ubuntu2) focal; urgency=medium
 .
   * d/p/fix-castellan-secret-store-encoding.patch: Fix inconsistent encoding
     of SecretDTO objects (LP: #1946787).