On-disk encryption is borken with Python 3

Bug #1847755 reported by Thomas Goirand
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
High
Unassigned

Bug Description

When activating keymaster encryption, we get this if using Python 3:

Oct 11 11:46:00 s1-swiftproxy-3 proxy-server: get_keys(): from callback: key: expected bytes or bytearray, but got 'str':
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swift/common/middleware/crypto/crypto_utils.py", line 170, in get_keys
    keys = fetch_crypto_keys(key_id=key_id)
  File "/usr/lib/python3/dist-packages/swift/common/middleware/crypto/keymaster.py", line 122, in fetch_crypto_keys
    path, secret_id=secret_id)
  File "/usr/lib/python3/dist-packages/swift/common/middleware/crypto/keymaster.py", line 296, in create_key
    digestmod=hashlib.sha256).digest()
  File "/usr/lib/python3.7/hmac.py", line 153, in new
    return HMAC(key, msg, digestmod)
  File "/usr/lib/python3.7/hmac.py", line 49, in __init__
    raise TypeError("key: expected bytes or bytearray, but got %r" % type(key).__name__)
TypeError: key: expected bytes or bytearray, but got 'str' (txn: tx2b59918356794c709b34b-005da06b77)

The fix is trivial:

- return hmac.new( key, wsgi_to_bytes(path),
+ return hmac.new( bytes(key, 'latin-1'), wsgi_to_bytes(path),

after this, encryption just works. Let me open the PR...

Revision history for this message
Tim Burke (1-tim-z) wrote :

From IRC, this was the Barbican keymaster, which explains why we hadn't caught this earlier.

I finally got an instance up and running and... it just worked. I wonder if maybe it's because I requested a payload-content-type of application/octet-stream when I ran `openstack secret order create`. What were you using? Should be able to find out with `openstack secret get` -- on my machine, that gives me something like

+---------------+----------------------------------------------------------------------+
| Field | Value |
+---------------+----------------------------------------------------------------------+
| Secret href | http://hostname:9311/v1/secrets/de423521-8a7b-491c-be39-74f040e30827 |
| Name | swift_root_secret |
| Created | 2019-10-11T23:27:12.985712+00:00 |
| Status | ACTIVE |
| Content types | {'default': 'application/octet-stream'} |
| Algorithm | aes |
| Bit length | 256 |
| Secret type | symmetric |
| Mode | ctr |
| Expiration | None |
+---------------+----------------------------------------------------------------------+

And for completeness, I've got a handful of follow-up questions:
- Is this on upload, download, container listing? Probably, all three.
- What version of python-barbicanclient is installed? What version of castellan?

Revision history for this message
Thomas Goirand (thomas-goirand) wrote :

Hi,

I'm using this:

openstack secret store --name swift-encryption-key \
  --payload-content-type=text/plain --algorithm aes \
  --bit-length 256 --mode ctr --secret-type symmetric \
  --payload $(openssl rand -hex 32)

Can you try the same way?

This happens when doing:

openstack object create

Container creation seems ok, listing probably too as well.

Thomas

Revision history for this message
Tim Burke (1-tim-z) wrote :
Download full text (3.8 KiB)

Oh, interesting! Rather than generating the secret locally, I was requesting that barbican generate one with

  openstack secret order create --name swift_root_secret \
   --payload-content-type="application/octet-stream" \
   --algorithm aes --bit-length 256 --mode ctr key

When I tried a --payload-content-type=text/plain, starting the proxy would bomb out with

  Key manager error: Not Acceptable: Secret payload retrieval
  issue seen - Wrong payload content-type.

Server-side, barbican had some error logging:

  Traceback (most recent call last):
    File ".../barbican/plugin/util/translations.py", line 106, in denormalize_after_decryption
      unencrypted = base64.decode_as_text(unencrypted)
    File ".../oslo_serialization/base64.py", line 92, in decode_as_text
      return decoded.decode(encoding)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 3: invalid continuation byte

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File ".../barbican/api/controllers/__init__.py", line 101, in handler
      return fn(inst, *args, **kwargs)
    File ".../barbican/api/controllers/__init__.py", line 87, in enforcer
      return fn(inst, *args, **kwargs)
    File ".../barbican/api/controllers/secrets.py", line 207, in payload
      **kwargs)
    File ".../barbican/api/controllers/secrets.py", line 185, in _on_get_secret_payload
      transport_key)
    File ".../barbican/plugin/resources.py", line 147, in get_secret
      requesting_content_type)
    File ".../barbican/plugin/util/translations.py", line 108, in denormalize_after_decryption
      raise s.SecretAcceptNotSupportedException(content_type)
  barbican.plugin.interface.secret_store.SecretAcceptNotSupportedException: Secret Accept of 'text/plain' not supported

Using your method, I can repro just fine. It also gave me some fun ideas, like not using hex but instead declaring the raw bytes to be latin-1. Nope, no dice:

  $ openstack secret store --payload "$(openssl rand 32)" \
    --name 'rand-no-hex' --payload-content-type \
    text/plain;charset=latin-1
  5xx Server error: Internal Server Error: Secret creation
  failure seen - please contact site administrator.
  Internal Server Error: Secret creation failure seen -
  please contact site administrator.

Barbican logs said

  Traceback (most recent call last):
    File ".../barbican/api/controllers/__init__.py", line 101, in handler
      return fn(inst, *args, **kwargs)
    File ".../barbican/api/controllers/__init__.py", line 87, in enforcer
      return fn(inst, *args, **kwargs)
    File ".../barbican/api/controllers/__init__.py", line 149, in content_types_enforcer
      return fn(inst, *args, **kwargs)
    File ".../barbican/api/controllers/secrets.py", line 424, in on_post
      data = api.load_body(pecan.request, validator=self.validator)
    File ".../barbican/api/__init__.py", line 70, in load_body
      parsed_body = validator.validate(parsed_body)
    File ".../barbican/common/validators.py", line 210, in validate
      payload = self._extract_payload(json_data)
    File ".../barbican/common/validators.py", line 319, in _extract_payload
      i...

Read more...

Changed in swift:
status: New → Confirmed
importance: Undecided → High
tags: added: python3
Revision history for this message
Thomas Goirand (thomas-goirand) wrote :

Hi Tim,

Thanks for taking this seriously, and I'm happy that you've been able to reproduce the issue too.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.opendev.org/688113
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=85d3658d6297efbe9a588d1b4364a2a68f9e5bb7
Submitter: Zuul
Branch: master

commit 85d3658d6297efbe9a588d1b4364a2a68f9e5bb7
Author: Thomas Goirand <email address hidden>
Date: Fri Oct 11 14:12:01 2019 +0200

    Fix kms_keymaster under Python 3

    Depending on how the key was stored in Barbican, it may come out of
    Castellan as a native string, which would not be suitable on Python 3.
    Now, check that the secret is a byte string, and if it isn't, encode as
    UTF-8 (to match Barbican's internal encoding).

    Change-Id: I6da047716c05e4f2a9e1e74ca19afb62e812d172
    Closes-Bug: #1847755

Changed in swift:
status: Confirmed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/688767

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (stable/train)

Reviewed: https://review.opendev.org/688767
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=dd51658949ca937abd9d87a5184704cd48135682
Submitter: Zuul
Branch: stable/train

commit dd51658949ca937abd9d87a5184704cd48135682
Author: Thomas Goirand <email address hidden>
Date: Fri Oct 11 14:12:01 2019 +0200

    Fix kms_keymaster under Python 3

    Depending on how the key was stored in Barbican, it may come out of
    Castellan as a native string, which would not be suitable on Python 3.
    Now, check that the secret is a byte string, and if it isn't, encode as
    UTF-8 (to match Barbican's internal encoding).

    Change-Id: I6da047716c05e4f2a9e1e74ca19afb62e812d172
    Closes-Bug: #1847755
    (cherry picked from commit 85d3658d6297efbe9a588d1b4364a2a68f9e5bb7)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.23.1

This issue was fixed in the openstack/swift 2.23.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/losf)

Fix proposed to branch: feature/losf
Review: https://review.opendev.org/693283

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/losf)
Download full text (9.6 KiB)

Reviewed: https://review.opendev.org/693283
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=dbfb6a6fc90cb3367ed2248572089ee3cbafc15a
Submitter: Zuul
Branch: feature/losf

commit 7cb92810acfd3cd598860aefadff8fba47e3efdd
Author: SeongSoo Cho <email address hidden>
Date: Wed Nov 6 15:32:56 2019 +0800

    Fix bug internal client's delete_object logs 499

    Internal Client's doest not read body for 404.
    So, When internal deleing object that not existed, proxy-server logs
    its response code to 499.

    In this patch, add a code that drain the reponse body to prevent
    unexpected disconnect in proxy-server

    Change-Id: I6d170fead798d0d539c69d27a6fa8a2d0123ca99
    Closes-Bug: #1835324

commit c512574e110c214396658743e75ae497ba76e2ff
Author: Tim Burke <email address hidden>
Date: Tue Nov 5 01:45:26 2019 +0000

    Make our py2 func tests more explicitly py2

    ...because otherwise, I think they might be running under py3 right now.

    Change-Id: I96ad9e5ed87a7e08d06e889233730cec476d3af7

commit a505d02b3762ecbe7ea80b968ddec90ca3446549
Author: Chris Smart <email address hidden>
Date: Sat Nov 2 20:48:47 2019 +1100

    Tweak Swift AIO docs, update CentOS and Fedora

    The Swift AIO documentation is out of date for CentOS and Fedora.

    This patch updates the documentation to ensure that the instructions are
    clear and accurate.

    It also cleans up a few other sections along the way and adds some new
    headings to make it easier to read. Some explanatory notes are added,
    such as the need for XFS storage and test directories. XFS tmp loopback
    device is moved out of the user's home directory to /srv to match
    storage.

    Change-Id: Ieb0341536b7149c99139a1cf620828eba25f4bc6
    Signed-off-by: Chris Smart <email address hidden>

commit 281ffab2a52f348aa3f8ad39e800a43c6047e1cf
Author: Thiago da Silva <email address hidden>
Date: Fri Nov 1 09:57:34 2019 +0200

    pin lower-constraints to run against python 2.7

    Change-Id: I5c856012a04ef8f910c28b0248ea7bcda47e4816

commit 0c1b485ad671fc0abac15cd0bbec894e517e1dd9
Author: John Dickinson <email address hidden>
Date: Thu Oct 17 09:43:55 2019 -0700

    exclude utf8 tests under py3

    These are known to not work until https://bugs.python.org/issue37093
    is addressed in CPython upstream.

    Change-Id: I4a6877907d14b632a9a477c887913488427b62b7

commit dd235b6ac0757e34805b041d331511c957764f3e
Author: Clay Gerrard <email address hidden>
Date: Mon Oct 28 11:27:58 2019 -0500

    use request_helpers import namespace for tests

    Change-Id: I8e18ff2036312d3049a420f3332a6ea6110447c4

commit b75d5939352c405ab82ab52c3a9a1c3ad030aa26
Author: Clay Gerrard <email address hidden>
Date: Fri Oct 25 14:34:41 2019 -0500

    Refactor timestamp iter in db backend tests

    Change-Id: I61d044a6d778e158af73a3f4d9fc75d642af0beb

commit 739c3704107bd991493472c18bfe7ac5f933077c
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Oct 18 07:10:33 2019 +0000

    Imported Translations from Zanata

    For more information about this automat...

Read more...

tags: added: in-feature-losf
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.24.0

This issue was fixed in the openstack/swift 2.24.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.