Object encryption key truncated sometimes when used with Barbican

[Simeon Gourlin]

Hello,

We use in production a Swift cluster with encryption activated, using Barbican keymaster.
We found that sometimes object encryption key is truncated, object part in path (a/c/o) is not complete.

Example, for encrypted object in object-store :
$ curl -g -I -XHEAD "http://1.2.3.4:6209/sdk/112738/AUTH_abcdef/default/Infomaniak%20Swiss%20Backup%20Todo.txt"
X-Object-Sysmeta-Crypto-Body-Meta: %7B%22body_key%22%3A+%7B%22iv%22%3A+%22qSSUcWpajm6I2AvFCsEnjA%3D%3D%22%2C+%22key%22%3A+%22OdQxb9oIoPlw1OlY6YzU3L%2BfS9YQWy%2BkhQkzNRo7yYk%3D%22%7D%2C+%22cipher%22%3A+%22AES_CTR_256%22%2C+%22iv%22%3A+%22jMwFlUytpkEorq4zBoew%2Fg%3D%3D%22%2C+%22key_id%22%3A+%7B%22path%22%3A+%22%2FAUTH_abcdef%2Fdefault%2FInfomaniak+Swiss+Backup+.txt%22%2C+%22v%22%3A+%221%22%7D%7D

urldecoded :
X-Object-Sysmeta-Crypto-Body-Meta: {"body_key": {"iv": "qSSUcWpajm6I2AvFCsEnjA==", "key": "OdQxb9oIoPlw1OlY6YzU3L+fS9YQWy+khQkzNRo7yYk="}, "cipher": "AES_CTR_256", "iv": "jMwFlUytpkEorq4zBoew/g==", "key_id": {"path": "/AUTH_abcdef/default/Infomaniak Swiss Backup .txt", "v": "1"}}

Object path is '/AUTH_abcdef/default/Infomaniak Swiss Backup Todo.txt' and stored object key in metadata is '/AUTH_abcdef/default/Infomaniak Swiss Backup .txt'
*** note the 'Todo' missing at the end of the path ***

As a consequence, the object is decoded wrongly, since object key is calculated from object path during decryption.

We wrote a workaround to decrypt such broken objects but we didn't figure out yet where the bug is.

[Tim Burke (1-tim-z)]

Can we get some more info from logs about the history of this object? I'd go grepping for both names, "AUTH_abcdef/default/Infomaniak Swiss Backup Todo.txt" and "/AUTH_abcdef/default/Infomaniak Swiss Backup .txt" (and probably *also* go grepping for those url-encoded, and even double-url-encoded :-/)

In particular, I'd be interested in seeing

- whether the object with the truncated name ever existed
- when the object with the full name got PUT, and
- whether the data for that PUT seemed to be coming from the client or from the cluster (for example, as part of a server-side copy)

What does your proxy-server pipeline look like? Is the copy middleware explicitly placed, or was it auto-inserted? If it was auto-inserted, there should be some log lines like

Jan 26 02:03:47 saio proxy-server: Adding required filter copy to pipeline at position 10
Jan 26 02:03:47 saio proxy-server: Adding required filter listing_formats to pipeline at position 4
Jan 26 02:03:47 saio proxy-server: Adding required filter gatekeeper to pipeline at position 1
Jan 26 02:03:47 saio proxy-server: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck proxy-logging cache listing_formats container_sync bulk tempurl s3api tempauth staticweb copy slo dlo versioned_writes symlink kmip_keymaster encryption proxy-logging proxy-server".

during proxy-server start-up -- I'd be very interested in what the new pipeline looks like. Makes me wonder if like in https://bugs.launchpad.net/swift/+bug/1758710 and https://bugs.launchpad.net/swift/+bug/1806468, our auto-insertion logic is falling down somewhere -- in particular, it would be problematic if we're auto-inserting copy *after* encryption.

That said, even if the auto-insertion is the problem, we have to be able to still read what's been written, so https://review.openstack.org/#/c/633671/ seems like a great idea.

[Tim Burke (1-tim-z)]

Oh yeah, and a couple other notes:

1. The Barbican keymaster shares enough code with the trivial keymaster and KMIP keymaster that they would surely also be affected.

2. It seems really *really* weird to me that the basename is truncated rather than the full name -- how did we lose the end of the name but not the extension??

[Tim Burke (1-tim-z)]

So I could definitely repro... first by manually inserting the copy middleware after encryption, then again by only having *one* proxy-logging middleware and letting auto-insertion do its thing (though at that point container listings were *also* broken, as I also let listing_formats get auto-inserted and it got inserted right of encryption as well, leading to a JSON decode error in symlink, just like in https://bugs.launchpad.net/swift/+bug/1806468).

[Matthew Oliver (matt-0)]

Sounds like we need to seriously think about rejigging how our auto-insert and maybe even pipeline management works.

[Tim Burke (1-tim-z)]

Yup. I wonder if a decent starting point may be to start raising errors in some middlewares if we see that some other (known-problematic) middleware has already registered in get_swift_info... like, encryption could check for copy and raise an exception during proxy-server start-up saying "hey, copy needs to be *left* of encryption".

[Simeon Gourlin (6meon)]

Hello Tim,

Thank you for investigations, and sorry for late answer.

We don't have debug logs enabled in production so it's difficult to provide more logs or to tell you how many objects are concerned by the bug. I saw a dozen of cases until now, then we disabled encryption and put the patch in production to be able to decrypt broken objects. We didn't reactivated encryption since.
I can provide full output from object server, or raw metadata on container and object itself, but if you already reproduced the bug maybe it's not useful now.

Our pipeline looks like this :
pipeline = catch_errors healthcheck kms_keymaster encryption proxy-logging cache container_sync bulk ratelimit authtoken keystone copy container-quotas account-quotas slo dlo versioned_writes proxy-server
So it's missing the second "proxy-logging" at the end if I understand correctly.

If correct, we'll add second proxy-logging entry on our pipeline and try to reactivate encryption ; but we'll need to keep our patch for decryption since we have broken objects in our cluster now.

Many thanks for your help !

[Tim Burke (1-tim-z)]

Ah! Yep, looks like a pipeline problem. When you add the second proxy-logging, you'll want to make sure you move encryption (and probably kms_keymaster) over near the new proxy-logging, so your pipeline looks like

   pipeline = ... versioned_writes kms_keymaster encryption proxy-logging proxy-server

I'd try to do that as soon as you can -- this should get you back to a better-tested configuration and stop further objects from getting into this state once you turn encryption on again. Further, I expect other features aren't working real well, eiither -- in particular, SLO (and possibly DLO).

Finally, I'm fairly certain it was a COPY (or PUT with X-Copy-From) request that caused the trouble for *this* object, but note that versioned_writes could also have caused some problems given your current configuration.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/635380

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/635380
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=43103319d0aa27f24e6520c0962bd19e55568ad4
Submitter: Zuul
Branch: master

commit 43103319d0aa27f24e6520c0962bd19e55568ad4
Author: Tim Burke <email address hidden>
Date: Wed Feb 6 16:48:17 2019 -0800

    encryption: Stop being cutesy with os.path.join()

    Turns out, we *care* about the path, and object paths *don't follow
    filesystem semantics*!

    Be explicit: /<account>/<container>/<object>

    Bump the key version number so we know whether we can trust the full path
    or not.

    Change-Id: Ide9d44cc18575306363126a93d91f662c6ee23e0
    Related-Bug: 1813725

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: feature/losf
Review: https://review.openstack.org/637142

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637142
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=b68bef5cd80e2a1b71a1ef544e122b39dfa7ac57
Submitter: Zuul
Branch: feature/losf

commit 926a024135d380999d9f8494b19b59bb87a7f5b6
Author: Tim Burke <email address hidden>
Date: Thu Feb 14 21:02:01 2019 +0000

    Fix up flakey TestContainer.test_PUT_bad_metadata

    Change-Id: I7489f2bb95c27d1ddd5e8fa7e5786904100fb567

commit 002d21991e100ee6199e79679ae990c96ea05730
Author: Tim Burke <email address hidden>
Date: Wed Feb 13 17:02:08 2019 +0000

    Make get_data/async/tmp_dir explicit

    functools.partial is all well and good in code, but apparently it
    doesn't play real well with docs.

    Change-Id: Ia460473af9038d890346502784e3cf4d0e1d1c40

commit ac01d186b44856385a13fa77ecf527238c803443
Author: Pete Zaitcev <email address hidden>
Date: Mon Feb 11 21:42:34 2019 -0600

    Leave less garbage in /var/tmp

    All our tests that invoked broker.set_sharding_state() created
    /var/tmp/tmp, when it called DatabaseBroker.get_device_path(),
    then added "tmp" to it. We provided 1 less level, so it walked up
    ouside of the test's temporary directory.

    The case of "cleanUp" instead of "tearDown" didn't break out of
    jail, but left trash in /var/tmp all the same.

    Change-Id: I8030ea49e2a977ebb7048e1d5dcf17338c1616df

commit bb1a2d45685a3b2230f21f7f6ff0e998e666723e
Author: Tim Burke <email address hidden>
Date: Fri Jul 27 20:03:36 2018 +0000

    Display crypto data/metadata details in swift-object-info

    Change-Id: If577c69670a10decdbbf5331b1a38d9392d12711

commit ea8e545a27f06868323ff91c1584d18ab9ac6cda
Author: Clay Gerrard <email address hidden>
Date: Mon Feb 4 15:46:40 2019 -0600

    Rebuild frags for unmounted disks

    Change the behavior of the EC reconstructor to perform a fragment
    rebuild to a handoff node when a primary peer responds with 507 to the
    REPLICATE request.

    Each primary node in a EC ring will sync with exactly three primary
    peers, in addition to the left & right nodes we now select a third node
    from the far side of the ring. If any of these partners respond
    unmounted the reconstructor will rebuild it's fragments to a handoff
    node with the appropriate index.

    To prevent ssync (which is uninterruptible) receiving a 409 (Conflict)
    we must give the remote handoff node the correct backend_index for the
    fragments it will recieve. In the common case we will use
    determistically different handoffs for each fragment index to prevent
    multiple unmounted primary disks from forcing a single handoff node to
    hold more than one rebuilt fragment.

    Handoff nodes will continue to attempt to revert rebuilt handoff
    fragments to the appropriate primary until it is remounted or
    rebalanced. After a rebalance of EC rings (potentially removing
    unmounted/failed devices), it's most IO efficient to run in
    handoffs_only mode to avoid unnecessary rebuilds.

    Closes-Bug: #1510342

    Change-Id: Ief44ed39d97f65e4270bf73051da9a2dd0ddbaec

commit 8a6159f67b6a3e7917e68310e4c24aae81...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633671
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=9b3ca9423eb8cf9420a3e98f60cd56dd281b4208
Submitter: Zuul
Branch: master

commit 9b3ca9423eb8cf9420a3e98f60cd56dd281b4208
Author: Simeon Gourlin <email address hidden>
Date: Tue Jan 29 09:13:16 2019 +0100

    Fix decryption for broken objects

    Try to get decryption object key from stored metadata (key_id path from
    X-Object-Sysmeta-Crypto-Body-Meta) because sometime object.path is wrong
    during encryption process.
    This patch doesn't solve the underlying issue, but is needed to decrypt
    already wrongly stored objects.

    Change-Id: I1a6bcdebdb46ef03c342428aeed73ae76db29922
    Co-Author: Thomas Goirand <email address hidden>
    Partial-Bug: #1813725

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/losf
Review: https://review.openstack.org/648245

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648245
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=6afc1130fd753306d64745c9bee7712182b273d3
Submitter: Zuul
Branch: feature/losf

commit 89e5927f7dd94fc28b3847944eb7dd227d516fa8
Author: Thiago da Silva <email address hidden>
Date: Tue Mar 26 10:46:02 2019 -0400

    Fix mocking time

    When running on Centos the side_effect was returning a MagicMock
    object instead of the intended int.

    Change-Id: I73713a9a96dc415073a637d85a40304021f76072

commit 50715acb1838fbde628e447e7b02545ce8469180
Author: OpenStack Release Bot <email address hidden>
Date: Mon Mar 25 17:07:54 2019 +0000

    Update master for stable/stein

    Add file to the reno documentation build to show release notes for
    stable/stein.

    Use pbr instruction to increment the minor version number
    automatically so that master versions are higher than the versions on
    stable/stein.

    Change-Id: I6109bff3227f87d914abf7bd1d76143aaf91419d
    Sem-Ver: feature

commit 179fa7ccd4d6faeacc989715887b69f9422a17b2
Author: John Dickinson <email address hidden>
Date: Mon Mar 18 17:09:31 2019 -0700

    authors/changelog update for 2.21.0 release

    Change-Id: Iac51a69c71491e5a8db435aae396178a6c592c73

commit 64eec5fc93eb670e581cbb3a6dedb6a7aa501e99
Author: Tim Burke <email address hidden>
Date: Thu Mar 7 14:36:02 2019 -0800

    Fix how we UTF-8-ify func tests

    I noticed while poking at the DLO func tests that we don't actually use
    non-ascii chars when we set up the test env.

    By patching the create name function earlier (in SetUpClass) we can
    ensure we get some more interesting characters in our object names.

    Change-Id: I9480ddf74463310aeb11ad876b79527888d8c871

commit fe3a20f2e4b745bf7d81f9bda97082b593e8794a
Author: Tim Burke <email address hidden>
Date: Tue Mar 19 14:52:19 2019 -0700

    Remove uncalled function

    Change-Id: Ica67815f0ddf4b00bce1ffe183735490c7f7c0b5
    Related-Change: I5629de9f2e9b2331ed3f455d253efc69d030df72

commit adc568c97f5b30d9d4628eaf448f81d736ad4e51
Author: John Dickinson <email address hidden>
Date: Fri Mar 15 15:18:36 2019 -0700

    Fix bulk responses when using xml and Expect 100-continue

    When we fixed bulk response heartbeating in https://review.openstack.org/#/c/510715/,
    code review raised the issue of moving the xml header down to after the
    early-exit clauses. At the time, it didn't seem to break anything, so
    it was left in place. However, that insight was correct.

    The purpose of the earlier patch was to force eventlet to use chunked
    transfer encoding on the response in order to prevent eventlet from
    buffering the whole response, thus defeating the purpose of the
    heartbeat responses.

    Moving the first line of the body lower (ie after the early exit
    checks), allows other headers in a chunked transfer encoding response
    to be appropriately processed before sending the headers. Sending the
    xml declaration early causes it to get intermingled in the 100-continue
    protocol, thus breaking the chunked transfer encoding semantics.

    Closes-Bug: #1819...

[Tim Burke (1-tim-z)]

I should have triaged this earlier, but definitely confirmed, and with https://opendev.org/openstack/swift/commit/9b3ca94 and https://opendev.org/openstack/swift/commit/4310331 landed, most of bad is contained. We should definitely make it harder to configured known-problematic pipelines (including ones that become problematic because of *our own* auto-insertion logic), but I *think* this is reasonable at Medium now? FWIW, we've got https://review.opendev.org/#/c/635040/ proposed to help with that.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/690629

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/690636

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690629
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8fc622b28a4fc7ed23ba79ee1db2103a541866b7
Submitter: Zuul
Branch: stable/rocky

commit 8fc622b28a4fc7ed23ba79ee1db2103a541866b7
Author: Tim Burke <email address hidden>
Date: Wed Feb 6 16:48:17 2019 -0800

    encryption: Stop being cutesy with os.path.join()

    Turns out, we *care* about the path, and object paths *don't follow
    filesystem semantics*!

    Be explicit: /<account>/<container>/<object>

    Bump the key version number so we know whether we can trust the full path
    or not.

    Change-Id: Ide9d44cc18575306363126a93d91f662c6ee23e0
    Related-Bug: 1813725
    (cherry picked from commit 43103319d0aa27f24e6520c0962bd19e55568ad4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690636
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=4af86c78f364c8c3bd8a576420d1c71ff89b4a6c
Submitter: Zuul
Branch: stable/rocky

commit 4af86c78f364c8c3bd8a576420d1c71ff89b4a6c
Author: Simeon Gourlin <email address hidden>
Date: Tue Jan 29 09:13:16 2019 +0100

    Fix decryption for broken objects

    Try to get decryption object key from stored metadata (key_id path from
    X-Object-Sysmeta-Crypto-Body-Meta) because sometime object.path is wrong
    during encryption process.
    This patch doesn't solve the underlying issue, but is needed to decrypt
    already wrongly stored objects.

    Change-Id: I1a6bcdebdb46ef03c342428aeed73ae76db29922
    Co-Author: Thomas Goirand <email address hidden>
    Partial-Bug: #1813725
    (cherry picked from commit 9b3ca9423eb8cf9420a3e98f60cd56dd281b4208)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/742756

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tim Burke (<email address hidden>) on branch: master
Review: https://review.opendev.org/742756
Reason: Squashed into parent patch, which probably shouldn't land without this functionality.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/742033
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=7d429318ddb854a23cdecfe35721b1ecbe8bcccc
Submitter: Zuul
Branch: master

commit 7d429318ddb854a23cdecfe35721b1ecbe8bcccc
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/748882

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/748883

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748883
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=eef87ee21229c8f8cc794b29e0f9abc159fc98e7
Submitter: Zuul
Branch: stable/train

commit eef87ee21229c8f8cc794b29e0f9abc159fc98e7
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725
    (cherry picked from commit 7d429318ddb854a23cdecfe35721b1ecbe8bcccc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748882
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8ed74c264605e12717c93e9b07f4ff50924e619f
Submitter: Zuul
Branch: stable/ussuri

commit 8ed74c264605e12717c93e9b07f4ff50924e619f
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725
    (cherry picked from commit 7d429318ddb854a23cdecfe35721b1ecbe8bcccc)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/749532

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/749536

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/749532
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=db485392fc238b32e29c8aca12d2e70dd9561004
Submitter: Zuul
Branch: stable/stein

commit db485392fc238b32e29c8aca12d2e70dd9561004
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725
    (cherry picked from commit 7d429318ddb854a23cdecfe35721b1ecbe8bcccc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/749536
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=ca66e2e96f2272870a03cebfb8c24b9b6bbacf16
Submitter: Zuul
Branch: stable/rocky

commit ca66e2e96f2272870a03cebfb8c24b9b6bbacf16
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725