Encryption writes different paths for key_id on py2 and py3

[Tim Burke]

On py2, I see crypto meta like

Data crypto details: {
  "body_key": {
    "iv": "P8mgGrapX7D1+GTuu3MU1Q==",
    "key": "mUp+9gJVOgJbshYZ6pEKzMuimkZnQ3bpbNtdIl0BXLk="
  },
  "cipher": "AES_CTR_256",
  "iv": "at+NLMIYulxU5+/WrV61HQ==",
  "key_id": {
    "path": "/AUTH_test/c/\ud83c\udf34",
    "secret_id": "2018",
    "v": "2"
  }
}

but on py3, it looks like

Data crypto details: {
  "body_key": {
    "iv": "tthtblP9/pXJejlgEXqslQ==",
    "key": "pmkcVNE6wpo1f1L0KZL4UOjNIJPjbH/u2pwHYD0ibBQ="
  },
  "cipher": "AES_CTR_256",
  "iv": "q+D9/nJ3c/m871RbgMcvaA==",
  "key_id": {
    "path": "/AUTH_test/c/\u00f0\u009f\u008c\u00b4",
    "secret_id": "2018",
    "v": "2"
  }
}

That is, we passed the WSGI string to json.dumps(). Of course, this is going to cause upgrade issues when moving from py2 to py3. If you've got data written down on py2, upgrading your proxy to py3 gives tracebacks like

Jul 18 01:37:56 saio-py2 proxy-server: Path stored in meta ('/AUTH_test/c/🌴') does not match path from request ('/AUTH_test/c/ð\x9f\x8c´')! Using path from meta. (txn: txc22c1da047394f0d8e446-005f125274)
Jul 18 01:37:56 saio-py2 proxy-server: get_keys(): from callback: 'latin-1' codec can't encode character '\U0001f334' in position 13: ordinal not in range(256):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/swift/common/middleware/crypto/crypto_utils.py", line 170, in get_keys
    keys = fetch_crypto_keys(key_id=key_id)
  File "/usr/local/lib/python3.6/dist-packages/swift/common/middleware/crypto/keymaster.py", line 130, in fetch_crypto_keys
    path, secret_id=secret_id)
  File "/usr/local/lib/python3.6/dist-packages/swift/common/middleware/crypto/keymaster.py", line 299, in create_key
    return hmac.new(key, wsgi_to_bytes(path),
  File "/usr/local/lib/python3.6/dist-packages/swift/common/swob.py", line 279, in wsgi_to_bytes
    return wsgi_str.encode('latin1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\U0001f334' in position 13: ordinal not in range(256) (txn: txc22c1da047394f0d8e446-005f125274) (client_ip: 127.0.0.1)

[Tim Burke (1-tim-z)]

Also worth noting: in a mixed py2/py3 cluster, data written through a proxy on py3 may be returned corrupted when read on py2:

vagrant@saio:~/swift$ curl -H x-auth-token:$OS_AUTH_TOKEN http://saio/v1/AUTH_test/🌴/🌴 -v | hd | head -n 2
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.1.1...
* TCP_NODELAY set
* Connected to saio (127.0.1.1) port 80 (#0)
> GET /v1/AUTH_test/🌴/🌴 HTTP/1.1
> Host: saio
> User-Agent: curl/7.58.0
> Accept: */*
> x-auth-token:AUTH_tkb0683aed8c044d12a4e02a9f678dd2d9
>
< HTTP/1.1 200 OK
< Etag: "����C)���);������v,:���M��Ӻ"
< X-Object-Meta-Mtime: c�V����I��T��+.
< Content-Length: 4788
< Accept-Ranges: bytes
< Last-Modified: Mon, 20 Jul 2020 16:46:22 GMT
< X-Timestamp: 1595263581.06434
< Content-Type: application/octet-stream
< X-Trans-Id: tx0d091151085941a0901b1-005f15cddd
< X-Openstack-Request-Id: tx0d091151085941a0901b1-005f15cddd
< Date: Mon, 20 Jul 2020 17:01:17 GMT
<
{ [4788 bytes data]
00000000 bd e4 ea 56 2e 0a fd 2f 8d 70 b5 d3 e8 42 c6 93 |...V.../.p...B..|
00000010 35 bb bc 0f 3d 3c 4b c4 54 e7 6f d2 27 41 3f d8 |5...=<K.T.o.'A?.|
100 4788 100 4788 0 0 34446 0 --:--:-- --:--:-- --:--:-- 34446

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/742756

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tim Burke (<email address hidden>) on branch: master
Review: https://review.opendev.org/742756
Reason: Squashed into parent patch, which probably shouldn't land without this functionality.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/742033
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=7d429318ddb854a23cdecfe35721b1ecbe8bcccc
Submitter: Zuul
Branch: master

commit 7d429318ddb854a23cdecfe35721b1ecbe8bcccc
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/748882

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/748883

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748883
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=eef87ee21229c8f8cc794b29e0f9abc159fc98e7
Submitter: Zuul
Branch: stable/train

commit eef87ee21229c8f8cc794b29e0f9abc159fc98e7
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725
    (cherry picked from commit 7d429318ddb854a23cdecfe35721b1ecbe8bcccc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748882
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8ed74c264605e12717c93e9b07f4ff50924e619f
Submitter: Zuul
Branch: stable/ussuri

commit 8ed74c264605e12717c93e9b07f4ff50924e619f
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725
    (cherry picked from commit 7d429318ddb854a23cdecfe35721b1ecbe8bcccc)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/749532

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/749536

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/749532
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=db485392fc238b32e29c8aca12d2e70dd9561004
Submitter: Zuul
Branch: stable/stein

commit db485392fc238b32e29c8aca12d2e70dd9561004
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725
    (cherry picked from commit 7d429318ddb854a23cdecfe35721b1ecbe8bcccc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/749536
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=ca66e2e96f2272870a03cebfb8c24b9b6bbacf16
Submitter: Zuul
Branch: stable/rocky

commit ca66e2e96f2272870a03cebfb8c24b9b6bbacf16
Author: Tim Burke <email address hidden>
Date: Mon Jul 20 14:18:33 2020 -0700

    py3: Work with proper native string paths in crypto meta

    Previously, we would work with these paths as WSGI strings -- this would
    work fine when all data were read and written on the same major version
    of Python, but fail pretty badly during and after upgrading Python.

    In particular, if a py3 proxy-server tried to read existing data that
    was written down by a py2 proxy-server, it would hit an error and
    respond 500. Worse, if an un-upgraded py2 proxy tried to read data that
    was freshly-written by a py3 proxy, it would serve corrupt data back to
    the client (including a corrupt/invalid ETag and Content-Type).

    Now, ensure that both py2 and py3 write down paths as native strings.
    Make an effort to still work with WSGI-string metadata, though it can be
    ambiguous as to whether a string is a WSGI string or not. The heuristic
    used is if

     * the path from metadata does not match the (native-string) request
       path and
     * the path from metadata (when interpreted as a WSGI string) can be
       "un-wsgi-fied" without any encode/decode errors and
     * the native-string path from metadata *does* match the native-string
       request path

    then trust the path from the request. By contrast, we usually prefer the
    path from metadata in case there was a pipeline misconfiguration (see
    related bug).

    Add the ability to read and write a new, unambiguous version of metadata
    that always has the path as a native string. To support rolling
    upgrades, a new config option is added: meta_version_to_write. This
    defaults to 2 to support rolling upgrades without configuration changes,
    but the default may change to 3 in a future release.

    UpgradeImpact
    =============
    When upgrading from Swift 2.20.0 or Swift 2.19.1 or earlier, set

        meta_version_to_write = 1

    in your keymaster's configuration. Regardless of prior Swift version, set

        meta_version_to_write = 3

    after upgrading all proxy servers.

    When switching from Python 2 to Python 3, first upgrade Swift while on
    Python 2, then upgrade to Python 3.

    Change-Id: I00c6693c42c1a0220b64d8016d380d5985339658
    Closes-Bug: #1888037
    Related-Bug: #1813725

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/swift rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/swift stein-eol release.