Swift on py3 doesn't like message/rfc822 data

[Tim Burke]

While trying to upload:

$ curl http://saio:8090/v1/AUTH_test/c/o -X PUT -d '' -H content-type:message/rfc822 -v
* Trying 127.0.1.1...
* TCP_NODELAY set
* Connected to saio (127.0.1.1) port 8090 (#0)
> PUT /v1/AUTH_test/c/o HTTP/1.1
> Host: saio:8090
> User-Agent: curl/7.58.0
> Accept: */*
> content-type:message/rfc822
> Content-Length: 0
>
* Empty reply from server
* Connection #0 to host saio left intact
curl: (52) Empty reply from server

Proxy logs spew errors like:

Feb 13 05:50:38 saio proxy-server: STDERR: Traceback (most recent call last):
Feb 13 05:50:38 saio proxy-server: STDERR: File ".../eventlet/hubs/hub.py", line 461, in fire_timers
    timer()
Feb 13 05:50:38 saio proxy-server: STDERR: File ".../eventlet/hubs/timer.py", line 59, in __call__
    cb(*args, **kw)
Feb 13 05:50:38 saio proxy-server: STDERR: File ".../eventlet/greenthread.py", line 221, in main
    result = function(*args, **kwargs)
Feb 13 05:50:38 saio proxy-server: STDERR: File ".../eventlet/wsgi.py", line 818, in process_request
    proto.__init__(conn_state, self)
Feb 13 05:50:38 saio proxy-server: STDERR: File "/vagrant/swift/swift/common/wsgi.py", line 437, in __init__
    wsgi.HttpProtocol.__init__(self, *args, **kwargs)
Feb 13 05:50:38 saio proxy-server: STDERR: File ".../eventlet/wsgi.py", line 357, in __init__
    self.handle()
Feb 13 05:50:38 saio proxy-server: STDERR: File ".../eventlet/wsgi.py", line 390, in handle
    self.handle_one_request()
Feb 13 05:50:38 saio proxy-server: STDERR: File ".../swift/common/wsgi.py", line 555, in handle_one_request
    got = wsgi.HttpProtocol.handle_one_request(self)
Feb 13 05:50:38 saio proxy-server: STDERR: File ".../eventlet/wsgi.py", line 461, in handle_one_request
    self.environ = self.get_environ()
Feb 13 05:50:38 saio proxy-server: STDERR: File ".../swift/common/wsgi.py", line 508, in get_environ
    for line in header_payload.rstrip('\r\n').split('\n'):
Feb 13 05:50:38 saio proxy-server: STDERR: AttributeError: 'list' object has no attribute 'rstrip'

If you've got a py2 proxy and py3 backends, proxy gets the hangups and client gets a 503; if you've already got data in the cluster (because it was uploaded when everything was still py2), the GET fails with a 503 as every backend request dies like

Feb 13 06:00:40 saio proxy-server: ERROR with Object server 127.0.0.1:6040/sdb4 re: Trying to GET /v1/AUTH_test/c/o:
Traceback (most recent call last):
  File ".../swift/proxy/controllers/base.py", line 1273, in _make_node_request
    possible_source = conn.getresponse()
  File ".../swift/common/bufferedhttp.py", line 232, in getresponse
    response = HTTPConnection.getresponse(self)
  File ".../eventlet/green/http/client.py", line 1426, in getresponse
    response.begin()
  File ".../swift/common/bufferedhttp.py", line 93, in begin
    for line in header_payload.rstrip('\r\n').split('\n'):
AttributeError: 'list' object has no attribute 'rstrip'

The core of it comes down to our attempt [1] to fix up a bug in eventlet [2] that's inherited from cpython [3]; turns out using the email module to parse HTTP is just a terrible idea all around. Experimentally, it looks like any message/* content-type will trigger the bad?

[1] https://opendev.org/openstack/swift/commit/76fde89261e1940daadb720c41df1a3595314a97
[2] https://github.com/eventlet/eventlet/pull/574
[3] https://bugs.python.org/issue37093

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707312
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=04cc11b938aac3e9225870d4dd2b7f3fc89f20a3
Submitter: Zuul
Branch: master

commit 04cc11b938aac3e9225870d4dd2b7f3fc89f20a3
Author: Tim Burke <email address hidden>
Date: Tue Feb 11 21:34:05 2020 -0800

    py3: stop barfing on message/rfc822 Content-Types

    Closes-Bug: #1863053
    Change-Id: I7493d3e201e26df9f200e16bc081d8a0f30308b9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/718312

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718312
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=afb572f60c9b950dabb1904d52c9a58bd2a9c7dd
Submitter: Zuul
Branch: stable/train

commit afb572f60c9b950dabb1904d52c9a58bd2a9c7dd
Author: Tim Burke <email address hidden>
Date: Tue Feb 11 21:34:05 2020 -0800

    py3: stop barfing on message/rfc822 Content-Types

    Closes-Bug: #1863053
    Change-Id: I7493d3e201e26df9f200e16bc081d8a0f30308b9
    (cherry picked from commit 04cc11b938aac3e9225870d4dd2b7f3fc89f20a3)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/losf
Review: https://review.opendev.org/720819

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720819
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=4e594b43d03ac52eb5dbff9d357e63f07d1f28c8
Submitter: Zuul
Branch: feature/losf

commit 0c37da906e053964df24910cca364899aac91899
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Apr 16 09:55:40 2020 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

    Change-Id: I4022f269a6066a06bb9f8fa9b6a2d52653e7966e

commit 9a7ab4b3b26281d0d55d2e9f6a39cd59e1c637fe
Author: OpenStack Proposal Bot <email address hidden>
Date: Sat Apr 11 08:43:31 2020 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

    Change-Id: Iebcfa847fbacdedb6d9cf1acd17188f7beb12952

commit 415011e162fad469801e5f5e88a11960f638bdbf
Author: Romain de Joux <email address hidden>
Date: Tue Apr 7 22:03:11 2020 +0200

    s3api: Propagate backend PATH_INFO in environ for other middleware

    Use swift.backend_path entry in wsgi environment to propagate
    backend PATH_INFO.

    Needed by ceilometermiddleware to extract account/container info
    from PATH_INFO, patch: https://review.opendev.org/#/c/718085/

    Change-Id: Ifb3c6c30835d912c5ba4b2e03f2e0b5cb392671a

commit 875a142980b77f2b6eacb3a4d7b8bcb8bb29eacd
Author: Michele Valsecchi <email address hidden>
Date: Thu Apr 2 19:44:14 2020 +0900

    docs: Encourage usage of UUID inside /etc/fstab in examples

    Update doc examples to use explicit UUID of devices, instead of using
    unstable device names.

    Change-Id: I3a2eb7bbe4b4091d2567897904d939df1df6b251
    Closes-Bug: #1817966

commit 668242c4229d8bdb0f1b2d3183efcc33c6b67154
Author: Tim Burke <email address hidden>
Date: Fri Apr 3 10:44:25 2020 -0700

    pep8: Turn on E305

    Change-Id: Ia968ec7375ab346a2155769a46e74ce694a57fc2

commit 96b56519bf5756022d495aedc0033cd44e0e3267
Author: Andreas Jaeger <email address hidden>
Date: Fri Apr 3 10:53:34 2020 +0200

    Update hacking for Python3

    The repo is Python using both Python 2 and 3 now, so update hacking to
    version 2.0 which supports Python 2 and 3. Note that latest hacking
    release 3.0 only supports version 3.

    Fix problems found.

    Remove hacking and friends from lower-constraints, they are not needed
    for installation.

    Change-Id: I9bd913ee1b32ba1566c420973723296766d1812f

commit bbea88cb1a2f11d8609e4d43ac57f80717576ae8
Author: Romain de Joux <email address hidden>
Date: Mon Oct 7 18:02:21 2019 +0200

    Extend MemcacheRing.delete() API to manage server_key

    Memcachering provided set_multi/get_multi to set/get list of key on an
    unique memcached server selected in ring with server_key value. But
    current api doesn't allow to deleted value save with set_multi.

    This change add a server_key optional keyword to delete method to allow
    to delete entry in the memcached selected with se...