Troubles with unicode account names

[Tim Burke]

With our most-recent release, there was a note [1] that

> tempauth user names now support unicode characters.

...which made me think that we'd expect a proxy-server.conf like

 [filter:tempauth]
 use = egg:swift#tempauth
 user_admin_admin = admin .admin .reseller_admin
 user_tést_tester = testing .admin
 user_tést2_tester2 = testing2 .admin
 user_tést_tester3 = testing3

and test.conf like

 account = tést
 username = tester
 password = testing
 account2 = tést2
 username2 = tester2
 password2 = testing2
 username3 = tester3
 password3 = testing3

to Just Work™. Nope. After hitting Ctrl-C on test.functional.test_slo.TestSlo.test_slo_ranged_get (which hangs, so we don't even get a complete test run) you get

 Ran 305 tests in 170.202s

 FAILED (SKIP=176, errors=31, failures=18)

This seems to be a combination of problems: there are test problems like

Traceback (most recent call last):
  File "/vagrant/swift/test/functional/test_object.py", line 123, in tearDown
    resp = retry(delete, container, obj)
  File "/vagrant/swift/test/functional/__init__.py", line 1122, in retry
    *args, **kwargs)
  File "/vagrant/swift/test/functional/test_object.py", line 108, in delete
    'DELETE', '/'.join([parsed.path, container, obj['name']]), '',
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128)

There are ACL problems like

Traceback (most recent call last):
  File "/vagrant/swift/test/functional/__init__.py", line 1202, in wrapper
    rv = f(*args, **kwargs)
  File "/vagrant/swift/test/functional/test_account.py", line 330, in test_admin_acl
    self.assertIn(resp.status, (200, 204))
AssertionError: 403 not found in (200, 204)

There are wonky request problems like

Traceback (most recent call last):
  File "/vagrant/swift/test/functional/test_slo.py", line 871, in test_slo_put_heartbeating
    self.assertIn('Response Status: 201 Created', body_lines)
AssertionError: 'Response Status: 201 Created' not found in ['Response Body: Bad Request', 'The server could not comply with the request since it is either malformed or otherwise incorrect.', 'Response Status: 400 Bad Request', 'Errors:', '/d94a68c9f5bd47a6a29c97c4e62530a3/seg_b, Size Mismatch', '/d94a68c9f5bd47a6a29c97c4e62530a3/seg_b, Etag Mismatch', '']

It'd be nice if "supporting Unicode account names" included being able to run functional tests against Unicode account names.

[1] https://github.com/openstack/swift/blob/2.18.0/CHANGELOG#L70

[clayg (clay-gerrard)]

Running functests with tim's configs resulted in logs like this for me:

May 31 18:49:41 ubuntu-bionic proxy-server: Error: An error occurred: #012Traceback (most recent call last):#012 File "/vagrant/swift/swift/common/middleware/catch_errors.py", line 41, in handle_request#012 resp = self._app_call(env)#012 File "/vagrant/swift/swift/common/wsgi.py", line 1177, in _app_call#012 resp = self.app(env, self._start_response)#012 File "/vagrant/swift/swift/common/middleware/gatekeeper.py", line 123, in __call__#012 return self.app(env, gatekeeper_response)#012 File "/vagrant/swift/swift/common/middleware/healthcheck.py", line 47, in __call__#012 if req.path == '/healthcheck':#012 File "/vagrant/swift/swift/common/swob.py", line 934, in path#012 self.environ['PATH_INFO'])#012 File "/usr/lib/python2.7/urllib.py", line 1298, in quote#012 return ''.join(map(quoter, s))#012KeyError: u'\xc3' (txn: txda76f573d4db42cbb8d6f-005b1043c5)

I agree with Tim we don't really "support unicode account names" unless we can AT LEAST get tempauth passing functests...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/571904

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/571906

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/571907

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/571908

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571904
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=7cec5a8fdb8ea376aa3f495181f0f295c1c84571
Submitter: Zuul
Branch: master

commit 7cec5a8fdb8ea376aa3f495181f0f295c1c84571
Author: Tim Burke <email address hidden>
Date: Wed May 30 15:57:12 2018 -0700

    tempauth: Send back url-encoded account names

    Taking user_tést_tester and sending back a link to http://saio:8080/v1/AUTH_tést
    may work, but it would be better as http://saio:8080/v1/AUTH_t%C3%A9st

    This is particularly important if you define something like user_test%ff_tester.

    Change-Id: I19d21af94c21fccb4fb835acae231dec424790bb
    Related-Bug: 1774238

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571048
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=9d0162a793115d55dd11c40b619657db41fa9968
Submitter: Zuul
Branch: master

commit 9d0162a793115d55dd11c40b619657db41fa9968
Author: Tim Burke <email address hidden>
Date: Tue May 29 16:37:42 2018 -0700

    Make ACLs work with Unicode in user/account names

    There were two separate complications:

      - Account ACLs are stored as JSON, so they were getting deserialized
        to Unicode. That's fine for ASCII names (as u'a' == b'a' under py2),
        but not arbitrary Unicode (as u'\u1234' != u'\u1234'.encode('utf8')
        for both py2 and py3). So, under py2, encode all account ACL members
        as UTF-8.

      - Container ACLs are stored as comma-separated values in a header, but
        values may contain arbitrary characters including not only non-ASCII
        Unicode but also commas and newlines. Fortunately, we have precedent
        for using URL-encoding in headers to resolve this. See crypto,
        symlink, dlo, versioned_writes, copy...

    Change-Id: I37a97bb9d039a963c7cc57bd97876d0ec2134cf1
    Partial-Bug: 1774238

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571905
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=fa678949ae310aa0499938fef788ec04409625d9
Submitter: Zuul
Branch: master

commit fa678949ae310aa0499938fef788ec04409625d9
Author: Tim Burke <email address hidden>
Date: Wed May 30 11:43:40 2018 -0700

    Fix quoting for large objects

    Change-Id: I46bdb6da8f778a6c86e0f8e883b52fc31e9fd44e
    Partial-Bug: 1774238
    Closes-Bug: 1678022
    Closes-Bug: 1598093
    Closes-Bug: 1762997

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/losf
Review: https://review.openstack.org/648245

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648245
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=6afc1130fd753306d64745c9bee7712182b273d3
Submitter: Zuul
Branch: feature/losf

commit 89e5927f7dd94fc28b3847944eb7dd227d516fa8
Author: Thiago da Silva <email address hidden>
Date: Tue Mar 26 10:46:02 2019 -0400

    Fix mocking time

    When running on Centos the side_effect was returning a MagicMock
    object instead of the intended int.

    Change-Id: I73713a9a96dc415073a637d85a40304021f76072

commit 50715acb1838fbde628e447e7b02545ce8469180
Author: OpenStack Release Bot <email address hidden>
Date: Mon Mar 25 17:07:54 2019 +0000

    Update master for stable/stein

    Add file to the reno documentation build to show release notes for
    stable/stein.

    Use pbr instruction to increment the minor version number
    automatically so that master versions are higher than the versions on
    stable/stein.

    Change-Id: I6109bff3227f87d914abf7bd1d76143aaf91419d
    Sem-Ver: feature

commit 179fa7ccd4d6faeacc989715887b69f9422a17b2
Author: John Dickinson <email address hidden>
Date: Mon Mar 18 17:09:31 2019 -0700

    authors/changelog update for 2.21.0 release

    Change-Id: Iac51a69c71491e5a8db435aae396178a6c592c73

commit 64eec5fc93eb670e581cbb3a6dedb6a7aa501e99
Author: Tim Burke <email address hidden>
Date: Thu Mar 7 14:36:02 2019 -0800

    Fix how we UTF-8-ify func tests

    I noticed while poking at the DLO func tests that we don't actually use
    non-ascii chars when we set up the test env.

    By patching the create name function earlier (in SetUpClass) we can
    ensure we get some more interesting characters in our object names.

    Change-Id: I9480ddf74463310aeb11ad876b79527888d8c871

commit fe3a20f2e4b745bf7d81f9bda97082b593e8794a
Author: Tim Burke <email address hidden>
Date: Tue Mar 19 14:52:19 2019 -0700

    Remove uncalled function

    Change-Id: Ica67815f0ddf4b00bce1ffe183735490c7f7c0b5
    Related-Change: I5629de9f2e9b2331ed3f455d253efc69d030df72

commit adc568c97f5b30d9d4628eaf448f81d736ad4e51
Author: John Dickinson <email address hidden>
Date: Fri Mar 15 15:18:36 2019 -0700

    Fix bulk responses when using xml and Expect 100-continue

    When we fixed bulk response heartbeating in https://review.openstack.org/#/c/510715/,
    code review raised the issue of moving the xml header down to after the
    early-exit clauses. At the time, it didn't seem to break anything, so
    it was left in place. However, that insight was correct.

    The purpose of the earlier patch was to force eventlet to use chunked
    transfer encoding on the response in order to prevent eventlet from
    buffering the whole response, thus defeating the purpose of the
    heartbeat responses.

    Moving the first line of the body lower (ie after the early exit
    checks), allows other headers in a chunked transfer encoding response
    to be appropriately processed before sending the headers. Sending the
    xml declaration early causes it to get intermingled in the 100-continue
    protocol, thus breaking the chunked transfer encoding semantics.

    Closes-Bug: #1819...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571906
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=e5e22ebeba2140b1da2e9d52d6b8ecd92bc75c88
Submitter: Zuul
Branch: master

commit e5e22ebeba2140b1da2e9d52d6b8ecd92bc75c88
Author: Tim Burke <email address hidden>
Date: Fri Jun 1 15:38:10 2018 -0700

    Make symlink work with Unicode account names

    Also, ensure that the stored symlink headers really *are* url-encoded as
    we've been assuming.

    Change-Id: I1f300d69bec43f0deb430294da05a4ec04308040
    Related-Bug: 1774238
    Closes-Bug: #1821240

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571907
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=e5ff405ec967bfe19b3eaeaac5b5a32c95218f9d
Submitter: Zuul
Branch: master

commit e5ff405ec967bfe19b3eaeaac5b5a32c95218f9d
Author: Tim Burke <email address hidden>
Date: Fri Jun 1 15:59:25 2018 -0700

    Make staticweb return URL-encoded Location headers

    Also, enhance tests to verify proper encoding in generated responses.

    Change-Id: Ib696b1c3a34722f3a6de010973b90ef8b1917012
    Partial-Bug: 1774238

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/losf
Review: https://review.opendev.org/655630

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/655630
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=d391957fd70a02866615c9c5edc070dbde2b23ea
Submitter: Zuul
Branch: feature/losf

commit 32bf43990c3c9ce771fca5968242f102155467c1
Author: ZhongShengping <[e-mail]>
Date: Mon Apr 22 11:05:10 2019 +0800

    Replace git.openstack.org URLs with opendev.org URLs

    Thorough replacement of git.openstack.org URLs with their opendev.org
    counterparts.

    Change-Id: I5e77307de6a484fd69b2a5863423ceb357e8601f

commit 89eced960c5bf5c2e14b6245c70b615dc23d45a6
Author: OpenDev Sysadmins <email address hidden>
Date: Fri Apr 19 19:28:47 2019 +0000

    OpenDev Migration Patch

    This commit was bulk generated and pushed by the OpenDev sysadmins
    as a part of the Git hosting and code review systems migration
    detailed in these mailing list posts:

    http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
    http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

    Attempts have been made to correct repository namespaces and
    hostnames based on simple pattern matching, but it's possible some
    were updated incorrectly or missed entirely. Please reach out to us
    via the contact information listed at https://opendev.org/ with any
    questions you may have.

commit b6ebabee78b77108b2a3e5261873f038772d304d
Author: Tim Burke <email address hidden>
Date: Wed Apr 17 09:15:27 2019 -0700

    Clean up dlo unit tests

    We don't actually need so many py2/py3 branches, and once we
    clean those up, there's hardly any reason to import six.

    Change-Id: Ia3b4f02e7eb99ad1a76aa35c39dc198528fd39ad

commit 893acffbc01ba817361f8d8735513dc784fb92c0
Author: Pete Zaitcev <email address hidden>
Date: Wed Feb 20 17:09:08 2019 -0600

    py3: port dlo

    Change-Id: I7236ddea0acde93d0789ad8affa76df0097a86aa

commit 7cb276da425592280cd3f522c06cfb7b02311bde
Author: Tim Burke <email address hidden>
Date: Tue Apr 2 17:42:21 2019 -0700

    Move from py35 to py37 for gating py3 jobs

    Looking at https://governance.openstack.org/tc/reference/runtimes/train.html
    OpenStack is no longer targetting Python 3.5, so it doesn't really make sense
    to gate on it. Move to Python 3.7, just to make it so we can go longer before
    having to do this shuffle again.

    Presumably, once we actually declare support for Python 3.6, we should
    make that job voting, too? We've had enough gate flaky-ness lately that
    I'm not keen on it.

    Change-Id: I9c9e5df76a6592ab9c512b7c1ba807f9a0053059

commit 65d3cb75f817b3683aa85b73239d606e29c73d1c
Author: zhufl <email address hidden>
Date: Mon Apr 15 15:12:30 2019 +0800

    Use assertIn to check whether substring is in str

    This is to use assertIn to check whether substring is in str,
    go get clearer failure message.

    Change-Id: I40aff29454c423389755a5330751d2f69a227a05

commit e5ff405ec967bfe19b3eaeaac5b5a32c95218f9d
Author: Tim Burke <email address hidden>
Date: Fri Jun 1 15:59:25 2018 -0700

    Make staticweb return URL-encoded Location headers

 ...