Older Boto Framework fails with keystone + swift functional testing

[Kottur]

Keystone dropped the support for xml, and supports only json payload and responses. Older Boto framework depends on xml payload and response. This breaks the functional tests shipped with swift.

Is there "config" way for skipping old boto framework based tests? Are there plans dropping the old framework in support of newer Boto3 framework?

Swift Version 2.31.0 (Antelope)
Ubuntu 22.04

[Tim Burke (1-tim-z)]

I'm confused -- boto should only be talking to swift, which in turn will talk to keystone. When does boto talk to keystone?

Do you have some example (scrubbed) logs demonstrating the problem? Probably want separate logs for the client, swift proxy, and keystone.

Currently, the only way to configure the boto tests to skip would be to remove the s3_access_key*/s3_secret_key* configs, which would also cause the boto3 tests to skip. Especially given the age of the boto library, I think we'd be amenable to a patch that would make it an optional test dependency, though, and skip tests if it's not installed.

[Kottur (skottur)]

Here is the error. BTW, do not have access to keystone logs and proxy logs.

botocore/parsers.py", line 508, in _parse_xml_string_to_dom
    raise ResponseParserError(
botocore.parsers.ResponseParserError: Unable to parse response (not well-formed (invalid token): line 1, column 0), invalid XML received. Further retries may succeed:
b'{"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "https://identity.keystone.swift-local.xyz:5000/v3/", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}'

[Tim Burke (1-tim-z)]

What's your config look like? Do you have s3_storage_url configured? It should be pointing at your swift endpoint, while auth_uri should be pointing at keystone. If s3_storage_url is not configured, I think it may get defaulted from auth_uri (under the assumption that you'd be using something like swauth or tempauth).

Maybe we should just skip *all* s3api tests if s3_storage_url is not configured...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/swift/+/918144

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/swift/+/918144
Committed: https://opendev.org/openstack/swift/commit/dd8b7656daad1c244c94aa6d2bc8b4eb59c8cb3b
Submitter: "Zuul (22348)"
Branch: master

commit dd8b7656daad1c244c94aa6d2bc8b4eb59c8cb3b
Author: Tim Burke <email address hidden>
Date: Fri May 3 13:24:02 2024 -0700

    Skip boto 2.x tests if boto is not installed

    The boto library was last updated two years ago and has
    rusted to the point that it's unusable on py312 -- see
    https://github.com/boto/boto/issues/3951

    We should transition all of these tests to boto3 equivalents,
    but this should help out in the meantime.

    Related-Bug: #1557260
    Related-Bug: #2063367
    Change-Id: If95f45371f352c6a2d16be1a3e1b64e265bccfb4

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: feature/mpu
Review: https://review.opendev.org/c/openstack/swift/+/922311

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/swift/+/922311
Committed: https://opendev.org/openstack/swift/commit/51fd293f1f947f6972e4fa6751f94f750bbd86f2
Submitter: "Zuul (22348)"
Branch: feature/mpu

commit 5be20f46df164b4b62e59916cfef15aebf880a7f
Author: Tim Burke <email address hidden>
Date: Mon Jun 17 15:58:56 2024 -0700

    CI: update known failures for the ceph tests

    For some reason, when we switched from py36 on centos8 to py39 on
    centos9, these two tests started failing. Looks like a disagreement
    about whether the canonical path for a bucket request should have
    a trailing slash or not.

    Mark them as known-failures for now so we can stay aware of any
    other new breakage brought on by swift code changes.

    Related-Change: I4f6b9c07af7bc768654f1a5d0c66b048e0f2c9c1
    Change-Id: If990752c7ef7667182dbe18e49679e48c0e3d42d

commit a290f445efe812ad4886ca23cca00c8bb3b48c2b
Author: Tim Burke <email address hidden>
Date: Mon Jun 17 14:43:10 2024 -0700

    CI: make sure old swift is truly gone for rolling-upgrade jobs

    While switching how some executable scripts were configured, I saw
    some strange rolling-upgrade failures that seemed to indicate that
    the new invocation method was trying to be used with old code. It
    seems like it maybe has something to do with whether swift was
    installed to /usr/local/lib/python3.9/site-packages/ or
    /usr/local/lib64/python3.9/site-packages/ but I'm not entirely
    sure.

    At any rate, a proper package manager ought to uninstall the old
    version then install the new one, so it seems reasonable to do that
    with pip, too.

    Change-Id: I12e84745e7601d162755bc9d0f1cda7b63e92197

commit 90da23c7d232ba443b0814fd71c39f4585d30e4c
Author: Thomas Goirand <email address hidden>
Date: Thu Jun 13 11:27:55 2024 +0200

    kms_keymaster: allow specifying barbican_endpoint

    Under a multi-region deployment with a single Keystone server,
    specifying the Keystone auth credentials isn't enough. Indeed,
    Castellan succeeds when logging-in, but may use the wrong
    Barbican endpoint (if there are 2 Barbican deployed). This is
    what happened to us, when deploying our 2nd region.

    They way to fix it would be to tell Castellan what region to use,
    unfortunately, there's no such option in Castellan. Though we may
    specify the barbican_endpoint, which is what this patch allows.

    Change-Id: Ib7f4219ef5fdef65e9cfd5701e28b5288741783e

commit 3637b1abd9240f7feeebc7f299d42fe8815d7dcd
Author: Anish Kachinthaya <email address hidden>
Date: Mon Mar 11 19:02:10 2024 -0400

    add bytes of expiring objects to queue entry

    The size in bytes from object metadata of expiring objects are stored in
    expirey queue entries under the content_type field.

    The x-content-type-timestamp take from object metadata is provided along
    with the x-content-type update so the container replicator resolves the
    latest content-type and ensures eventual consistency.

    UpgradeImpact: During rolling upgrades you should expect expirer queue
    entries to continue lacking swift_expirer_bytes= annotati...