OpenStack Object Storage (swift)

Swift3 functests should transition to using boto3

Bug #1557260 reported by Tim Burke on 2016-03-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	New	Undecided	Unassigned

Bug Description

With the sig V4 effort in https://review.openstack.org/#/c/211933/, we're finding a few issues with current boto. And from their README [1], it sounds like the likelihood of getting them fixed in the 2.x series is diminished:

> Boto3, the next version of Boto, is now stable and recommended for
> general use. It can be used side-by-side with Boto in the same project,
> so it is easy to start using Boto3 in your existing projects as well as
> new projects. Going forward, API updates and all new feature work will
> be focused on Boto3.
>
> To assist users who still depend on Boto and cannot immediately switch
> over, we will be triaging and addressing critical issues and PRs in Boto
> in the short term. As more users make the switch to Boto3, we expect to
> reduce our maintenance involvement over time. If we decide on a cutoff
> date or any significant changes to our maintenance plan, we will make
> pre-announcements well ahead of schedule to allow ample time for our
> users to adapt/migrate.

I don't know whether sig V4 issues would be covered by "critical issues" or not, but getting on 3.x [2] sooner rather than later is probably a good idea.

[1] https://github.com/boto/boto/blob/develop/README.rst
[2] https://github.com/boto/boto3

See original description

Tags:

Tim Burke (1-tim-z) on 2016-03-15

description:

updated

Revision history for this message

Andrey Pavlov (apavlov-e) wrote on 2016-03-16:

Can I do this work?

Revision history for this message

Tim Burke (1-tim-z) wrote on 2018-09-27:

New development will be on s3api in Swift's tree.

affects:

swift3 → swift

Tim Burke (1-tim-z) on 2019-05-03

tags:

added: s3api

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-22: Fix merged to swift (master)

Reviewed: https://review.opendev.org/657149
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=d4e7940457cd31b4ee91a01a0c492781179fcfe4
Submitter: Zuul
Branch: master

commit d4e7940457cd31b4ee91a01a0c492781179fcfe4
Author: Timur Alperovich <email address hidden>
Date: Sat May 4 12:11:54 2019 -0700

Start transition to boto3 in s3api tests.

    Adds the scaffolding required for tests to use boto3 and converts the
    test_bucket.py tests to the new interface. Follow on patches will
    convert the other tests to use the boto3 library.

    Notable changes: we no longer try to reach for the equivalent of
    `boto.make_request()` and instead rely on the boto3/botocore event
    system to mutate requests as necessary (or to disable pre-flight
    validators).

Partial-Bug: 1557260
Change-Id: I3d77ef4a6b878c49ebfa0c8b8647d7199d87601e

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-05: Fix proposed to swift (feature/losf)

Fix proposed to branch: feature/losf
Review: https://review.opendev.org/663260

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-05: Fix merged to swift (feature/losf)

Download full text (11.7 KiB)

Reviewed: https://review.opendev.org/663260
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=3ba65cb7eca6948d32b53ba91157037c48f06002
Submitter: Zuul
Branch: feature/losf

commit 79ab3a490ada6aa4db28efa5b71a3f1a080920f7
Author: Clay Gerrard <email address hidden>
Date: Tue Jun 4 11:45:35 2019 -0500

Note s3api compatability requirements in sample.conf

Change-Id: I12f6e4a5a880a2d5e0f4fff4618e4196be311267

commit eaf180c68dc7cba866df0c70199ae48fb382c6e0
Author: Pete Zaitcev <email address hidden>
Date: Mon Jun 3 22:00:41 2019 -0500

Fix a few Sphinx warnings

This precludes us from landing anything in the gate, because we
treat Sphinx warnings as errors in OpenStack.

Specifically, 3 warnings are fixed:

    /home/zuul/src/opendev.org/openstack/swift/swift/common/middleware\
    /s3api/etree.py:docstring of\
     swift.common.middleware.s3api.etree.Element:1:\
    Inline strong start-string without end-string.

    duplicate object description of\
     swift.common.middleware.versioned_writes, other instance in\
     middleware, use :noindex: for one of them

    duplicate object description of\
     swift.common.ring.composite_builder, other instance in\
     overview_ring, use :noindex: for one of them

Change-Id: I5e0bd61a301d46b2674b4605882462575b635220

commit 4f9595f1130f6c45f8e3e08db55dddc31844be93
Author: Matthew Oliver <email address hidden>
Date: Tue Apr 23 12:15:32 2019 +1000

py3: port symlink middleware

    This patch ports the symlink middleware to py3.
    The middleware itself seems to be mostly fine and most changes
    are in the symlink unit tests.

Change-Id: I973c2e1bb8969cf6bffece8ce68881c393efbaef

commit 2c4f284055e1031f17f97229056616017385d8bc
Author: zengjia <email address hidden>
Date: Thu May 23 10:53:31 2019 +0800

Modify the version path URL of the configuration file

    In the main branch, I think it is necessary to modify
    the configuration file path in the document as the
    main branch path, and to change the path to a stable
    version when the version is released.

Change-Id: Ib1bc4c59ed4c7624782b8b1cdea20847c14c90bf

commit ff04ef05cd5a8f9675cbc8f292fa666d03ae6644
Author: Tim Burke <email address hidden>
Date: Wed May 22 16:36:50 2019 -0700

Rework private-request-method interface

    Instead of taking a X-Backend-Allow-Method that *must match* the
    REQUEST_METHOD, take a truish X-Backend-Allow-Private-Methods and
    expand the set of allowed methods. This allows us to also expose
    the full list of available private methods when returning a 405.

    Drive-By: make async-delete tests a little more robust:
      * check that end_marker and prefix are preserved on subsequent
        listings
      * check that objects with a leading slash are correctly handled

Change-Id: I5542623f16e0b5a0d728a6706343809e50743f73

commit 04c8015b889e3ddcbf111f3b0fa99bbbe12b8624
Author: Clay Gerrard <email address hidden>
Date: Wed May 22 16:23:42 2019 -0500

Add more tests for async expirer

Related-Change-Id: Ia13ee5da3d1b5c536eccaadc7a6fdcd997374443
...

Reviewed:  https://review.opendev.org/663260
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=3ba65cb7eca6948d32b53ba91157037c48f06002
Submitter: Zuul
Branch:    feature/losf

commit 79ab3a490ada6aa4db28efa5b71a3f1a080920f7
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Jun 4 11:45:35 2019 -0500

Note s3api compatability requirements in sample.conf
    
    Change-Id: I12f6e4a5a880a2d5e0f4fff4618e4196be311267

commit eaf180c68dc7cba866df0c70199ae48fb382c6e0
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Mon Jun 3 22:00:41 2019 -0500

Fix a few Sphinx warnings
    
    This precludes us from landing anything in the gate, because we
    treat Sphinx warnings as errors in OpenStack.
    
    Specifically, 3 warnings are fixed:
    
    /home/zuul/src/opendev.org/openstack/swift/swift/common/middleware\
    /s3api/etree.py:docstring of\
     swift.common.middleware.s3api.etree.Element:1:\
    Inline strong start-string without end-string.
    
    duplicate object description of\
     swift.common.middleware.versioned_writes, other instance in\
     middleware, use :noindex: for one of them
    
    duplicate object description of\
     swift.common.ring.composite_builder, other instance in\
     overview_ring, use :noindex: for one of them
    
    Change-Id: I5e0bd61a301d46b2674b4605882462575b635220

commit 4f9595f1130f6c45f8e3e08db55dddc31844be93
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Tue Apr 23 12:15:32 2019 +1000

py3: port symlink middleware
    
    This patch ports the symlink middleware to py3.
    The middleware itself seems to be mostly fine and most changes
    are in the symlink unit tests.
    
    Change-Id: I973c2e1bb8969cf6bffece8ce68881c393efbaef

commit 2c4f284055e1031f17f97229056616017385d8bc
Author: zengjia <zengjia@awcloud.com>
Date:   Thu May 23 10:53:31 2019 +0800

Modify the version path URL of the configuration file
    
    In the main branch, I think it is necessary to modify
    the configuration file path in the document as the
    main branch path, and to change the path to a stable
    version when the version is released.
    
    Change-Id: Ib1bc4c59ed4c7624782b8b1cdea20847c14c90bf

commit ff04ef05cd5a8f9675cbc8f292fa666d03ae6644
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 22 16:36:50 2019 -0700

Rework private-request-method interface
    
    Instead of taking a X-Backend-Allow-Method that *must match* the
    REQUEST_METHOD, take a truish X-Backend-Allow-Private-Methods and
    expand the set of allowed methods. This allows us to also expose
    the full list of available private methods when returning a 405.
    
    Drive-By: make async-delete tests a little more robust:
      * check that end_marker and prefix are preserved on subsequent
        listings
      * check that objects with a leading slash are correctly handled
    
    Change-Id: I5542623f16e0b5a0d728a6706343809e50743f73

commit 04c8015b889e3ddcbf111f3b0fa99bbbe12b8624
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed May 22 16:23:42 2019 -0500

Add more tests for async expirer
    
    Related-Change-Id: Ia13ee5da3d1b5c536eccaadc7a6fdcd997374443
    Change-Id: Ic70eb61b32b196d356c9a3b40a22807456c7b0f2

commit 40bd9f16a348236a8710b98b66115f659c47354b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 15 15:29:08 2019 -0700

py3: port diskfile
    
    Change-Id: Ie73f9df460f9014014168d6e60767e66b24d30b0

commit 8868e2cd5062541f2d9b3d5c2091e4122a830f83
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 15 12:59:57 2019 -0700

Clean up test_write_read_metadata
    
    I'm pretty sure I botched the intent of the test when I touched it in
    https://github.com/openstack/swift/commit/36c4297
    
    As a side-effect, clean up a warning on py2:
    
        UnicodeWarning: Unicode equal comparison failed to convert
        both arguments to Unicode - interpreting them as being unequal
    
    Change-Id: I4da8264f85e688bc5109016c197378ca6dfe06a5

commit 83d01619910c34f2e3fafeda20bc4f7e7e8f482f
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed Mar 27 15:28:50 2019 -0700

Add operator tool to async-delete some or all objects in a container
    
    Adds a tool, swift-container-deleter, that takes an account/container
    and optional prefix, marker, and/or end-marker; spins up an internal
    client; makes listing requests against the container; and pushes the
    found objects into the object-expirer queue with a special
    application/async-deleted content-type.
    
    In order to do this enqueuing efficiently, a new internal-to-the-cluster
    container method is introduced: UPDATE. It takes a JSON list of object
    entries and runs them through merge_items.
    
    The object-expirer is updated to look for work items with this
    content-type and skip the X-If-Deleted-At check that it would normally
    do.
    
    Note that the target-container's listing will continue to show the
    objects until data is actually deleted, bypassing some of the concerns
    raised in the related change about clearing out a container entirely and
    then deleting it.
    
    Change-Id: Ia13ee5da3d1b5c536eccaadc7a6fdcd997374443
    Related-Change: I50e403dee75585fc1ff2bb385d6b2d2f13653cf8

commit 4927b1f29c4d993cb29786312a9efc00850c10cb
Author: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com>
Date:   Thu May 16 18:08:03 2019 +0200

Specify pickle protocol in REPLICATE()
    
    The default pickle protocol in python3 is version 3. This is not
    readable by a python2 interpreter.
    
    Force the use of version 2 in the object server REPLICATE() function,
    for compatibility with python 2.
    
    Change-Id: I19d23570ff3a084d288de1308e059cfd8134d6ad

commit 61a4567a8de62160afabfd63ba416fa940f9b742
Author: zengjia <zengjia@awcloud.com>
Date:   Wed May 22 14:55:16 2019 +0800

Controller install Debian missing proxy-server.conf configuration
    
    Missing proxy-server.conf configuration document for installing
    control nodes in Debian
    
    Change-Id: I174ba8841ed638732bcf933602c25aad8e62f0ca

commit 6008b0a83f862b2396ad6642b7863a30a258520b
Author: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com>
Date:   Thu Apr 18 16:35:07 2019 +0200

Fix quarantine when object path is not a directory
    
    In the diskfile module, there are a couple of cases where we would
    quarantine the suffix dir if a single object within is not a directory.
    
    Change the code so that we quarantine only the object.
    
    Change-Id: I35eb16f568c00d973eb089aedd0e5a5619c25a2f

commit d4e7940457cd31b4ee91a01a0c492781179fcfe4
Author: Timur Alperovich <timuralp@swiftstack.com>
Date:   Sat May 4 12:11:54 2019 -0700

Start transition to boto3 in s3api tests.
    
    Adds the scaffolding required for tests to use boto3 and converts the
    test_bucket.py tests to the new interface. Follow on patches will
    convert the other tests to use the boto3 library.
    
    Notable changes: we no longer try to reach for the equivalent of
    `boto.make_request()` and instead rely on the boto3/botocore event
    system to mutate requests as necessary (or to disable pre-flight
    validators).
    
    Partial-Bug: 1557260
    Change-Id: I3d77ef4a6b878c49ebfa0c8b8647d7199d87601e

commit bd8c3067b40092d53552ff686bdce030b9807fa8
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Tue Feb 19 17:36:11 2019 -0600

py3: slo
    
    This adds wsgi_to_str(self.path_info) everywhere we forgot it,
    not only in the slo module itself.
    
    Dropping the body=''.join(body) after call_slo() is obvious:
    the latter only returns strings of bytes, not lists of such.
    
    Change-Id: I6b4d87e4cda4945bc128dbc9c1edd39e736a59d2

commit ad034023f30f4d00e9c21bf3e095d3c04b4b7609
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri May 17 10:30:21 2019 -0700

Be explicit that a code path is just for py2
    
    Change-Id: I7d0345a712f71e76034910fa7bb2bbaf42b4d2ff

commit add5a411ace942663d99f92e443b1effc6e7a867
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 15 15:39:25 2019 -0700

Make build/upload/promote image jobs non-voting
    
    Change-Id: If284513ead5ff0d7f9e997ebf2a8820c9acf79e3

commit bec2cccd430845aba7437075feb3dbc7099fc5c2
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue May 14 17:10:54 2019 -0700

py3: port expirer
    
    Change-Id: Ib5aeda94fe9b165f54ad7412d68d71b62e427a40

commit 8b64381371b542d46fadba428d3b910dcbcd8d93
Author: Timur Alperovich <timuralp@swiftstack.com>
Date:   Mon May 13 18:38:24 2019 -0700

Set Content-Type with s3api metadata updates.
    
    S3 supports two metadata operations on object copy: COPY and REPLACE.
    When using REPLACE, the Content-Type should be set to the one supplied
    by the caller. When using COPY, the existing object's Content-Type value
    is used.
    
    Change-Id: Ic7c6278dedef308c9219eb45751abfa5655f144f
    Closes-Bug: #1828907

commit 5cd2a2ac2f0552247ada19cc02ec0ca5389375bf
Author: Tim Burke <tim.burke@gmail.com>
Date:   Sun May 5 12:38:57 2019 -0700

py3: bulk -- alternate approach
    
    Related-Change: I3603247e0c36299de2107aa8e494a3f87647696f
    Change-Id: Iaf14c1ab3e236e6774492ed711fccef066a16fca

commit 89c9c6f0b27fcc208996590610df8b04bec2ebc8
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri May 3 16:38:29 2019 -0700

Have a separate s3api functional test suite
    
    The idea is that we should have a suite of pure-S3 tests that we can
    point at AWS to verify that we've written accurate tests, then point at
    Swift-with-s3api to verify that we've correctly implemented the S3 api.
    
    As a start, just check GET Service; go ahead and create a few buckets
    so we can see them in the service listing.
    
    Change-Id: I283757cd3084b1c83a1e9bf0f46b6ce9d7ee8eb9

commit 4f4456561fb0443389e30efbe81c51c0e80cb6e6
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Mar 22 12:18:51 2019 -0700

Add py37 func test job
    
    Make a new func-py3 tox env that will use whatever version of py3 is
    available, and set up a new check/gate job that runs it under py37.
    
    Change-Id: I1ef233276697be994494ed858e88ce97634f814e

commit 506279235d12f1590bd928de51d991b7fb425e6e
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Mar 1 12:43:42 2019 -0800

Get functional/tests.py running under py3
    
    Note that you need a pretty recent eventlet to pick up
    https://github.com/eventlet/eventlet/commit/f0bc79e
    
    Change-Id: I6b006b972e7431c406039f4e0f6890a8f74a4432

commit dade632b0fbb10ef9419f2f9bcd063f3f962dbbf
Author: Timur Alperovich <timuralp@swiftstack.com>
Date:   Tue May 7 16:15:59 2019 -0700

S3Api: handle non-ASCII markers in v1 listings.
    
    Added a test for S3 v1 listings that use URL encoding and have non-ASCII
    characters. In the process discovered that the XML schema for
    ListBucketResult had a small problem: Delimiter and EncodingType needed
    to be reordered.
    
    Change-Id: Ib3124ea079a73a577b86de97657603a64b16f965

commit 9290f29e1cd622b5155ebec3600cc4c6e6fb0b31
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Apr 15 15:40:15 2019 -0700

Wait longer for log lines in unit test
    
    We've seen a noteworthy (literally -- there's a comment about it) number
    of failures for test_multiphase_put_drains_extra_commit_junk_disconnect;
    I'd rather waste an extra few hundredths of a second every run than have
    to recheck most patches we want merged.
    
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Change-Id: Iebec6c18866e4f8ef40faab539b173f017d658e3

commit 9468e28ccf4e84915c300e4f466ec2e31690495d
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed Apr 17 21:55:09 2019 -0700

Make test_read_write_acl_listings more robust
    
    When retrying in func tests, you may get a different status if the
    previous attempt succeeded.
    
    Change-Id: Ie9bf99af1298fae5cd91fb03d5b5f13ff4873b3c

tags:

added: in-feature-losf

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-05-03: Related fix proposed to swift (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/swift/+/918144

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-06-18: Related fix merged to swift (master)

Reviewed: https://review.opendev.org/c/openstack/swift/+/918144
Committed: https://opendev.org/openstack/swift/commit/dd8b7656daad1c244c94aa6d2bc8b4eb59c8cb3b
Submitter: "Zuul (22348)"
Branch: master

commit dd8b7656daad1c244c94aa6d2bc8b4eb59c8cb3b
Author: Tim Burke <email address hidden>
Date: Fri May 3 13:24:02 2024 -0700

Skip boto 2.x tests if boto is not installed

    The boto library was last updated two years ago and has
    rusted to the point that it's unusable on py312 -- see
    https://github.com/boto/boto/issues/3951

We should transition all of these tests to boto3 equivalents,
but this should help out in the meantime.

    Related-Bug: #1557260
    Related-Bug: #2063367
    Change-Id: If95f45371f352c6a2d16be1a3e1b64e265bccfb4

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-06-19: Related fix proposed to swift (feature/mpu)

Related fix proposed to branch: feature/mpu
Review: https://review.opendev.org/c/openstack/swift/+/922311

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-06-24: Related fix merged to swift (feature/mpu)

Download full text (8.6 KiB)

Reviewed: https://review.opendev.org/c/openstack/swift/+/922311
Committed: https://opendev.org/openstack/swift/commit/51fd293f1f947f6972e4fa6751f94f750bbd86f2
Submitter: "Zuul (22348)"
Branch: feature/mpu

commit 5be20f46df164b4b62e59916cfef15aebf880a7f
Author: Tim Burke <email address hidden>
Date: Mon Jun 17 15:58:56 2024 -0700

CI: update known failures for the ceph tests

    For some reason, when we switched from py36 on centos8 to py39 on
    centos9, these two tests started failing. Looks like a disagreement
    about whether the canonical path for a bucket request should have
    a trailing slash or not.

Mark them as known-failures for now so we can stay aware of any
other new breakage brought on by swift code changes.

Related-Change: I4f6b9c07af7bc768654f1a5d0c66b048e0f2c9c1
Change-Id: If990752c7ef7667182dbe18e49679e48c0e3d42d

commit a290f445efe812ad4886ca23cca00c8bb3b48c2b
Author: Tim Burke <email address hidden>
Date: Mon Jun 17 14:43:10 2024 -0700

CI: make sure old swift is truly gone for rolling-upgrade jobs

    While switching how some executable scripts were configured, I saw
    some strange rolling-upgrade failures that seemed to indicate that
    the new invocation method was trying to be used with old code. It
    seems like it maybe has something to do with whether swift was
    installed to /usr/local/lib/python3.9/site-packages/ or
    /usr/local/lib64/python3.9/site-packages/ but I'm not entirely
    sure.

    At any rate, a proper package manager ought to uninstall the old
    version then install the new one, so it seems reasonable to do that
    with pip, too.

Change-Id: I12e84745e7601d162755bc9d0f1cda7b63e92197

commit 90da23c7d232ba443b0814fd71c39f4585d30e4c
Author: Thomas Goirand <email address hidden>
Date: Thu Jun 13 11:27:55 2024 +0200

kms_keymaster: allow specifying barbican_endpoint

    Under a multi-region deployment with a single Keystone server,
    specifying the Keystone auth credentials isn't enough. Indeed,
    Castellan succeeds when logging-in, but may use the wrong
    Barbican endpoint (if there are 2 Barbican deployed). This is
    what happened to us, when deploying our 2nd region.

    They way to fix it would be to tell Castellan what region to use,
    unfortunately, there's no such option in Castellan. Though we may
    specify the barbican_endpoint, which is what this patch allows.

Change-Id: Ib7f4219ef5fdef65e9cfd5701e28b5288741783e

commit 3637b1abd9240f7feeebc7f299d42fe8815d7dcd
Author: Anish Kachinthaya <email address hidden>
Date: Mon Mar 11 19:02:10 2024 -0400

add bytes of expiring objects to queue entry

The size in bytes from object metadata of expiring objects are stored in
expirey queue entries under the content_type field.

    The x-content-type-timestamp take from object metadata is provided along
    with the x-content-type update so the container replicator resolves the
    latest content-type and ensures eventual consistency.

UpgradeImpact: During rolling upgrades you should expect expirer queue
entries to continue lacking swift_expirer_bytes= annotati...

Reviewed:  https://review.opendev.org/c/openstack/swift/+/922311
Committed: https://opendev.org/openstack/swift/commit/51fd293f1f947f6972e4fa6751f94f750bbd86f2
Submitter: "Zuul (22348)"
Branch:    feature/mpu

commit 5be20f46df164b4b62e59916cfef15aebf880a7f
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Jun 17 15:58:56 2024 -0700

CI: update known failures for the ceph tests
    
    For some reason, when we switched from py36 on centos8 to py39 on
    centos9, these two tests started failing. Looks like a disagreement
    about whether the canonical path for a bucket request should have
    a trailing slash or not.
    
    Mark them as known-failures for now so we can stay aware of any
    other new breakage brought on by swift code changes.
    
    Related-Change: I4f6b9c07af7bc768654f1a5d0c66b048e0f2c9c1
    Change-Id: If990752c7ef7667182dbe18e49679e48c0e3d42d

commit a290f445efe812ad4886ca23cca00c8bb3b48c2b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Jun 17 14:43:10 2024 -0700

CI: make sure old swift is truly gone for rolling-upgrade jobs
    
    While switching how some executable scripts were configured, I saw
    some strange rolling-upgrade failures that seemed to indicate that
    the new invocation method was trying to be used with old code. It
    seems like it maybe has something to do with whether swift was
    installed to /usr/local/lib/python3.9/site-packages/ or
    /usr/local/lib64/python3.9/site-packages/ but I'm not entirely
    sure.
    
    At any rate, a proper package manager ought to uninstall the old
    version then install the new one, so it seems reasonable to do that
    with pip, too.
    
    Change-Id: I12e84745e7601d162755bc9d0f1cda7b63e92197

commit 90da23c7d232ba443b0814fd71c39f4585d30e4c
Author: Thomas Goirand <thomas@goirand.fr>
Date:   Thu Jun 13 11:27:55 2024 +0200

kms_keymaster: allow specifying barbican_endpoint
    
    Under a multi-region deployment with a single Keystone server,
    specifying the Keystone auth credentials isn't enough. Indeed,
    Castellan succeeds when logging-in, but may use the wrong
    Barbican endpoint (if there are 2 Barbican deployed). This is
    what happened to us, when deploying our 2nd region.
    
    They way to fix it would be to tell Castellan what region to use,
    unfortunately, there's no such option in Castellan. Though we may
    specify the barbican_endpoint, which is what this patch allows.
    
    Change-Id: Ib7f4219ef5fdef65e9cfd5701e28b5288741783e

commit 3637b1abd9240f7feeebc7f299d42fe8815d7dcd
Author: Anish Kachinthaya <akachinthaya@nvidia.com>
Date:   Mon Mar 11 19:02:10 2024 -0400

add bytes of expiring objects to queue entry
    
    The size in bytes from object metadata of expiring objects are stored in
    expirey queue entries under the content_type field.
    
    The x-content-type-timestamp take from object metadata is provided along
    with the x-content-type update so the container replicator resolves the
    latest content-type and ensures eventual consistency.
    
    UpgradeImpact: During rolling upgrades you should expect expirer queue
    entries to continue lacking swift_expirer_bytes= annotations until ALL
    object servers replicas have been upgraded to new code.
    
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    
    Change-Id: Ie4b25f1bd16def4069878983049b83de06f68e54

commit eee7741251b701742b6954a986c102510d07d96e
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Jun 4 08:58:19 2024 -0700

CI: Move off CentOS 8
    
    Remove swift-tox-py36-centos-8-stream job entirely.
    
    Move the following jobs to CentOS 9:
    
      - swift-tox-func-s3api-ceph-s3tests-tempauth
      - swift-tox-func-s3api-tests-tempauth
      - swift-multinode-rolling-upgrade, as well as the other rolling
        upgrade jobs
    
    Remove the swift-multinode-rolling-upgrade-victoria job, as py39
    support (required for CentOS 9) was not added until wallaby.
    
    Change-Id: I4f6b9c07af7bc768654f1a5d0c66b048e0f2c9c1

commit a666010aae8aa9ba67341eeb5e69380fc20d02f5
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Jun 4 12:26:55 2024 -0500

Lazy import is not needed
    
    There was an abandoned change that made reference to a RecussionError
    when running a probe test that imported boto3 that had something to do
    with eventlet, ssl and a transitive dependency on requests-mock, but the
    fix that actually got merged seemed to depend on another change to
    tox.ini that disables request-mock when we run pytest.
    
    Either way, we already import from boto3 at the top of probe tests and
    it's in test-requirements; so we require it to be installed even if you
    don't have s3api in your pipeline.
    
    Related-Change: I789b257635c031ac0cb6e4b5980f741e0cb5244d
    Related-Change: I2793e335a08ad373c49cbbe6759d4e97cc420867
    Related-Change: If14e4d2c1af2efcbc99e9b6fe10973a7eb94d589
    
    Change-Id: Id2662bfc5ef2f21f901f1c98e6389c4cb01818a2

commit 99c629edb813c7d9fcd736f5d34b3a47d224a457
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue May 14 18:43:34 2024 -0500

tests: realistic task_container names
    
    Change-Id: Ie5d8b555489d28c3b901e5bbebdcecbde7bb3367

commit f3adce13753f9b32be60a68b49c79321cb552a61
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu May 23 09:28:58 2024 -0500

expirer: bad config should not loop forever
    
    Change-Id: I9413c72f41465fb8026848f71ec3b39fa990c3b7

commit a2df74ffe267565b7bb4af875141b73668388c39
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu May 23 10:11:14 2024 -0500

tests: new test_config module for utils.config
    
    Drive-by: fix ValueError message for non_negative_int
    
    Change-Id: I06508279d59fa57296dd85548f271a7812aeb45f

commit b447234b2f1e005025b5ce50010993ca226c2283
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 15 15:11:17 2024 -0700

Allow StatsdClients to no-op if no host provided
    
    We've been working toward separating our logger from our statsd client.
    This is generally a good idea; it's always been a little weird to have
    our special-case loggers that would allow you to *also* increment some
    counters.
    
    The end goal is to take a bunch of places that look like
    
        logger = utils.get_logger(conf)
        ...
        logger.info(...)
        logger.increment(...)
    
    and turn them into something more like
    
        logger = logs.get_adapted_logger(conf)
        stats = statsd_client.get_statsd_client(conf, logger=logger)
        ...
        logger.info(...)
        stats.increment(...)
    
    Take a lesson from logging: callers don't need to know whether the
    log_level is high enough that their message will be logged, or even
    whether logging is enabled at all. Code wanting to emit stats shouldn't
    need to know whether statsd collection has been configured, either.
    
    Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
    Change-Id: I6eb5b27a387cc2b7310ee11cc49d38fd2b6cbab8

commit 893deca2743cc7590dbea90d5b0adfcf999bbc8b
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu May 16 10:44:19 2024 -0500

test: more test for s3api v4 checksum
    
    Related-Change: I9924ff3b8d7d246631fe61b916823e028e2c01f2
    Change-Id: I2a4581e408e3012a44bfb0cd58563b82040e23fc

commit a3ee04152d6abbcb2aefcf8adbadcdca811f4502
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 15 08:27:55 2024 -0700

docs: Update further-reading URL for cross-domain policies
    
    The old one now just redirects to https://developer.adobe.com/
    
    Change-Id: Iba281bb8dceedef9bbb702abf0b93473bf91d9f7

commit dd8b7656daad1c244c94aa6d2bc8b4eb59c8cb3b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri May 3 13:24:02 2024 -0700

Skip boto 2.x tests if boto is not installed
    
    The boto library was last updated two years ago and has
    rusted to the point that it's unusable on py312 -- see
    https://github.com/boto/boto/issues/3951
    
    We should transition all of these tests to boto3 equivalents,
    but this should help out in the meantime.
    
    Related-Bug: #1557260
    Related-Bug: #2063367
    Change-Id: If95f45371f352c6a2d16be1a3e1b64e265bccfb4

commit 9b8e5a5c955340f71fbe9dad314f6241bd126d91
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu May 25 17:25:19 2023 -0700

docs: Add tempurl/formpost metrics
    
    Change-Id: I04478c44957032e911069bcd4859caf3baaddb63

commit 336c97bbf8a577a5f74dacec4dd245663d626289
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Jul 28 15:33:13 2022 -0700

replicator: Ensure handoffs can clear with large handoff_delete
    
    Change-Id: I7976e6bb151802dcf0c969e2468e686f2dfa78a7
    Closes-Bug: #1982201

tags:

added: in-feature-mpu

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

auto-github-boto-boto #3951
[open] Edit

Bug watches keep track of this bug in other bug trackers.