OpenStack Object Storage (swift)

Reclaim of tombstone rows is unbounded and causes LockTimeout (10s)

Bug #1877651 reported by clayg on 2020-05-08

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Fix Released	Medium	clayg

Bug Description

While troubleshooting some big slow container dbs I noticed that their was a noticeable periodicity to their successful response rate:

204 894
500 0
Fri May 8 18:32:46 UTC 2020
204 912
500 0
Fri May 8 18:33:16 UTC 2020
204 890
500 0
Fri May 8 18:33:46 UTC 2020
204 888
500 0
Fri May 8 18:34:16 UTC 2020
204 906
500 0
Fri May 8 18:34:46 UTC 2020
204 908
500 0
Fri May 8 18:35:16 UTC 2020
204 217
500 380
Fri May 8 18:35:46 UTC 2020
204 659
500 139
Fri May 8 18:36:16 UTC 2020
204 897
500 0
Fri May 8 18:36:46 UTC 2020
204 906
500 0
Fri May 8 18:37:16 UTC 2020
204 922
500 0

500s would spike and success would dip - and it was happening every 5-10ms across nodes. We were seeing a lot of LockTimeout (10s)

At first I thought it was something related to our WAL (.pending files) - but after some flailing I turned off sharders and replicators and remembered that one time in the past I sort of remembered I hated this query:

https://github.com/openstack/swift/blob/master/swift/common/db.py#L989

I'm not even sure it uses the (name, deleted) index (might need a "WHERE name > ''" - but even if it can filter NOT deleted rows there could easily still be many many millons of tombstones in a given db that it has to page through to check their `created_at` < reclaim_age

which you know, might not even delete anything for a couple of weeks - and it's not like we need to get rid of reclaimed tombstones *immediately* or anything (they've been there for weeks already)

Yet we do it every time we replicate the database... and no one else can get in sql queries while that one is running.

I nooped that function and my LockTimeouts went away and there were no more dips in success rates.

Now I just need to figure out to LIMIT the query effectively so we amortize the work across multiple queries and ideally even a less frequent interval.

Tags:

Revision history for this message

Tim Burke (1-tim-z) wrote on 2020-05-13:

FWIW, looks like we *do* use the index; doing an EXPLAIN on that delete query gets me something like http://paste.openstack.org/raw/793483/ (note the use of SeekGE/IdxGT/IdxRowid).

Trying to cap the number of rows deleted per query might be tricky -- our usual sync-point system that we use for the replicator, container sync, and the reconciler uses ROWID, but that won't be useful with our index. Maybe we could first do something like

SELECT name FROM object WHERE deleted=1 AND name > ? ORDER BY NAME LIMIT 1 OFFSET ?

to find the next sync point, then

DELETE FROM object WHERE deleted=1 AND name > ? AND name < ? AND created_at < ?

to clean up just a portion of the namespace at a time. If the SELECT returns no rows, set the sync-point to the empty string and skip the DELETE. Probably make the offset to use something reasonably low like 1,000 or 10,000. I think I'd advocate for an easy-to-change constant somewhere over a new config option, though.

Note that if we're seriously considering bug 1521363, resolving that would almost certainly require a new (deleted, created_at) index that would make our existing query quite well-optimized... but I don't know that anyone actually wants to add a new index to container DBs, especially given how difficult it would be to make it play well with sharding.

Revision history for this message

Tim Burke (1-tim-z) wrote on 2020-05-13:

Oh, also: I wonder if this was at the core of the complaint in bug 1260460 -- 10M objects in a 2.5GB DB doesn't seem that big. (These days, anyway; maybe I've grown too accustomed to having DBs on flash...)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-13: Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.opendev.org/727876

Changed in swift:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-26: Fix merged to swift (master)

Reviewed: https://review.opendev.org/727876
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=aab45880f8aaddfed15fe641939aac0d2eccc465
Submitter: Zuul
Branch: master

commit aab45880f8aaddfed15fe641939aac0d2eccc465
Author: Clay Gerrard <email address hidden>
Date: Wed May 13 13:32:18 2020 -0500

Breakup reclaim into batches

    We want to do the table scan without locking and group the locking
    deletes into small indexed operations to minimize the impact of
    background processes calling reclaim each cycle.

    Change-Id: I3ccd145c14a9b68ff8a9da61f79034549c9bc127
    Co-Authored-By: Tim Burke <email address hidden>
    Closes-Bug: #1877651

Changed in swift:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-12: Fix proposed to swift (feature/losf)

Fix proposed to branch: feature/losf
Review: https://review.opendev.org/735381

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-06-19: Fix merged to swift (feature/losf)

Download full text (20.6 KiB)

Reviewed: https://review.opendev.org/735381
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=481f126e6b59689599f438e5d27f7328f5b3e813
Submitter: Zuul
Branch: feature/losf

commit 51a587ed8dd5700b558ad26d70dcb7facc0f91e4
Author: Tim Burke <email address hidden>
Date: Tue Jun 16 11:34:01 2020 -0700

Use ensure-pip role

Hopefully this will fix the currently-broken probe test gate?

Depends-On: https://review.opendev.org/#/c/736070/
Change-Id: Ib652534b35236fdb6bcab131c7dc08a079bf72f6

commit 79811df34c84b416ce9f445926b31a23a32ea1a4
Author: Tim Burke <email address hidden>
Date: Fri Apr 10 22:02:57 2020 -0700

Use ini_file to update timeout instead of crudini

crudini seems to have trouble on py3 -- still not sure *why* it's using
py3 for the losf job, though...

Change-Id: Id98055994c8d59e561372417c9eb4aec969afc6a

commit e4586fdcde5267f39056bb1b5f413a411bb8e7a0
Author: Tim Burke <email address hidden>
Date: Tue Jun 9 10:50:07 2020 -0700

memcached: Plumb logger into MemcacheRing

This way proxies log memcached errors in the normal way instead of
to the root logger (which eventually gets them out on STDERR).

If no logger is provided, fall back to the root logger behavior.

Change-Id: I2f7b3e7d5b976fab07c9a2d0a9b8c0bd9a840dfd

commit 1dfa41dada30c139129cb2771b0d68c95fd84e32
Author: Tim Burke <email address hidden>
Date: Tue Apr 28 10:45:27 2020 -0700

swift-get-nodes: Allow users to specify either quoted or unquoted paths

    Now that we can have null bytes in Swift paths, we need a way for
    operators to be able to locate such containers and objects. Our usual
    trick of making sure the name is properly quoted for the shell won't
    suffice; running something like

swift-get-nodes /etc/swift/container.ring.gz $'AUTH_test/\0versions\0container'

has the path get cut off after "AUTH_test/" because of how argv works.

So, add a new option, --quoted, to let operators indicate that they
already quoted the path.

Drive-bys:

      * If account, container, or object are explicitly blank, treat them
        as though they were not provided. This provides better errors when
        account is explicitly blank, for example.
      * If account, container, or object are not provided or explicitly
        blank, skip printing them. This resolves abiguities about things
        like objects whose name is actually "None".
      * When displaying account, container, and object, quote them (since
        they may contain newlines or other control characters).

    Change-Id: I3d10e121b403de7533cc3671604bcbdecb02c795
    Related-Change: If912f71d8b0d03369680374e8233da85d8d38f85
    Closes-Bug: #1875734
    Closes-Bug: #1875735
    Closes-Bug: #1875736
    Related-Bug: #1791302

commit 1b6c8f7fdf630458affe2778fc7be86df3ef1674
Author: Tim Burke <email address hidden>
Date: Fri Jun 5 16:36:32 2020 -0700

Remove etag-quoter from 2.25.0 release notes

This was released in 2.24.0, which already has a release note for it.

Change-Id: I9837df281ec8baa19e8e4a7976f415e8add4a2da

commi...

Reviewed:  https://review.opendev.org/735381
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=481f126e6b59689599f438e5d27f7328f5b3e813
Submitter: Zuul
Branch:    feature/losf

commit 51a587ed8dd5700b558ad26d70dcb7facc0f91e4
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Jun 16 11:34:01 2020 -0700

Use ensure-pip role
    
    Hopefully this will fix the currently-broken probe test gate?
    
    Depends-On: https://review.opendev.org/#/c/736070/
    Change-Id: Ib652534b35236fdb6bcab131c7dc08a079bf72f6

commit 79811df34c84b416ce9f445926b31a23a32ea1a4
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Apr 10 22:02:57 2020 -0700

Use ini_file to update timeout instead of crudini
    
    crudini seems to have trouble on py3 -- still not sure *why* it's using
    py3 for the losf job, though...
    
    Change-Id: Id98055994c8d59e561372417c9eb4aec969afc6a

commit e4586fdcde5267f39056bb1b5f413a411bb8e7a0
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Jun 9 10:50:07 2020 -0700

memcached: Plumb logger into MemcacheRing
    
    This way proxies log memcached errors in the normal way instead of
    to the root logger (which eventually gets them out on STDERR).
    
    If no logger is provided, fall back to the root logger behavior.
    
    Change-Id: I2f7b3e7d5b976fab07c9a2d0a9b8c0bd9a840dfd

commit 1dfa41dada30c139129cb2771b0d68c95fd84e32
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Apr 28 10:45:27 2020 -0700

swift-get-nodes: Allow users to specify either quoted or unquoted paths
    
    Now that we can have null bytes in Swift paths, we need a way for
    operators to be able to locate such containers and objects. Our usual
    trick of making sure the name is properly quoted for the shell won't
    suffice; running something like
    
       swift-get-nodes /etc/swift/container.ring.gz $'AUTH_test/\0versions\0container'
    
    has the path get cut off after "AUTH_test/" because of how argv works.
    
    So, add a new option, --quoted, to let operators indicate that they
    already quoted the path.
    
    Drive-bys:
    
      * If account, container, or object are explicitly blank, treat them
        as though they were not provided. This provides better errors when
        account is explicitly blank, for example.
      * If account, container, or object are not provided or explicitly
        blank, skip printing them. This resolves abiguities about things
        like objects whose name is actually "None".
      * When displaying account, container, and object, quote them (since
        they may contain newlines or other control characters).
    
    Change-Id: I3d10e121b403de7533cc3671604bcbdecb02c795
    Related-Change: If912f71d8b0d03369680374e8233da85d8d38f85
    Closes-Bug: #1875734
    Closes-Bug: #1875735
    Closes-Bug: #1875736
    Related-Bug: #1791302

commit 1b6c8f7fdf630458affe2778fc7be86df3ef1674
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Jun 5 16:36:32 2020 -0700

Remove etag-quoter from 2.25.0 release notes
    
    This was released in 2.24.0, which already has a release note for it.
    
    Change-Id: I9837df281ec8baa19e8e4a7976f415e8add4a2da

commit 11dd0da29a5f348549236eed1a315bbb98441c94
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Jun 5 14:19:57 2020 -0700

tests: Bump up timeout for unit and in-process func tests
    
    We've been seeing more TIMEOUT failures lately where the jobs still seem
    to be making steady (if slow) progress.
    
    Change-Id: I19c1f48bace551c78ad0c6c8b6ccad75e44e8904

commit fc731198acec365cd46d2c5e0c7c297e948ebe66
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Jun 5 09:49:01 2020 -0500

Quiet eventlet exceptions in test
    
    This traceback would probably show up in the wild, but it's not relevant
    to this test.
    
    Change-Id: I9e6e7679f674bddddcc4440b38d5741aaece3393

commit ce4c0fb14b6c92cab4b80e4143171df75c5f47b1
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu May 28 22:20:21 2020 -0700

Don't auto-create shard containers
    
    ...unless the client requests it specifically using a new flag:
    
       X-Backend-Auto-Create: true
    
    Previously, you could get real jittery listings during a rebalance:
    
     * Partition with a shard DB get reassigned, so one primary has no DB.
     * Proxy makes a listing, gets a 404, tries another node. Likely, one of
       the other shard replicas responds. Things are fine.
     * Update comes in. Since we use the auto_create_account_prefix
       namespace for shards, container DB gets created and we write the row.
     * Proxy makes another listing. There's a one-in-three chance that we
       claim there's only one object in that whole range.
    
    Note that unsharded databases would respond to the update with a 404 and
    wait for one of the other primaries (or the old primary that's now a
    hand-off) to rsync a whole DB over, keeping us in the happy state.
    
    Now, if the account is in the shards namespace, 404 the object update if
    we have no DB. Wait for replication like in the unsharded case.
    
    Continue to be willing to create the DB when the sharder is seeding all
    the CREATED databases before it starts cleaving, though.
    
    Change-Id: I15052f3f17999e6f432951ba7c0731dcdc9475bb
    Closes-Bug: #1881210

commit 3d105b623d13d8fe6bb4db676196a44ac3db464f
Author: Andreas Jaeger <aj@suse.com>
Date:   Tue Jun 2 18:53:59 2020 +0200

Switch to newer openstackdocstheme and reno versions
    
    Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
    these versions will allow especially:
    * Linking from HTML to PDF document
    * Allow parallel building of documents
    * Fix some rendering problems
    
    Update Sphinx version as well.
    
    Disable openstackdocs_auto_name to use 'project' variable as name.
    
    Set openstackdocs_pdf_link to link to PDF file. Note that
    the link to the published document only works on docs.openstack.org
    where the PDF file is placed in the top-level html directory. The
    site-preview places the PDF in a pdf directory.
    
    Change pygments_style to 'native' since old theme version always used
    'native' and the theme now respects the setting and using 'sphinx' can
    lead to some strange rendering.
    
    Remove docs requirements from lower-constraints, they are not needed
    during install or test but only for docs building.
    
    openstackdocstheme renames some variables, so follow the renames
    before the next release removes them. A couple of variables are also
    not needed anymore, remove them.
    
    See also
    http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
    
    Change-Id: I131850d2a5c6164dfd48c9c95885d4754b5236c6

commit 019bade19cd20aaebb6dd1f423ab2dbb15cd12a1
Author: Andreas Jaeger <aj@suse.com>
Date:   Tue Jun 2 21:42:05 2020 +0200

Remove <py3.5 dependencies from requirements.txt
    
    The requirements repo is support python 3.5 as oldest python version
    while swift still supports py27.
    
    thus, requirements-check will fail on a couple of lines in swift.  The
    check is only run when these files are touched.
    
    The py2.7 packagers we know about aren't depending on upstream
    requirements.txt for correctness and aside from all the production
    deployments running on py2.7 we only realistically support >=py3.7
    
    There's no good reason for our requirements.txt to be "unspported" by
    the openstack requirements check job.  Since they only support >=py3.5
    we can change our requirements.txt inline with that.  This should be
    fine for everything we could hope to get out of both our
    requirements.txt and the check!
    
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Change-Id: Ibf8000498528c401707be8b0b91b8355cd993786

commit 738514c16422cb7215fbf3996282395edc09c290
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Jun 2 21:51:00 2020 -0700

docs: s3api doesn't support tagging
    
    Change-Id: I227e8c2defe96fb5f33054167677ae551830d9e8

commit 20c6bdb71c89d20c8efcdad904425cdee10e522f
Author: Thiago da Silva <thiagodasilva@gmail.com>
Date:   Mon Oct 28 22:10:42 2019 +0200

Enable s3api and staticweb tests across all func tests
    
    This patch removed the separate s3api, staticweb functional tests
    gate jobs and added them across all other functional test jobs.
    
    Change-Id: Ie1c606132a054defc2b3cc14a66031090e7b8449

commit 99947150dd923ce19112a8f8c35c41f5a1271d72
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Nov 19 21:25:45 2019 -0800

func tests: work with etag-quoter on by default
    
    Also, run the in-process encryption func tests like that.
    
    Change-Id: I984ab8d1304d23b89589973950b10dda4aea0db3

commit ede9dad9f6e56831f3068c805cdd61dbb3fac7c0
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Jun 1 09:20:59 2020 -0500

Better functest quarantine cleanup
    
    Change-Id: I9218aaeb5fcd21f1bc2a5d655e3216059a209aeb

commit cedec8c5ef661c7d057cfc74f8ee39411c664dff
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Aug 6 18:17:14 2019 -0700

Latch shard-stat reporting
    
    The idea is, if none of
    
      - timestamp,
      - object_count,
      - bytes_used,
      - state, or
      - epoch
    
    has changed, we shouldn't need to send an update back to the root
    container.
    
    This is more-or-less comparable to what the container-updater does to
    avoid unnecessary writes to the account.
    
    Closes-Bug: #1834097
    Change-Id: I1ee7ba5eae3c508064714c4deb4f7c6bbbfa32af

commit 60f052f69aee346cf6f53426327bc81e2580fb6b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 27 15:07:52 2020 -0700

dsvm: Run service-user tests under keystoneauth
    
    Change-Id: I5b0ae0b78d37a31928e0b34a1b6118802a3a8236

commit 984b57a873e0729c425c2fa8b29fa4d03edc395b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue May 26 22:40:37 2020 -0700

tests: Skip s3api copy-version test when OV not enabled
    
    Change-Id: I175b9e1464494454b1193b9b9d5311cb6fd6503f

commit 3aa4e3ec85fb6d8686bdadcb455817bb3b80543e
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 27 13:16:55 2020 -0700

Remove swift-dsvm-functional-py3 job
    
    Following https://github.com/openstack/devstack/commit/6b6bdc711,
    the swift-dsvm-functional job has been running everything under
    Python 3 anyway.
    
    Change-Id: Ie285f513e1ed71dbaf4e12fe747ea6087664f843

commit 650272eacb6a71d24ba690599c48e710bf8993f9
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed May 27 10:03:59 2020 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: I1bb2aa262730d45a5852eaa3b3351f455a6ab337

commit fa768b4342192128ca51be90b07796c45d938a49
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue May 26 16:46:19 2020 -0700

Simplify wsgify()
    
    Change-Id: Iec399aa8b58e72152a17265f2af1131f02667131

commit a2feefb0453e387bd281b22f6f33d861272c8e2e
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue May 26 16:27:38 2020 -0700

dsvm: Use devstack's s3api "service"
    
    ...instead of hacking up the pipeline ourselves.
    
    Depends-On: https://review.opendev.org/731003
    Depends-On: https://review.opendev.org/731065
    Change-Id: Iea8a42ef54e1a2fd9c1d6132c840a20015cc5d7e

commit 5034916c1853f490daf0f02bedc41e2432ea2076
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue May 26 16:42:52 2020 -0700

Skip tests when only changing README.rst, CONTRIBUTING.rst, etc.
    
    Change-Id: Ib78398111ee3d150b6f56aa4e59a65750aa115dd

commit 78cce72f8a495dd7b2daf50623c60348fcb95adf
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Tue May 12 11:08:57 2020 +1000

Ussuri contrib docs community goal
    
    This patch standardizes the CONTRIBUTING.rst file and adds the
    required doc/source/contributor/contributing.rst
    
    Swift already had a detailed CONTIRBUTING.rst and an informative
    REVIEW_GUIDELINES.rst in the root of the repo. So we are also pulling
    them into the contributor documentation so they can not only be easily
    found in the checked repo but in the online documentation.
    
    Change-Id: I4c84efbe50eb25ab922c9d6b69198dae341af48b

commit 73f0b143d18e4920ec4d8cc35833573491e853d6
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue May 26 12:54:17 2020 -0700

dsvm: Run service-user tests under tempauth
    
    Change-Id: I0bdd3a1d044f8f99873f6270ca821862bb994d72

commit 63b16a368366d7c1fb6976841ce28a914128ce6a
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue May 26 12:41:03 2020 -0700

dsvm: Run domain_remap tests with tempauth
    
    Change-Id: Idcc33c71869ccf7ad5a3134767d36ae94aab4598

commit a8e03f42e09de3f3c660e0d4786bb35fe1317482
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Apr 28 16:48:52 2020 -0700

versioning: Have versioning symlinks make pre-auth requests to reserved container
    
    Previously, the lack of container ACLs on the reserved container would
    mean that attempting to grant access to the user-visible container would
    not work; the user could not access the backing object.
    
    Now, have symlinks with the allow-reserved-names sysmeta set be
    pre-authed. Note that the user still has to be authorized to read the
    symlink, and if the backing object was *itself* a symlink, that will be
    authed separately.
    
    Change-Id: Ifd744044421ef2ca917ce9502b155a6514ce8ecf
    Closes-Bug: #1880013

commit 63e02fa9fa4e43cb20dd96eee81d0749edcca7e6
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri May 22 15:00:59 2020 -0500

Test for Versioned Object If-Match
    
    Regular objects can do conditional requests, versioned objects should be
    able to as well.
    
    Change-Id: If69dbf2a4c876fa4b830f6d8652f22f39ceaf549

commit bb9b0326fde08768e6d609a210a1d1a5ec1c32ff
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Apr 23 16:26:53 2020 -0700

swift-dsvm: Enable s3api
    
    Depends-On: https://review.opendev.org/#/c/571021/
    Change-Id: I3ac3288cd61b745ce7dbf2bded8eade026d0418f

commit aab45880f8aaddfed15fe641939aac0d2eccc465
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed May 13 13:32:18 2020 -0500

Breakup reclaim into batches
    
    We want to do the table scan without locking and group the locking
    deletes into small indexed operations to minimize the impact of
    background processes calling reclaim each cycle.
    
    Change-Id: I3ccd145c14a9b68ff8a9da61f79034549c9bc127
    Co-Authored-By: Tim Burke <tim.burke@gmail.com>
    Closes-Bug: #1877651

commit f25705ee13e418d873a925c7f016a032f48cea88
Author: zhangboye <zhangboye@inspur.com>
Date:   Fri May 22 15:16:34 2020 +0800

Add py38 package metadata
    
    Change-Id: Icd5312dec647f69a5df111c3a6766093ecfcb6ef

commit 67598b3a4a6f841726b7358245ae18edbfc58250
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed May 20 15:22:10 2020 -0700

Bump up timeouts for DSVM and probe test jobs
    
    DSVM recently got a bunch more middlewares enabled, so it's running more
    tests than it used to.
    
    I can't think of much that's changed for probe tests, but I feel like
    I've seen it popping timeouts more often lately.
    
    Note that the new timeouts are still lower than the typical run-time of
    our longest-running jobs (currently grenade / tempest-ipv6-only).
    
    Change-Id: Iffbb567124096df02b04981550faec8204d5d1ec
    Related-Change: I3cbbcd2ea9ced0923bee4a6b0783e4cf5e82e95b

commit 1db11df4f25ee4ee5d60e5214ba8d9577c484d67
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon May 18 13:28:47 2020 -0700

ratelimit: Allow multiple placements
    
    We usually want to have ratelimit fairly far left in the pipeline -- the
    assumption is that something like an auth check will be fairly expensive
    and we should try to shield the auth system so it doesn't melt under the
    load of a misbehaved swift client.
    
    But with S3 requests, we can't know the account/container that a request
    is destined for until *after* auth. Fortunately, we've already got some
    code to make s3api play well with ratelimit.
    
    So, let's have our cake and eat it, too: allow operators to place
    ratelimit once, before auth, for swift requests and again, after auth,
    for s3api. They'll both use the same memcached keys (so users can't
    switch APIs to effectively double their limit), but still only have each
    S3 request counted against the limit once.
    
    Change-Id: If003bb43f39427fe47a0f5a01dbcc19e1b3b67ef

commit f4bc9515082c75c393f6d9540932f354aa0c3663
Author: Thiago da Silva <thiago@redhat.com>
Date:   Tue May 29 16:57:57 2018 -0400

fix s3api functional tests
    
    Connection was hard coded to use saio config
    
    Change-Id: I9c11162de89fa3aa2a78aea093b187d0309860f5
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit 1358d37328e1ee3245a61777c2b6aea269af8832
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri May 15 16:03:02 2020 -0700

versioning: Use CONTAINER_LISTING_LIMIT when doing container listings
    
    Change-Id: If274a01a6660776cf01b71efbdd102159c520794

commit dee98a74d43771d48a58d62647a0628ef7d1cf76
Author: Tim Burke <tim.burke@gmail.com>
Date:   Sat May 9 23:16:04 2020 -0700

updater: Shuffle suffixes so we don't keep hitting the same failures
    
    When tuning your updater, you often want to try a new config, see how it
    changes your metrics, then adjust concurrency up or down depending on
    how your container layer is responding.
    
    If your containers haven't been doing well, though, and you've got a
    giant backlog of async pendings to work through, updater restarts to
    change concurrency previously posed a problem: the updater would walk
    the suffix directories in the same order every start-up. So, if you
    found a config that was making decent progress for a while but still had
    *some* failures, and you wanted to try tweaking settings to see if you
    could *reduce* those failures -- you'd likely start getting *all*
    failures as it went to retry the failed ones first and all at once. If
    you continued trying to tweak configs to get your failures to a
    reasonable rate, you'd almost certainly over-correct for these handful
    of overwhelmed DBs and not the overall cluster.
    
    Now, shuffle the suffixes before we walk them.
    
    Change-Id: I3ef34119f0cb563ab405a6517335a24dbaf2b4c3
    Closes-Bug: #1878056

commit f57d4cfa71888c887e0e8e0ce349f2a5befb57a5
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon May 11 00:09:49 2020 -0700

object-updater: Ignore ENOENT when trying to unlink stale pending files
    
    Change-Id: Iaac1fb891d70707af38c567d9cca5913b8355b7d
    Closes-Bug: #1877924

commit 1af995f0e81994671583ce0fff4135164be11b6a
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Apr 23 17:21:22 2020 -0700

s3api: Check whether versioning is enabled more
    
    Previously, attempting to GET, HEAD, or DELETE an object with a non-null
    version-id would cause 500s, with logs complaining about how
    
        version-aware operations require that the container is versioned
    
    Now, we'll early-return with a 404 (on GET or HEAD) or 204 (on DELETE).
    
    Change-Id: I46bfd4ae7d49657a94734962c087f350e758fead
    Closes-Bug: 1874295

commit 3061ec803f1bf11ba88d8eaec96db2c007307c3a
Author: Romain LE DISEZ <romain.le-disez@corp.ovh.com>
Date:   Thu Nov 14 16:26:48 2019 -0500

relinker: Improve performance by limiting I/O
    
    This commit reduce the number of I/O done by the swift-object-relinker.
    
    First, it saves a progress state of relinking and cleanup in case the
    process is interrupted during the operation. This allow to resume
    operation without rescanning all partitions.
    
    Secondly, it prevents from being scanned by relink and cleanup all
    partitions that are bigger than 2^part_power (or (2^next_part_power)/2).
    These partitions were not existing before the beginning of the part_power
    increase, so there is nothing to relink or cleanup.
    
    Thirdly, it reverse-orders the partitions to scan so that some useless
    work is avoided. If a device contains partitions 1 and 3, relinking
    partition 1 will create "new" objects in partition 3, that will need to
    be scanned when the relinker will work on partition 3. It is useless. If
    partition 3 is done first, it will only contain the objects that need to
    be relinked.
    
    Fourthly, it allows to specify a unique device to work on.
    
    To do that, some hooks were added in audit_location_generator to allow
    to execute some custom code before/after iterating a
    device/partition/suffix/hash.
    
    Change-Id: If1bf8ed9036fb0ec619b0d4f16061a81a1af2082

tags:

added: in-feature-losf

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-17: Fix proposed to swift (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/741753

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-17: Fix proposed to swift (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/741755

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-22: Fix merged to swift (stable/train)

Reviewed: https://review.opendev.org/741755
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=170dddaacc609493d4b3e7ba1f6a9168e7633216
Submitter: Zuul
Branch: stable/train

commit 170dddaacc609493d4b3e7ba1f6a9168e7633216
Author: Clay Gerrard <email address hidden>
Date: Wed May 13 13:32:18 2020 -0500

Breakup reclaim into batches

    We want to do the table scan without locking and group the locking
    deletes into small indexed operations to minimize the impact of
    background processes calling reclaim each cycle.

    Change-Id: I3ccd145c14a9b68ff8a9da61f79034549c9bc127
    Co-Authored-By: Tim Burke <email address hidden>
    Closes-Bug: #1877651
    (cherry picked from commit aab45880f8aaddfed15fe641939aac0d2eccc465)

tags:

added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-07-22: Fix merged to swift (stable/ussuri)

#10

Reviewed: https://review.opendev.org/741753
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bb0a283a07df3c2c73eb696c5c3ac75e4a366f5a
Submitter: Zuul
Branch: stable/ussuri

commit bb0a283a07df3c2c73eb696c5c3ac75e4a366f5a
Author: Clay Gerrard <email address hidden>
Date: Wed May 13 13:32:18 2020 -0500

Breakup reclaim into batches

    We want to do the table scan without locking and group the locking
    deletes into small indexed operations to minimize the impact of
    background processes calling reclaim each cycle.

tags:

added: in-stable-ussuri

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.