Recently compacted shard DBs can return 500s during listings

[Tim Burke]

Seen in the wild:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/swift/container/server.py", line 893, in __call__
    res = getattr(self, req.method)(req)
  File "/usr/local/lib/python3.9/site-packages/swift/common/utils.py", line 2045, in _timing_stats
    resp = func(ctrl, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/swift/container/server.py", line 766, in GET
    container_list = broker.get_shard_ranges(
  File "/usr/local/lib/python3.9/site-packages/swift/container/backend.py", line 1806, in get_shard_ranges
    filler_sr.lower = last_upper
  File "/usr/local/lib/python3.9/site-packages/swift/common/utils.py", line 5222, in lower
    raise ValueError(
ValueError: lower ('...') must be less than or equal to upper ('...')

The trouble comes down to the DB being "sharded" for the purposes of the proxy's "x-backend-override-shard-name-filter: sharded" header (so we clear marker/end_marker) and fill_gaps being true (since we're listing). I *think* the fix is just to reset marker/end_marker to our own_shard_range's bounds.

[Tim Burke (1-tim-z)]

FWIW, you can readily see this while running probe tests -- just have another window running something like

   tail -f /var/log/syslog | sed -e '/#012/!d;s/#012/\n/g'

while you run the sharding probe tests.

[Alistair Coles (alistair-coles)]

https://review.opendev.org/c/openstack/swift/+/784929

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/swift 2.28.0 release.