dark_data object audit watcher deletes not-dark object in sharded containers

[Alistair Coles]

The dark_data object audit watcher plugin uses a direct client to check if an object is in its container. The direct client container request will NOT return a complete object listing for a sharded container because the root container does not have object records. As a result, depending on its configured action, the dark_data watcher will erroneously report/quarantine/delete the object diskfiles for objects in sharded containers.

A client container listing request will still list the object, but download will obviously fail.

To reproduce in a SAIO environment:

Run a probe test to create a sharded container with a few objects:

  nosetests ./test/probe/test_sharder.py:TestManagedContainerSharding.test_manage_shard_ranges

Use the shard-info tool to get the random container name that the probe test created
 python swift/cli/shard-info.py

Restart swift services
 swift-init restart main

List the container
 swift list container-67d19b5d-a191-4861-9693-1bb8ca228e95

Download the object - all OK :)
  swift download container-67d19b5d-a191-4861-9693-1bb8ca228e95 obj-0000

See the .data files on disk:
 tree /srv/

Add dark_data watcher to object-server.conf (only in a disposable SAIO, do not do this on a cluster with data you care about):
 emacs /etc/swift/object-server/1.conf.d/40_replication.conf

    [object-auditor]
    watchers = swift#dark_data

    [object-auditor:watcher:swift#dark_data]
    action=delete

Run the auditor:
 swift-init object-auditor once

Take a look at the logs
  journalctl -S -10m

Look for logs lines like:
  Apr 21 13:45:12 saio object-6030[18795]: [audit-watcher swift#dark_data] deleting dark data /srv/node3/sdb3/objects/7028/15f/dba4f461b85a6906382e00d48c31e15f

See the .data files have gone :(
 tree /srv/

Obj is still in listing
 swift list container-67d19b5d-a191-4861-9693-1bb8ca228e95

OOPS! Download the object...fails
  swift download container-67d19b5d-a191-4861-9693-1bb8ca228e95 obj-0000

[David Sariel (dsariel)]

Then seems like a problem will also happen when action=quarantine.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/swift/+/787656

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/swift/+/787656
Committed: https://opendev.org/openstack/swift/commit/6b91334298446352ef21f12cc64325e772083a57
Submitter: "Zuul (22348)"
Branch: master

commit 6b91334298446352ef21f12cc64325e772083a57
Author: Tim Burke <email address hidden>
Date: Thu Apr 22 16:59:19 2021 -0700

    Make the dark data watcher work with sharded containers

    Be willing to accept shards instead of objects when querying containers.
    If we receive shards, be willing to query them looking for the object.

    Change-Id: I0d8dd42f81b97dddd6cf8910afaef4ba85e67d27
    Partial-Bug: #1925346