OpenStack Object Storage (swift)

Bug #2003941
Comment #1

Comment 1 for bug 2003941

Revision history for this message

clayg (clay-gerrard) wrote on 2023-01-26:

One way you can get a ghost listing is you have an expired object (includes x-delete-at metdatadata) that hasn't yet been reaped by the object-expirer daemon (maybe becaues it's running behind or mis-configured ... do you monitor you .expiring_objects queue?). In this scenario using swift-get-nodes you would be able to discover the on-disk location and validate if there is still an object .data file or a tombstone with .ts

The other way you can get a ghost listing is you had a swift-container-server node with a containers database that was isolated from the cluster for longer than the configured reclaim_age (the default is only 7 days). After a reclaim_age the connected database servers will "reclaim" any "tombstone rows" indicating that an object was deleted. If an isolated database with a record of PUT at t1 rejoins the cluster and finds that all database records of the DELETE at t2 have been relcaimed the "missing" PUT at t1 will be replicated to the connected database servers w/o any tombstone row to prevent it. In this scenario the tombstone files in the object-data layer are likely also reclaimed, so you want find any object-data. But you might be able to determine which database were connected and isolated by examining the ROW_ID of the ghost rows.

FWIW s3api always returns 2XX for DELETE even if the object exists - but the swift API has always returned the status code that seemed most appropriate from what it observed when writing down the tombstone object for the delete (either there was a .data file => 2xx, or there was a tombstone or no data => 404). I agree it that we could "quibble" about the status - but it wouldn't be obviously helpful, and there may be barriers to changing the expected response from the swift v1 API to consider. Clients can probably infer, it's not such a big problem for DELETE to return "not found", but swiftclient at least may not be setting the best example here:

(nvidia) clayg@banana:~/Workspace/nvidia$ swift list test
test
(nvidia) clayg@banana:~/Workspace/nvidia$ swift delete test foo
Error Deleting: test/foo: Object DELETE failed: https://stg-swiftstack-maglev.ngc.nvidia.com/v1/AUTH_clayg/test/foo 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' (txn: txe9e3279a72184e69ba1f6-0063d2a9f5)
(nvidia) clayg@banana:~/Workspace/nvidia$ echo $?
1

One way you can get a ghost listing is you have an expired object (includes x-delete-at metdatadata) that hasn't yet been reaped by the object-expirer daemon (maybe becaues it's running behind or mis-configured ... do you monitor you .expiring_objects queue?).  In this scenario using swift-get-nodes you would be able to discover the on-disk location and validate if there is still an object .data file or a tombstone with .ts

The other way you can get a ghost listing is you had a swift-container-server node with a containers database that was isolated from the cluster for longer than the configured reclaim_age (the default is only 7 days).  After a reclaim_age the connected database servers will "reclaim" any "tombstone rows" indicating that an object was deleted.  If an isolated database with a record of PUT at t1 rejoins the cluster and finds that all database records of the DELETE at t2 have been relcaimed the "missing" PUT at t1 will be replicated to the connected database servers w/o any tombstone row to prevent it.  In this scenario the tombstone files in the object-data layer are likely also reclaimed, so you want find any object-data.  But you might be able to determine which database were connected and isolated by examining the ROW_ID of the ghost rows.

FWIW s3api always returns 2XX for DELETE even if the object exists - but the swift API has always returned the status code that seemed most appropriate from what it observed when writing down the tombstone object for the delete (either there was a .data file => 2xx, or there was a tombstone or no data => 404).  I agree it that we could "quibble" about the status - but it wouldn't be obviously helpful, and there may be barriers to changing the expected response from the swift v1 API to consider.  Clients can probably infer, it's not such a big problem for DELETE to return "not found", but swiftclient at least may not be setting the best example here:

(nvidia) clayg@banana:~/Workspace/nvidia$ swift list test
	test
	(nvidia) clayg@banana:~/Workspace/nvidia$ swift delete test foo
	Error Deleting: test/foo: Object DELETE failed: https://stg-swiftstack-maglev.ngc.nvidia.com/v1/AUTH_clayg/test/foo 404 Not Found  [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' (txn: txe9e3279a72184e69ba1f6-0063d2a9f5)
	(nvidia) clayg@banana:~/Workspace/nvidia$ echo $?
	1