seems like when the reconciler-ic randomly discovers the node with the live entry it'll overwrite it; but it took me like 3 tries: vagrant@saio:~$ swift-init container-reconciler once -nv WARNING: Unable to modify max process limit. Running as non-root? Removing stale pid file /var/run/swift/container-reconciler.pid.d Running container-reconciler once...(/etc/swift/container-reconciler.conf.d) container-reconciler: Starting 152380 container-reconciler-ic: Loaded override config for (default): ProxyOverrideOptions({}, {'sorting_method': 'shuffle', 'read_affinity': '', 'write_affinity': '', 'write_affinity_node_count': '2 * replicas', 'write_affinity_handoff_delete_count': None, 'rebalance_missing_suppression_count': 1, 'concurrent_gets': False, 'concurrency_timeout': 0.5, 'concurrent_ec_extra_requests': 0}, app) container-reconciler: pulling items from the queue container-reconciler: checking container 1698868800 container-reconciler: looking for objects in 1698868800 container-reconciler-ic: - - 01/Nov/2023/20/08/54 GET /v1/.misplaced_objects/1698868800%3Fformat%3Djson%26marker%3D%26end_marker%3D%26prefix%3D%26states%3Dlisting HTTP/1.0 404 - Swift%20Container%20Reconciler - - 70 - txde8dd968bf514c1097a4e-006542b056 - 0.0253 - - 1698869334.574397087 1698869334.599655628 0 container-reconciler: looking for containers in .misplaced_objects container-reconciler-ic: - - 01/Nov/2023/20/08/54 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D%26prefix%3D HTTP/1.0 200 - Swift%20Container%20Reconciler - - 95 - txf47471e7c26f4baab2396-006542b056 - 0.0114 - - 1698869334.604813814 1698869334.616210461 - container-reconciler-ic: - - 01/Nov/2023/20/08/54 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D1698865200%26end_marker%3D%26prefix%3D HTTP/1.0 200 - Swift%20Container%20Reconciler - - 2 - txf9932f0095a14fa59f95a-006542b056 - 0.0102 - - 1698869334.619512558 1698869334.629701376 - container-reconciler: checking container 1698865200 container-reconciler: looking for objects in 1698865200 container-reconciler-ic: - - 01/Nov/2023/20/08/54 GET /v1/.misplaced_objects/1698865200%3Fformat%3Djson%26marker%3D%26end_marker%3D%26prefix%3D%26states%3Dlisting HTTP/1.0 200 - Swift%20Container%20Reconciler - - 2 - txb5b854b8072044b090aa3-006542b056 - 0.0113 - - 1698869334.636687517 1698869334.647985458 0 container-reconciler: Reconciler Stats: {} container-reconciler: Exited 152380 vagrant@saio:~$ for i in {1..4}; do echo server $i; curl -s "http://127.0.0.${i}:60${i}1/sdb${i}/${MISPLACED_CONTAINER_PART}/.misplaced_objects/${MISPLACED_CONTAINER}?format=json" | python -m json.tool; done # query backend queue server 1 [ { "bytes": 0, "hash": "1698868068.21431_3", "name": "1:/AUTH_test/\u0000container\u0000753a0681-cd2f-4eec-869f-be3b9e27150c/\u0000object\u000079b75334-cb71-42ff-94ce-ffd6521e5f5f", "content_type": "application/x-delete", "last_modified": "2023-11-01T19:47:48.214310" } ] server 2 Expecting value: line 1 column 1 (char 0) server 3 [ { "bytes": 0, "hash": "1698868068.21431_3", "name": "1:/AUTH_test/\u0000container\u0000753a0681-cd2f-4eec-869f-be3b9e27150c/\u0000object\u000079b75334-cb71-42ff-94ce-ffd6521e5f5f", "content_type": "application/x-delete", "last_modified": "2023-11-01T19:47:48.214310" } ] server 4 [] vagrant@saio:~$ swift-init container-reconciler once -nv WARNING: Unable to modify max process limit. Running as non-root? Removing stale pid file /var/run/swift/container-reconciler.pid.d Running container-reconciler once...(/etc/swift/container-reconciler.conf.d) container-reconciler: Starting 152398 container-reconciler-ic: Loaded override config for (default): ProxyOverrideOptions({}, {'sorting_method': 'shuffle', 'read_affinity': '', 'write_affinity': '', 'write_affinity_node_count': '2 * replicas', 'write_affinity_handoff_delete_count': None, 'rebalance_missing_suppression_count': 1, 'concurrent_gets': False, 'concurrency_timeout': 0.5, 'concurrent_ec_extra_requests': 0}, app) container-reconciler: pulling items from the queue container-reconciler: checking container 1698868800 container-reconciler: looking for objects in 1698868800 container-reconciler-ic: - - 01/Nov/2023/20/09/03 GET /v1/.misplaced_objects/1698868800%3Fformat%3Djson%26marker%3D%26end_marker%3D%26prefix%3D%26states%3Dlisting HTTP/1.0 404 - Swift%20Container%20Reconciler - - 70 - txc101d856c47e42b6b0d40-006542b05f - 0.0273 - - 1698869343.784635305 1698869343.811890364 0 container-reconciler: looking for containers in .misplaced_objects container-reconciler-ic: - - 01/Nov/2023/20/09/03 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D%26end_marker%3D%26prefix%3D HTTP/1.0 200 - Swift%20Container%20Reconciler - - 95 - tx20e0011d20934621bc2ea-006542b05f - 0.0095 - - 1698869343.816179276 1698869343.825676441 - container-reconciler-ic: - - 01/Nov/2023/20/09/03 GET /v1/.misplaced_objects%3Fformat%3Djson%26marker%3D1698865200%26end_marker%3D%26prefix%3D HTTP/1.0 200 - Swift%20Container%20Reconciler - - 2 - tx1730eec18f764751a2d00-006542b05f - 0.0096 - - 1698869343.828747988 1698869343.838352680 - container-reconciler: checking container 1698865200 container-reconciler: looking for objects in 1698865200 container-reconciler-ic: - - 01/Nov/2023/20/09/03 GET /v1/.misplaced_objects/1698865200%3Fformat%3Djson%26marker%3D%26end_marker%3D%26prefix%3D%26states%3Dlisting HTTP/1.0 200 - Swift%20Container%20Reconciler - - 268 - tx964ddb5202f947c089af9-006542b05f - 0.0146 - - 1698869343.844717503 1698869343.859283209 0 container-reconciler: checking placement for '/AUTH_test/\x00container\x00753a0681-cd2f-4eec-869f-be3b9e27150c/\x00object\x0079b75334-cb71-42ff-94ce-ffd6521e5f5f' (1698868068.214310) in policy_index 1 container-reconciler: checking for '/AUTH_test/\x00container\x00753a0681-cd2f-4eec-869f-be3b9e27150c/\x00object\x0079b75334-cb71-42ff-94ce-ffd6521e5f5f' (1698868068.214310) in destination policy_index 0 container-reconciler-ic: - - 01/Nov/2023/20/09/03 HEAD /v1/AUTH_test%3Fformat%3Djson HTTP/1.0 204 - Swift - - - - tx2d167f88af8f4c96b33dd-006542b05f - 0.0110 GET_ACCOUNT_INFO - 1698869343.909007549 1698869343.920008898 - container-reconciler-ic: - - 01/Nov/2023/20/09/03 GET /v1/.misplaced_objects/1698865200%3Fformat%3Djson%26marker%3D1%253A%252FAUTH_test%252F%2500container%2500753a0681-cd2f-4eec-869f-be3b9e27150c%252F%2500object%250079b75334-cb71-42ff-94ce-ffd6521e5f5f%26end_marker%3D%26prefix%3D%26states%3Dlisting HTTP/1.0 200 - Swift%20Container%20Reconciler - - 2 - tx63b733b00fb6473482121-006542b05f - 0.0648 - - 1698869343.866734266 1698869343.931525230 0 container-reconciler-ic: - - 01/Nov/2023/20/09/03 HEAD /v1/AUTH_test/%2500container%2500753a0681-cd2f-4eec-869f-be3b9e27150c%3Fformat%3Djson%26states%3Dlisting HTTP/1.0 204 - Swift - - - - tx2d167f88af8f4c96b33dd-006542b05f - 0.0141 GET_CONTAINER_INFO - 1698869343.922597170 1698869343.936706305 0 container-reconciler-ic: - - 01/Nov/2023/20/09/03 HEAD /v1/AUTH_test/%2500container%2500753a0681-cd2f-4eec-869f-be3b9e27150c/%2500object%250079b75334-cb71-42ff-94ce-ffd6521e5f5f HTTP/1.0 200 - Swift%20Container%20Reconciler - - - - tx2d167f88af8f4c96b33dd-006542b05f - 0.0497 - - 1698869343.900354385 1698869343.950062990 0 container-reconciler: found_object: '/AUTH_test/\x00container\x00753a0681-cd2f-4eec-869f-be3b9e27150c/\x00object\x0079b75334-cb71-42ff-94ce-ffd6521e5f5f' (1698868068.214310) in policy_index 0 is newer than queue (1698868068.214310) container-reconciler: cleanup_attempt: '/AUTH_test/\x00container\x00753a0681-cd2f-4eec-869f-be3b9e27150c/\x00object\x0079b75334-cb71-42ff-94ce-ffd6521e5f5f' (1698868068.214310) from policy_index 1 (1698868068.21431_0000000000000004) will be deleted container-reconciler-ic: - - 01/Nov/2023/20/09/04 DELETE /v1/AUTH_test/%2500container%2500753a0681-cd2f-4eec-869f-be3b9e27150c/%2500object%250079b75334-cb71-42ff-94ce-ffd6521e5f5f HTTP/1.0 404 - Swift%20Container%20Reconciler - - 70 - tx6d720630f5cb453a97944-006542b05f - 0.1645 - - 1698869343.956005573 1698869344.120494366 1 container-reconciler: cleanup_success: '/AUTH_test/\x00container\x00753a0681-cd2f-4eec-869f-be3b9e27150c/\x00object\x0079b75334-cb71-42ff-94ce-ffd6521e5f5f' (1698868068.214310) was successfully removed from policy_index 1 container-reconciler: success: '/AUTH_test/\x00container\x00753a0681-cd2f-4eec-869f-be3b9e27150c/\x00object\x0079b75334-cb71-42ff-94ce-ffd6521e5f5f' (1698868068.214310) in policy_index 1 was handled successfully container-reconciler: pop_queue: remove '/.misplaced_objects/1698865200/1:/AUTH_test/\x00container\x00753a0681-cd2f-4eec-869f-be3b9e27150c/\x00object\x0079b75334-cb71-42ff-94ce-ffd6521e5f5f' (1698868068.214310) from the queue (1698868068.21431_0000000000000004) container-reconciler: Reconciler Stats: {'found_object': 1, 'cleanup_attempt': 1, 'cleanup_success': 1, 'success': 1, 'pop_queue': 1} container-reconciler: Exited 152398 vagrant@saio:~$ for i in {1..4}; do echo server $i; curl -s "http://127.0.0.${i}:60${i}1/sdb${i}/${MISPLACED_CONTAINER_PART}/.misplaced_objects/${MISPLACED_CONTAINER}?format=json" | python -m json.tool; done # query backend queue server 1 [] server 2 Expecting value: line 1 column 1 (char 0) server 3 [] server 4 [] vagrant@saio:~$ for db in $(find /srv/node*/sdb*/container* -name \*.db); do echo $db; sqlite3 $db -line "select * from object"; done /srv/node1/sdb1/containers/30/555/78dcc7f6428b13d0f23b169461256555/78dcc7f6428b13d0f23b169461256555.db ROWID = 3 name = created_at = 1698868068.21431_4 size = 6 content_type = application/octet-stream etag = 0b4c12d7e0a73840c1c4f148fda3b037 deleted = 0 storage_policy_index = 0 ROWID = 4 name = created_at = 1698868068.21431_3 size = 0 content_type = application/deleted etag = noetag deleted = 1 storage_policy_index = 1 /srv/node1/sdb1/containers/56/cba/e33d8a3403f400d5ec3521d4cda9acba/e33d8a3403f400d5ec3521d4cda9acba.db ROWID = 2 name = 0:/AUTH_test/ created_at = 1698868068.21431_1 size = 0 content_type = application/deleted etag = noetag deleted = 1 storage_policy_index = 0 ROWID = 5 name = 1:/AUTH_test/ created_at = 1698868068.21431_4 size = 0 content_type = application/deleted etag = noetag deleted = 1 storage_policy_index = 0 /srv/node3/sdb3/containers/30/555/78dcc7f6428b13d0f23b169461256555/78dcc7f6428b13d0f23b169461256555.db ROWID = 4 name = created_at = 1698868068.21431_4 size = 6 content_type = application/octet-stream etag = 0b4c12d7e0a73840c1c4f148fda3b037 deleted = 0 storage_policy_index = 0 ROWID = 5 name = created_at = 1698868068.21431_3 size = 0 content_type = application/deleted etag = noetag deleted = 1 storage_policy_index = 1 /srv/node3/sdb3/containers/56/cba/e33d8a3403f400d5ec3521d4cda9acba/e33d8a3403f400d5ec3521d4cda9acba.db ROWID = 2 name = 0:/AUTH_test/ created_at = 1698868068.21431_1 size = 0 content_type = application/deleted etag = noetag deleted = 1 storage_policy_index = 0 ROWID = 5 name = 1:/AUTH_test/ created_at = 1698868068.21431_4 size = 0 content_type = application/deleted etag = noetag deleted = 1 storage_policy_index = 0 /srv/node4/sdb4/containers/30/555/78dcc7f6428b13d0f23b169461256555/78dcc7f6428b13d0f23b169461256555.db ROWID = 4 name = created_at = 1698868068.21431_4 size = 6 content_type = application/octet-stream etag = 0b4c12d7e0a73840c1c4f148fda3b037 deleted = 0 storage_policy_index = 0 ROWID = 5 name = created_at = 1698868068.21431_3 size = 0 content_type = application/deleted etag = noetag deleted = 1 storage_policy_index = 1 /srv/node4/sdb4/containers/56/cba/e33d8a3403f400d5ec3521d4cda9acba/e33d8a3403f400d5ec3521d4cda9acba.db ROWID = 2 name = 0:/AUTH_test/ created_at = 1698868068.21431_1 size = 0 content_type = application/deleted etag = noetag deleted = 1 storage_policy_index = 0 ROWID = 5 name = 1:/AUTH_test/ created_at = 1698868068.21431_4 size = 0 content_type = application/deleted etag = noetag deleted = 1 storage_policy_index = 0