maybe "get_to_final_state" isn't running the object-updater - i have some asyncs... and maybe no rows in the .misplaced_objects container? FAILED swift/test/probe/test_container_merge_policy_index.py::TestReservedNamespaceMergePolicyIndex::test_reconciler_move_object_twice - AssertionError: Found unexpected object {'bytes': 0, 'hash': '1698865367.58697_3', 'name': '1:/AUTH_test/\x00container\x00ade7d7b2-8cc0-40e0-b345-0beca938d4cc/\x00object\x000aa82... ====================================================================== 1 failed, 6 warnings in 125.37s (0:02:05) ====================================================================== vagrant@saio:~$ curl http://localhost:8090/v1/.misplaced_objects 1698865200 vagrant@saio:~$ for i in {1..10}; do curl "http://localhost:8090/v1/.misplaced_objects/1698865200?format=json"; done [][][][][][][][][][]vagrant@saio:~$ vagrant@saio:~$ for f in $(find /srv/node*/sdb*/container* -name \*.pending); do echo $f; cat $f; done /srv/node1/sdb1/containers/56/930/e1bf38d2ae3a333f5396d47ed4bf2930/e1bf38d2ae3a333f5396d47ed4bf2930.db.pending /srv/node3/sdb3/containers/56/930/e1bf38d2ae3a333f5396d47ed4bf2930/e1bf38d2ae3a333f5396d47ed4bf2930.db.pending /srv/node3/sdb3/containers/56/cba/e33d8a3403f400d5ec3521d4cda9acba/e33d8a3403f400d5ec3521d4cda9acba.db.pending /srv/node4/sdb4/containers/56/930/e1bf38d2ae3a333f5396d47ed4bf2930/e1bf38d2ae3a333f5396d47ed4bf2930.db.pending /srv/node4/sdb4/containers/56/cba/e33d8a3403f400d5ec3521d4cda9acba/e33d8a3403f400d5ec3521d4cda9acba.db.pending vagrant@saio:~$ for db in $(find /srv/node*/sdb*/container* -name \*.db); do echo $db; sqlite3 $db "select * from object"; done /srv/node1/sdb1/containers/56/930/e1bf38d2ae3a333f5396d47ed4bf2930/e1bf38d2ae3a333f5396d47ed4bf2930.db 2||1698865367.58697|6|application/octet-stream|0b4c12d7e0a73840c1c4f148fda3b037|0|0 /srv/node2/sdb2/containers/56/930/e1bf38d2ae3a333f5396d47ed4bf2930/e1bf38d2ae3a333f5396d47ed4bf2930.db /srv/node3/sdb3/containers/56/930/e1bf38d2ae3a333f5396d47ed4bf2930/e1bf38d2ae3a333f5396d47ed4bf2930.db 4||1698865367.58697_4|6|application/octet-stream|0b4c12d7e0a73840c1c4f148fda3b037|0|0 5||1698865367.58697_3|0|application/deleted|noetag|1|1 /srv/node3/sdb3/containers/56/cba/e33d8a3403f400d5ec3521d4cda9acba/e33d8a3403f400d5ec3521d4cda9acba.db 2|0:/AUTH_test/|1698865367.58697_1|0|application/deleted|noetag|1|0 4|1:/AUTH_test/|1698865367.58697_3|0|application/x-delete|1698865367.58697_3|0|0 /srv/node4/sdb4/containers/56/930/e1bf38d2ae3a333f5396d47ed4bf2930/e1bf38d2ae3a333f5396d47ed4bf2930.db 4||1698865367.58697_4|6|application/octet-stream|0b4c12d7e0a73840c1c4f148fda3b037|0|0 5||1698865367.58697_3|0|application/deleted|noetag|1|1 /srv/node4/sdb4/containers/56/cba/e33d8a3403f400d5ec3521d4cda9acba/e33d8a3403f400d5ec3521d4cda9acba.db 2|0:/AUTH_test/|1698865367.58697_1|0|application/deleted|noetag|1|0 4|1:/AUTH_test/|1698865367.58697_3|0|application/deleted|noetag|1|0 vagrant@saio:~$ for async in $(find /srv/node*/sdb*/async* -type f); do echo $async; cat $async | python -c "import sys, json, pickle; print(json.dumps(pickle.load(sys.stdin.buffer), indent=2))"; done /srv/node2/sdb2/async_pending/b57/eef0ca1efbca663b97c2f4d5eac4fb57-1698865367.58697_0000000000000004 { "op": "PUT", "account": "AUTH_test", "container": "\u0000container\u0000ade7d7b2-8cc0-40e0-b345-0beca938d4cc", "obj": "\u0000object\u00000aa82695-2677-43f0-80a7-f8f05e7ba617", "headers": { "X-Size": "6", "X-Content-Type": "application/octet-stream", "X-Timestamp": "1698865367.58697_0000000000000004", "X-Etag": "0b4c12d7e0a73840c1c4f148fda3b037", "X-Trans-Id": "txdf11764872844abbb1f67-006542a109", "Referer": "PUT http://localhost/sdb2/59/AUTH_test/%00container%00ade7d7b2-8cc0-40e0-b345-0beca938d4cc/%00object%000aa82695-2677-43f0-80a7-f8f05e7ba617", "X-Backend-Storage-Policy-Index": "0", "User-Agent": "object-server 143687" }, "successes": [ 2, 3 ] } /srv/node3/sdb3/async_pending-1/b57/eef0ca1efbca663b97c2f4d5eac4fb57-1698865367.58697_0000000000000003 { "op": "DELETE", "account": "AUTH_test", "container": "\u0000container\u0000ade7d7b2-8cc0-40e0-b345-0beca938d4cc", "obj": "\u0000object\u00000aa82695-2677-43f0-80a7-f8f05e7ba617", "headers": { "X-Timestamp": "1698865367.58697_0000000000000003", "X-Trans-Id": "txcf7d97f4a4544af69d3ee-006542a109", "Referer": "DELETE http://localhost/sdb3/59/AUTH_test/%00container%00ade7d7b2-8cc0-40e0-b345-0beca938d4cc/%00object%000aa82695-2677-43f0-80a7-f8f05e7ba617", "X-Backend-Storage-Policy-Index": "1", "User-Agent": "object-server 143684" }, "successes": [ 2, 3 ] } /srv/node3/sdb7/async_pending-1/b57/eef0ca1efbca663b97c2f4d5eac4fb57-1698865367.58697_0000000000000002 { "op": "PUT", "account": "AUTH_test", "container": "\u0000container\u0000ade7d7b2-8cc0-40e0-b345-0beca938d4cc", "obj": "\u0000object\u00000aa82695-2677-43f0-80a7-f8f05e7ba617", "headers": { "X-Size": "6", "X-Content-Type": "application/octet-stream", "X-Timestamp": "1698865367.58697_0000000000000002", "X-Etag": "0b4c12d7e0a73840c1c4f148fda3b037", "X-Trans-Id": "tx933b9bb3077a4b4e8f208-006542a0de", "Referer": "PUT http://localhost/sdb7/59/AUTH_test/%00container%00ade7d7b2-8cc0-40e0-b345-0beca938d4cc/%00object%000aa82695-2677-43f0-80a7-f8f05e7ba617", "X-Backend-Storage-Policy-Index": "1", "User-Agent": "object-server 143684" }, "successes": [ 2, 3 ] } /srv/node4/sdb4/async_pending/b57/eef0ca1efbca663b97c2f4d5eac4fb57-1698865367.58697_0000000000000001 { "op": "DELETE", "account": "AUTH_test", "container": "\u0000container\u0000ade7d7b2-8cc0-40e0-b345-0beca938d4cc", "obj": "\u0000object\u00000aa82695-2677-43f0-80a7-f8f05e7ba617", "headers": { "X-Timestamp": "1698865367.58697_0000000000000001", "X-Trans-Id": "tx7aa31d673a684c23ba275-006542a0de", "Referer": "DELETE http://localhost/sdb4/59/AUTH_test/%00container%00ade7d7b2-8cc0-40e0-b345-0beca938d4cc/%00object%000aa82695-2677-43f0-80a7-f8f05e7ba617", "X-Backend-Storage-Policy-Index": "0", "User-Agent": "object-server 143685" }, "successes": [ 2, 3 ] }