Hi
I encountered an error downloading the file
[root@storage-1 swiftfs(xenbckp:xenbckp)]$ swift --debug stat backup vpsbackup/config-111.dat
DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to https://api-storage/v3/auth/tokens
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api-storage:443
DEBUG:urllib3.connectionpool:https://api-storage:443 "POST /v3/auth/tokens HTTP/1.1" 201 1757
DEBUG:keystoneclient.auth.identity.v3.base:{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "2ef9173753d2439bb42c01dbf7061088", "name": "SwiftOperator"}], "expires_at": "2020-10-23T10:01:01.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6c6703bdbb3c4a9bb4991547a2bb52d9", "name": "xenbckp"}, "catalog": [{"endpoints": [{"url": "http://10.1.1.2:5000/v3/", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "4e22226594ab41d7a7a9d712e7222964"}, {"url": "https://api-storage/v3/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "83f13ad5f8584f678cba9000aa005f75"}, {"url": "http://10.1.1.2:5000/v3/", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "adb73cdf3b6045de83471dfe687691e8"}], "type": "identity", "id": "246e9d262e0547568a22de34ff82f356", "name": "keystone"}, {"endpoints": [{"url": "http://10.1.1.2:8080/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "71ef44380d2444cd956bec8bc280738b"}, {"url": "http://10.1.1.2:8080/v1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "d52430023dc747349ac3833f41e29c20"}, {"url": "https://storage/v1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "d61d39148b5e42ba99e4495ee6ec53d5"}], "type": "object-store", "id": "56649daa07a84ac4a8a6d3c3b54e0715", "name": "swift"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "xenbckp", "id": "848580da8faa49cc82e7edfb96621304"}, "audit_ids": ["O9Zf3gDYQKWF7EVI1KAGaw"], "issued_at": "2020-10-22T10:01:01.000000Z"}}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): storage:443
DEBUG:urllib3.connectionpool:https://storage:443 "HEAD /v1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat HTTP/1.1" 200 0
DEBUG:swiftclient:REQ: curl -i https://storage/v1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat -I -H "X-Auth-Token: gAAAAABfkVhdauRh3Ba7q9UD3gssH3WnPBDIRUBVqhDBOP7kREyq-a5h-L0WobrSrRR176ZKitv4YExZy2q-zx2Tfc7K8EzWCW-x5M6ip7VRODlil16FfaK7YUjEOPDRlpwEeOINqeq6VulQB9UYnH7_Jcd8fYITXQmZRTpHVGIigIVq9kOoeNU"
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'content-length': u'148', u'accept-ranges': u'bytes', u'x-manifest-etag': u'b0f6aeac0161ef43b140a32210f682a9', u'last-modified': u'Fri, 09 Oct 2020 19:28:55 GMT', u'etag': u'"65044e1e19730ad0f234a51c2db9ace3"', u'x-timestamp': u'1602271734.07438', u'x-trans-id': u'txce1bb0e77d8f4919aeb95-005f91585d', u'date': u'Thu, 22 Oct 2020 10:01:01 GMT', u'x-static-large-object': u'True', u'content-type': u'application/binary', u'x-openstack-request-id': u'txce1bb0e77d8f4919aeb95-005f91585d'}
Account: AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9
Container: backup
Object: vpsbackup/config-111.dat
Content Type: application/binary
Content Length: 148
Last Modified: Fri, 09 Oct 2020 19:28:55 GMT
ETag: "65044e1e19730ad0f234a51c2db9ace3"
Accept-Ranges: bytes
X-Manifest-Etag: b0f6aeac0161ef43b140a32210f682a9
X-Timestamp: 1602271734.07438
X-Trans-Id: txce1bb0e77d8f4919aeb95-005f91585d
X-Static-Large-Object: True
X-Openstack-Request-Id: txce1bb0e77d8f4919aeb95-005f91585d
[root@storage-1 swiftfs(xenbckp:xenbckp)]$ swift download backup vpsbackup/config-111.dat
Error downloading object 'backup/vpsbackup/config-111.dat': Object GET failed: https://storage/v1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat 409 Conflict [first 60 chars of response] <html><h1>Conflict</h1><p>There was a conflict when trying t
I'm looking at where the file is located
[root@storage-sw2 swift]# swift-get-nodes /etc/swift/object.ring.gz AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat
Account AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9
Container backup
Object vpsbackup/config-111.dat
Partition 1
Hash 22524e624956d50b4f31f89b4ea7643b
Server:Port Device 10.0.1.11:6068 cold7
Server:Port Device 10.0.1.12:6067 cold6
Server:Port Device 10.0.1.14:6068 cold7
Server:Port Device 10.0.1.13:6067 cold6
Server:Port Device 10.0.1.11:6067 cold6 [Handoff]
Server:Port Device 10.0.1.12:6065 cold4 [Handoff]
Server:Port Device 10.0.1.14:6065 cold4 [Handoff]
Server:Port Device 10.0.1.11:6065 cold4 [Handoff]
curl -g -I -XHEAD "http://10.0.1.11:6068/cold7/1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat"
curl -g -I -XHEAD "http://10.0.1.12:6067/cold6/1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat"
curl -g -I -XHEAD "http://10.0.1.14:6068/cold7/1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat"
curl -g -I -XHEAD "http://10.0.1.13:6067/cold6/1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat"
curl -g -I -XHEAD "http://10.0.1.11:6067/cold6/1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat" # [Handoff]
curl -g -I -XHEAD "http://10.0.1.12:6065/cold4/1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat" # [Handoff]
curl -g -I -XHEAD "http://10.0.1.14:6065/cold4/1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat" # [Handoff]
curl -g -I -XHEAD "http://10.0.1.11:6065/cold4/1/AUTH_6c6703bdbb3c4a9bb4991547a2bb52d9/backup/vpsbackup/config-111.dat" # [Handoff]
[root@storage-ru-2-sw2 swift]# ls -l /srv/node/object2//cold6/objects/1/43b/22524e624956d50b4f31f89b4ea7643b/1602*
-rwxr-xr-x 1 swift swift 204 Oct 9 19:28 /srv/node/object2//cold6/objects/1/43b/22524e624956d50b4f31f89b4ea7643b/1602271734.07438#1#d.data
-rwxr-xr-x 1 swift swift 0 Oct 16 20:28 /srv/node/object2//cold6/objects/1/43b/22524e624956d50b4f31f89b4ea7643b/1602880086.90647#1.data
-rwxr-xr-x 1 swift swift 0 Oct 16 20:28 /srv/node/object2//cold6/objects/1/43b/22524e624956d50b4f31f89b4ea7643b/1602880096.99298#1.data
-rwxr-xr-x 1 swift swift 0 Oct 16 20:28 /srv/node/object2//cold6/objects/1/43b/22524e624956d50b4f31f89b4ea7643b/1602880107.07151#1.data
-rwxr-xr-x 1 swift swift 0 Oct 16 20:28 /srv/node/object2//cold6/objects/1/43b/22524e624956d50b4f31f89b4ea7643b/1602880117.14871#1.data
-rwxr-xr-x 1 swift swift 0 Oct 16 20:28 /srv/node/object2//cold6/objects/1/43b/22524e624956d50b4f31f89b4ea7643b/1602880127.27188#1.data
-rwxr-xr-x 1 swift swift 0 Oct 16 20:28 /srv/node/object2//cold6/objects/1/43b/22524e624956d50b4f31f89b4ea7643b/1602880137.36603#1.data
But this can't be, there shouldn't be many segments in a single hash directory.
How do I fix this?
So the ...#1.data files are non-durable fragments -- which is to say, during the PUT,
* the proxy received 100 Continue responses from at least ndata object servers,
* it sent the data and footers out, but then
* it didn't receive enough acks to tell the object servers to mark the fragments durable.
(Alternatively, the proxy *did* attempt to send it, but the message was lost so the object-server never acted on it. In that case, hopefully some of the other object-servers did receive the message and the reconstructor will eventually make the other fragments for that timestamp durable as well.)
At the moment, only the ...#1#d.data file is durable, so only fragments with a timestamp before 1602271734.07438 can be cleaned up. The object-server keeps the non-durable fragments around for a time (see reclaim_age in object-
To resolve it, just wait -- eventually the non-durable frags should either be deleted or marked durable.