OpenStack Object Storage (swift)

Several Fragment objects are empty after data upload to Swift is complete.

Bug #1921614 reported by Jungwan Oh on 2021-03-28

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Expired	Undecided	Unassigned

Bug Description

I am using it by implementing it with the OpenStack Swift EC policy (9+3). When I downloaded the previously uploaded data, I got a 404 error result.

When looking up the fragment data, it was confirmed that the actual object was empty.

# swift-get-nodes /etc/swift/object.ring.gz AUTH_43ee02272fba4f05aa8dbbcd12f8f1db vcrm-prod-svc-50-raw-archv-20201105 20210106/193022985390242.zip

Account AUTH_43ee02272fba4f05aa8dbbcd12f8f1db
Container vcrm-prod-svc-50-raw-archv-20201105
Object 20210106/193022985390242.zip

Partition 232
Hash 741cae7a0b83ef460055ba39a0f5c855

Server:Port Device 192.168.17.129:6000 sdg
Server:Port Device 192.168.17.127:6000 sdm
Server:Port Device 192.168.17.110:6000 sdm
Server:Port Device 192.168.17.119:6000 sde
Server:Port Device 192.168.17.123:6000 sdm
Server:Port Device 192.168.17.103:6000 sdd
Server:Port Device 192.168.17.130:6000 sdb
Server:Port Device 192.168.17.126:6000 sdh
Server:Port Device 192.168.17.116:6000 sdl
Server:Port Device 192.168.17.114:6000 sdh
Server:Port Device 192.168.17.101:6000 sdc
Server:Port Device 192.168.17.124:6000 sdm
Server:Port Device 192.168.17.102:6000 sdc [Handoff]
Server:Port Device 192.168.17.112:6000 sdf [Handoff]
Server:Port Device 192.168.17.117:6000 sde [Handoff]
Server:Port Device 192.168.17.106:6000 sdh [Handoff]
Server:Port Device 192.168.17.118:6000 sdg [Handoff]
Server:Port Device 192.168.17.121:6000 sde [Handoff]
Server:Port Device 192.168.17.128:6000 sdb [Handoff]
Server:Port Device 192.168.17.111:6000 sdj [Handoff]
Server:Port Device 192.168.17.120:6000 sdk [Handoff]
Server:Port Device 192.168.17.125:6000 sdg [Handoff]
Server:Port Device 192.168.17.113:6000 sde [Handoff]
Server:Port Device 192.168.17.115:6000 sdm [Handoff]

Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 192.168.17.129 "ls -lah ${DEVICE:-/srv/node*}/sdg/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.127 "ls -lah ${DEVICE:-/srv/node*}/sdm/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.110 "ls -lah ${DEVICE:-/srv/node*}/sdm/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.119 "ls -lah ${DEVICE:-/srv/node*}/sde/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.123 "ls -lah ${DEVICE:-/srv/node*}/sdm/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.103 "ls -lah ${DEVICE:-/srv/node*}/sdd/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.130 "ls -lah ${DEVICE:-/srv/node*}/sdb/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.126 "ls -lah ${DEVICE:-/srv/node*}/sdh/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.116 "ls -lah ${DEVICE:-/srv/node*}/sdl/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.114 "ls -lah ${DEVICE:-/srv/node*}/sdh/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.101 "ls -lah ${DEVICE:-/srv/node*}/sdc/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.124 "ls -lah ${DEVICE:-/srv/node*}/sdm/objects/232/855/741cae7a0b83ef460055ba39a0f5c855"
ssh 192.168.17.102 "ls -lah ${DEVICE:-/srv/node*}/sdc/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.112 "ls -lah ${DEVICE:-/srv/node*}/sdf/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.117 "ls -lah ${DEVICE:-/srv/node*}/sde/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.106 "ls -lah ${DEVICE:-/srv/node*}/sdh/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.118 "ls -lah ${DEVICE:-/srv/node*}/sdg/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.121 "ls -lah ${DEVICE:-/srv/node*}/sde/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.128 "ls -lah ${DEVICE:-/srv/node*}/sdb/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.111 "ls -lah ${DEVICE:-/srv/node*}/sdj/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.120 "ls -lah ${DEVICE:-/srv/node*}/sdk/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.125 "ls -lah ${DEVICE:-/srv/node*}/sdg/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.113 "ls -lah ${DEVICE:-/srv/node*}/sde/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]
ssh 192.168.17.115 "ls -lah ${DEVICE:-/srv/node*}/sdm/objects/232/855/741cae7a0b83ef460055ba39a0f5c855" # [Handoff]

=========================================================================

If you make an HTTP GET request for that data, a 404 error is returned.

note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.
[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.129:6000/sdg/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Sun, 28 Mar 2021 09:29:01 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.127:6000/sdm/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 200 OK
Content-Length: 10976254
X-Backend-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Frag-Index: 1
X-Object-Sysmeta-Ec-Content-Length: 98716554
X-Object-Sysmeta-Ec-Etag: 59b9474becb74ad98d9929692fd9618e
X-Backend-Fragments: {"1609929565.18380": [1]}
Last-Modified: Wed, 06 Jan 2021 10:39:26 GMT
Etag: "c1bcca956f9fc65bcee9a146e15c7ba0"
X-Timestamp: 1609929565.18380
X-Backend-Data-Timestamp: 1609929565.18380
X-Backend-Durable-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Scheme: liberasurecode_rs_vand 9+3
Content-Type: application/zip
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Sun, 28 Mar 2021 09:29:01 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.110:6000/sdm/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 200 OK
Content-Length: 10976254
X-Backend-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Frag-Index: 2
X-Object-Sysmeta-Ec-Content-Length: 98716554
X-Object-Sysmeta-Ec-Etag: 59b9474becb74ad98d9929692fd9618e
X-Backend-Fragments: {"1609929565.18380": [2]}
Last-Modified: Wed, 06 Jan 2021 10:39:26 GMT
Etag: "05793ba05f6239e6cc4821da4c91b41d"
X-Timestamp: 1609929565.18380
X-Backend-Data-Timestamp: 1609929565.18380
X-Backend-Durable-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Scheme: liberasurecode_rs_vand 9+3
Content-Type: application/zip
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Sun, 28 Mar 2021 09:29:01 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.119:6000/sde/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 200 OK
Content-Length: 10976254
X-Backend-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Frag-Index: 3
X-Object-Sysmeta-Ec-Content-Length: 98716554
X-Object-Sysmeta-Ec-Etag: 59b9474becb74ad98d9929692fd9618e
Last-Modified: Wed, 06 Jan 2021 10:39:26 GMT
Etag: "8fc9b5dfe795faf9f5c72ae6b14a8dbb"
X-Timestamp: 1609929565.18380
X-Backend-Data-Timestamp: 1609929565.18380
X-Backend-Durable-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Scheme: liberasurecode_rs_vand 9+3
Content-Type: application/zip
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Sun, 28 Mar 2021 09:29:01 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.123:6000/sdm/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Sun, 28 Mar 2021 09:29:02 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.103:6000/sdd/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 200 OK
Content-Length: 10976254
X-Backend-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Frag-Index: 5
X-Object-Sysmeta-Ec-Content-Length: 98716554
X-Object-Sysmeta-Ec-Etag: 59b9474becb74ad98d9929692fd9618e
X-Backend-Fragments: {"1609929565.18380": [5]}
Last-Modified: Wed, 06 Jan 2021 10:39:26 GMT
Etag: "518cfec066588e43c445606b019c758a"
X-Timestamp: 1609929565.18380
X-Backend-Data-Timestamp: 1609929565.18380
X-Backend-Durable-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Scheme: liberasurecode_rs_vand 9+3
Content-Type: application/zip
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Sun, 28 Mar 2021 09:29:02 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.130:6000/sdb/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 507 Insufficient Storage
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Sun, 28 Mar 2021 09:29:02 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.126:6000/sdh/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Sun, 28 Mar 2021 09:29:02 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.116:6000/sdl/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Sun, 28 Mar 2021 09:29:02 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.114:6000/sdh/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Sun, 28 Mar 2021 09:29:02 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.101:6000/sdc/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Sun, 28 Mar 2021 09:29:02 GMT

[root@rndswiftp01 swift(keystone_analysis)]$ curl -g -I -XHEAD "http://192.168.17.124:6000/sdm/232/AUTH_43ee02272fba4f05aa8dbbcd12f8f1db/vcrm-prod-svc-50-raw-archv-20201105/20210106/193022985390242.zip"
HTTP/1.1 200 OK
Content-Length: 10976254
X-Backend-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Frag-Index: 11
X-Object-Sysmeta-Ec-Content-Length: 98716554
X-Object-Sysmeta-Ec-Etag: 59b9474becb74ad98d9929692fd9618e
X-Backend-Fragments: {"1609929565.18380": [11]}
Last-Modified: Wed, 06 Jan 2021 10:39:26 GMT
Etag: "bd37e543b19f305b4a5db23c5e5bf9c7"
X-Timestamp: 1609929565.18380
X-Backend-Data-Timestamp: 1609929565.18380
X-Backend-Durable-Timestamp: 1609929565.18380
X-Object-Sysmeta-Ec-Scheme: liberasurecode_rs_vand 9+3
Content-Type: application/zip
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Sun, 28 Mar 2021 09:29:04 GMT

And now, as a result of testing, when uploading, one or two objects are still lost.

Revision history for this message

clayg (clay-gerrard) wrote on 2021-03-29:

Many of the backend backend storage nodes are having errors, the HTTP/1.1 507 Insufficient Storage
is espeically bad.

Do you have the logs from the original upload? It looks like it was created earlier this year:

>>> print(datetime.fromtimestamp(1609929565.18380))
2021-01-06 04:39:25.183800

What sort of errors do you see in your object-reconstructor logs?

Changed in swift:
status:	New → Incomplete

Revision history for this message

Jungwan Oh (lovethesky) wrote on 2021-03-30:

Yes. The data was uploaded around January, and it was downloaded to use the data, but it did not work, so it appeared during the inspection.
Our Swift usage is about 86% in use.

[2021-03-30 00:24:30] Checking disk usage now
-> http://192.168.17.124:6000/recon/diskusage: timed out
Distribution Graph:
26% 1 *
27% 1 *
28% 1 *
39% 1 *
54% 1 *
57% 1 *
62% 1 *
63% 8 **********
64% 13 ****************
65% 6 *******
66% 2 **
67% 4 *****
77% 1 *
81% 2 **
82% 1 *
83% 3 ***
84% 1 *
85% 9 ***********
86% 20 *************************
87% 15 *******************
88% 28 ***********************************
89% 54 *********************************************************************
90% 48 *************************************************************
91% 53 *******************************************************************
92% 45 *********************************************************
93% 13 ****************
94% 9 ***********
95% 4 *****
97% 1 *
Disk usage: space used: 2409726967361536 of 2775598109818880
Disk usage: space free: 365871142457344 of 2775598109818880
Disk usage: lowest: 26.91%, highest: 97.17%, avg: 86.8182954455%
===============================================================================

And we keep the logs for a week. reconstructor has been running on all nodes since March 26th, and reconstructor has not been run before.
Even now, when uploading, the data of the fragment object is saved as missing one by one.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2021-05-29:

[Expired for OpenStack Object Storage (swift) because there has been no activity for 60 days.]

Changed in swift:
status:	Incomplete → Expired

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.