ssync subrequest failed with 499

Bug #1890574 reported by Anton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
New
Undecided
Unassigned

Bug Description

Errors appeared in my log:

Aug 6 10:21:05 sw2 object-server: ssync subrequest failed with 499: PUT /cold5/0/AUTH_cff692e324/%D0%97%D0%B0%D0%BA%D0%B0%D0%B7%D1%8B/%D0%A2%D1%80%D0%B5%D0%B1%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D1
%8F%20%D0%BE%20%D0%BF%D0%BE%D0%B3%D0%B0%D1%88%D0%B5%D0%BD%D0%B8%D0%B8%20%D0%B4%D0%BE%D0%BB%D0%B3%D0%B0%20%28%D0%B8%D1%81%D0%BA%D0%BE%D0%B2%D0%BE%D0%B5%29/28174.pdf

Aug 6 10:21:05 sw2 object-server: STDERR: Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 599, in handle_one_response#012 write(b'')#012 File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 538, in write#012 wfile.flush()#012 File "/usr/lib64/python2.7/socket.py", line 303, in flush#012 self._sock.sendall(view[write_offset:write_offset+buffer_size])#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 403, in sendall#012 tail = self.send(data, flags)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 397, in send#012 return self._send_loop(self.fd.send, data, flags)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in _send_loop#012 return send_method(data, *args)#012error: [Errno 32] Broken pipe

All services and all servers are available.
Information about the object:
swift-object-info /srv/node/object2/cold5/objects/0/14a/05ca2b8d9ad708ea9dba3978ca49f14a/1593157868.69436#1#d.data Path: /AUTH_cff692e3df0a4f83a24/Зазы/Треия (lkj)/28174.pdf
  Account: AUTH_cff692e3df0a4f8389b24
  Container: Зазы
  Object: Треия (lkj)/28174.pdf
  Object hash: 05ca2b8d9ad708ea9dba3978ca49f14a
Content-Type: application/pdf
Timestamp: 2020-06-26T07:51:08.694360 (1593157868.69436)
System Metadata:
  X-Object-Sysmeta-Ec-Content-Length: 0
  X-Object-Sysmeta-Ec-Etag: d41d8cd98f00b204e9800998ecf8427e
  X-Object-Sysmeta-Ec-Frag-Index: 1
  X-Object-Sysmeta-Ec-Scheme: liberasurecode_rs_vand 2+2
  X-Object-Sysmeta-Ec-Segment-Size: 1048576
Transient System Metadata:
  No metadata found
User Metadata:
  No metadata found
Other Metadata:
  X-Object-Manifest: %D0%97%D0%B0%D0%BA%D0%B0%D0%B7%D1%8B_segments/%D0%A2%D1%80%D0%B5%D0%B1%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D1%8F%20%D0%BE%20%D0%BF%D0%BE%D0%B3%D0%B0%D1%88%D0%B5%D0%BD%D0%B8%D0%B8%20%D0%B4%D0%BE%D0%BB%D0%B3%D0%B0%20%28%D0%B8%D1%81%D0%BA%D0%BE%D0%B2%D0%BE%D0%B5%29/28174.pdf/slo/1593157868.688028/67108864
ETag: d41d8cd98f00b204e9800998ecf8427e (valid)
Content-Length: 0 (valid)
Partition 0
Hash 05ca2b8d9ad708ea9dba3978ca49f14a

According to the information, the file was created more than a month ago.
2020-06-26T07:51:08.694360

A month ago, the file migrated to another disk:
-rwxr-xr-x 1 swift swift 0 Jul 6 03:53 /srv/node/object2/cold5/objects/0/14a/05ca2b8d9ad708ea9dba3978ca49f14a/1593157868.69436#1#d.data

Why is the PUT command called and it ends with an error?
How do I know what's wrong with an object?

Revision history for this message
Anton (a.porabkovich) wrote :

I found the reason for the error.
A DLO object was uploaded to the cloud storage.
Then the object type was changed to SLO.
But for some reason, the change only occurred on one server.
On all other servers, the object remains DLO.

Currently on the same server:
Content-Type: application/pdf;swift_bytes=1089467
Timestamp: 2020-06-26T07:51:09.214980 (1593157869.21498)
System Metadata:
  X-Object-Sysmeta-Container-Update-Override-Etag: 8d3a35d5c6405e655b58aaa4f9964e2d; slo_etag=ae9051fbff7dc0499ee00ce3bcde9fff
  X-Object-Sysmeta-Ec-Content-Length: 462
  X-Object-Sysmeta-Ec-Etag: 8d3a35d5c6405e655b58aaa4f9964e2d
  X-Object-Sysmeta-Ec-Frag-Index: 3
  X-Object-Sysmeta-Ec-Scheme: liberasurecode_rs_vand 2+2
  X-Object-Sysmeta-Ec-Segment-Size: 1048576
  X-Object-Sysmeta-Slo-Etag: ae9051fbff7dc0499ee00ce3bcde9fff
  X-Object-Sysmeta-Slo-Size: 1089467
Transient System Metadata:
  No metadata found
User Metadata:
  No metadata found
Other Metadata:
  X-Static-Large-Object: True
ETag: 537a67d6e47c9b22743dec30d8036b9d (valid)
Content-Length: 312 (valid)
Partition 0
Hash 05ca2b8d9ad708ea9dba3978ca49f14a

On all other servers:
Content-Type: application/pdf
Timestamp: 2020-06-26T07:51:08.694360 (1593157868.69436)
System Metadata:
  X-Object-Sysmeta-Ec-Content-Length: 0
  X-Object-Sysmeta-Ec-Etag: d41d8cd98f00b204e9800998ecf8427e
  X-Object-Sysmeta-Ec-Frag-Index: 1
  X-Object-Sysmeta-Ec-Scheme: liberasurecode_rs_vand 2+2
  X-Object-Sysmeta-Ec-Segment-Size: 1048576
Transient System Metadata:
  No metadata found
User Metadata:
  No metadata found
Other Metadata:
  X-Object-Manifest: %D0%97%D0%B0%D0%BA%D0%B0%D0%B7%D1%8B_segments/%D0%A2%D1%80%D0%B5%D0%B1%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D1%8F%20%D0%BE%20%D0%BF%D0%BE%D0%B3%D0%B0%D1%88%D0%B5%D0%BD%D0%B8%D0%B8%20%D0%B4%D0%BE%D0%BB%D0%B3%D0%B0%20%28%D0%B8%D1%81%D0%BA%D0%BE%D0%B2%D0%BE%D0%B5%29/28174.pdf/slo/1593157868.688028/67108864
ETag: d41d8cd98f00b204e9800998ecf8427e (valid)
Content-Length: 0 (valid)
Partition 0
Hash 05ca2b8d9ad708ea9dba3978ca49f14a

Only on one server the object became SLO, and on all other servers the object remained DLO
Why did this happen?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.