rebalance error in _finalize_durable

[Anton]

I added a new disk to the ring
I launched my rebalance

and I get errors

Jul 8 12:08:34 sw3 object-server: ERROR __call__ error with PUT /cold6/5/AUTH_69ffc05e682146879737ae4d2e229c4a/ftpbackup_segments/monthly/2020-01-15/accounts/d.tar.gz/1579044380.177554/0/536870912/00000006 : #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/obj/server.py", line 1282, in __call__#012 res = getattr(self, req.method)(req)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 1839, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/swift/obj/server.py", line 1016, in PUT#012 writer.commit(request.timestamp)#012 File "/usr/lib/python2.7/site-packages/swift/obj/diskfile.py", line 3091, in commit#012 self._finalize_durable, data_file_path, durable_data_file_path)#012 File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute#012 six.reraise(c, e, tb)#012 File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker#012 rv = meth(*args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/swift/obj/diskfile.py", line 3071, in _finalize_durable#012 raise exc#012DiskFileError: Problem making data file durable /srv/node/object3/cold6/objects/5/8df/a99929c68e4cb2d7bc269abdb729b8df/1579044430.84248#0#d.data ([Errno 2] No such file or directory)

I check the file is there
[root@sw3 sravnenie]# ls -l /srv/node/object3/cold6/objects/5/8df/a99929c68e4cb2d7bc269abdb729b8df/1579044430.84248#0#d.data
-rwxr-xr-x 1 swift swift 268476416 Jul 8 12:58 /srv/node/object3/cold6/objects/5/8df/a99929c68e4cb2d7bc269abdb729b8df/1579044430.84248#0#d.data

I'm looking at the object info
[root@sw3 sravnenie]# swift-object-info /srv/node/object3/cold6/objects/5/8df/a99929c68e4cb2d7bc269abdb729b8df/1579044430.84248#0#d.data|grep 1.13
Server:Port Device 10.0.1.13:6067 cold6
Server:Port Device 10.0.1.13:6064 cold3 [Handoff]
curl -g -I -XHEAD "http://10.0.1.13:6067/cold6/5/AUTH_69ffc05e682146879737ae4d2e229c4a/ftpbackup_segments/monthly/2020-01-15/accounts/d.tar.gz/1579044380.177554/0/536870912/00000006" -H "X-Backend-Storage-Policy-Index: 0"
curl -g -I -XHEAD "http://10.0.1.13:6064/cold3/5/AUTH_69ffc05e682146879737ae4d2e229c4a/ftpbackup_segments/monthly/2020-01-15/accounts/d.tar.gz/1579044380.177554/0/536870912/00000006" -H "X-Backend-Storage-Policy-Index: 0" # [Handoff]
ssh 10.0.1.13 "ls -lah ${DEVICE:-/srv/node*}/cold6/objects/5/8df/a99929c68e4cb2d7bc269abdb729b8df"
ssh 10.0.1.13 "ls -lah ${DEVICE:-/srv/node*}/cold3/objects/5/8df/a99929c68e4cb2d7bc269abdb729b8df" # [Handoff]

the segment is available
[root@sw3 sravnenie]# curl -g -I -XHEAD "http://10.0.1.13:6067/cold6/5/AUTH_69ffc05e682146879737ae4d2e229c4a/ftpbackup_segments/monthly/2020-01-15/accounts/d.tar.gz/1579044380.177554/0/536870912/00000006" -H "X-Backend-Storage-Policy-Index: 0"
HTTP/1.1 200 OK
Content-Length: 268476416
X-Backend-Timestamp: 1579044430.84248
X-Object-Sysmeta-Ec-Frag-Index: 0
X-Object-Sysmeta-Ec-Content-Length: 536870912
X-Object-Sysmeta-Ec-Etag: 7214ae92e90e46d3b2520d3b803ccd83
X-Backend-Fragments: {"1579044430.84248": [0]}
Last-Modified: Tue, 14 Jan 2020 23:27:11 GMT
Etag: "e834dbae7d276184206e1c4a153f2074"
X-Timestamp: 1579044430.84248
X-Backend-Data-Timestamp: 1579044430.84248
X-Backend-Durable-Timestamp: 1579044430.84248
X-Object-Sysmeta-Ec-Scheme: liberasurecode_rs_vand 2+2
Content-Type: application/octet-stream
X-Object-Sysmeta-Ec-Segment-Size: 1048576
Date: Wed, 08 Jul 2020 16:17:53 GMT

Why does an error appear in the log?
How do I fix the error?

[clayg (clay-gerrard)]

Can you confirm what version of swift you're running, or if it has this fix related to that traceback: https://review.opendev.org/#/c/722128/

[Anton (a.porabkovich)]

swift (2.21.1)

the patch was not applied

if I install the patch, will the problem be fixed? or will you need to do some more operations?