Comment 4 for bug 1889661

Revision history for this message
toto (locklock) wrote :

Hello !

My bad, you are correct :)

We did set the write affinity to "none" instead of removing it from our configuration, which was perhaps causing the same issue, we probably did a mistake in the configuration when we were testing your workaround.
Now that we have removed it from our configuration, it's all good.
As you can see from the below logs, we are seeing the correct behaviour now.

But it's quite impacting for us, as we are using the region feature in order to provider a better latency/response time to our internal clients to manage their object on different DCs.
It would be great indeed to have this functionnality back, with the write_affinity, which was working perfectly with Kilo version.
As for now, it's a regression for us to not have this post_as_copy feature anymore.

Do you know if it will be difficult to develop a working fix for this ? Or to reimplement it by any chance ?
We would be happy to help and try the fix you can provide to "beta-test" it.

In the meantime, we will probably remove the write_affinity on our production environnement due to this bug, which will cause some latency for some of our clients.

Thanks in advance for your answer.

----------

PUT objet

./chkjmeter.pl -s dc1 -e preprod -p TESTCONTAINER -c NEW -o testsanswriteaffinity -D 3 -a 1000

--------------------------------------------------
ENV :preprod
SITE :dc1
RESOURCE :NEW/testsanswriteaffinity.dc1
URL :http://api.swift.com/v1/AUTH_TESTCONTAINER
--------------------------------------------------
OBJECT CREATE OK CHECK OK http://api.swift.com/v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1
META ADD OK CHECK OK expired on Tue Aug 11 14:21:14 2020, NOW Tue Aug 11 14:04:35 2020

Proxy DC 1
Aug 11 14:04:30 localhost swift-proxy: 127.0.0.1 127.0.0.1 11/Aug/2020/12/04/30 PUT /v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1 HTTP/1.0 201 - HTTP-Tiny/0.070 gAAAAABfMokIHh2x... 93 - - tx9f716ff5a39a447eb3d03-005f32894d - 0.3986 - - 1597147469.603407621 1597147470.001974583 0
Aug 11 14:04:30 localhost swift-proxy: 127.0.0.1 127.0.0.1 11/Aug/2020/12/04/30 GET /v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1 HTTP/1.0 200 - HTTP-Tiny/0.070 gAAAAABfMokIHh2x... - 93 - tx4a6f045b844e4f12bd15d-005f32894e - 0.0176 - - 1597147470.022356033 1597147470.039961100 0
Aug 11 14:04:34 localhost swift-proxy: 127.0.0.1 127.0.0.1 11/Aug/2020/12/04/34 POST /v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1 HTTP/1.0 202 - HTTP-Tiny/0.070 gAAAAABfMokIHh2x... - 76 - txc4d7538e50394b9a93a46-005f328951 - 0.5977 - x-delete-at:1597148474 1597147474.152446032 1597147474.750192642 0
Aug 11 14:04:35 localhost swift-proxy: 127.0.0.1 127.0.0.1 11/Aug/2020/12/04/35 HEAD /v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1 HTTP/1.0 200 - HTTP-Tiny/0.070 gAAAAABfMokIHh2x... - - - tx213479caa1464bfa8c15d-005f328952 - 0.0184 - - 1597147475.246030569 1597147475.264401913 0

Storage DC 1

Aug 11 14:04:29 storage01.dc1 swift-object: X - - [11/Aug/2020:12:04:29 +0000] "PUT /d6/3633/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" 201 - "PUT http://api.swift.com/v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" "tx9f716ff5a39a447eb3d03-005f32894d" "proxy-server 294" 0.1916 "-" 111 0
Aug 11 14:04:30 storage01.dc1 swift-object: X - - [11/Aug/2020:12:04:30 +0000] "GET /d6/3633/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" 200 93 "GET http://api.swift.com/v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" "tx4a6f045b844e4f12bd15d-005f32894e" "proxy-server 294" 0.0020 "-" 111 0
ug 11 14:04:34 storage01.dc1 swift-object: X - - [11/Aug/2020:12:04:34 +0000] "POST /d6/3633/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" 202 76 "POST http://api.swift.com/v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" "txc4d7538e50394b9a93a46-005f328951" "proxy-server 292" 0.3339 "-" 113 0
Aug 11 14:04:35 storage01.dc1 swift-object: X - - [11/Aug/2020:12:04:35 +0000] "HEAD /d6/3633/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" 200 93 "HEAD http://api.swift.com/v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" "tx213479caa1464bfa8c15d-005f328952" "proxy-server 285" 0.0014 "-" 113 0

Storage DC 2
Aug 11 14:04:29 storage01.dc2 swift-object: X - - [11/Aug/2020:12:04:29 +0000] "PUT /d4/3633/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" 201 - "PUT http://api.swift.com/v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" "tx9f716ff5a39a447eb3d03-005f32894d" "proxy-server 294" 0.1458 "-" 110 0
Aug 11 14:04:34 storage01.dc2 swift-object: X - - [11/Aug/2020:12:04:34 +0000] "POST /d4/3633/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" 202 76 "POST http://api.swift.com/v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" "txc4d7538e50394b9a93a46-005f328951" "proxy-server 292" 0.5770 "-" 111 0

Storage DC 3
Aug 11 14:04:29 storage01.dc3 swift-object: X - - [11/Aug/2020:12:04:29 +0000] "PUT /d5/3633/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" 201 - "PUT http://api.swift.com/v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" "tx9f716ff5a39a447eb3d03-005f32894d" "proxy-server 294" 0.2208 "-" 113 0
ug 11 14:04:34 storage01.dc3 swift-object: X - - [11/Aug/2020:12:04:34 +0000] "POST /d5/3633/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" 202 76 "POST http://api.swift.com/v1/AUTH_TESTCONTAINER/NEW/testsanswriteaffinity.dc1" "txc4d7538e50394b9a93a46-005f328951" "proxy-server 292" 0.3796 "-" 111 0

----------

/# swift-get-nodes /etc/swift/object.ring.gz NEW "testsanswriteaffinity.dc1"
Warning: account/container specified but ring not named "container"

Account NEW
Container testsanswriteaffinity.dc1
Object None

Partition 3605
Hash 70aaa9106791d35ef322d6f82df65311

Server:Port Device X:6000 d5
Server:Port Device X:6000 d5
Server:Port Device X:6000 d3
Server:Port Device X:6000 d6 [Handoff]
Server:Port Device X:6000 d4 [Handoff]
Server:Port Device X:6000 d4 [Handoff]

curl -g -I -XHEAD "http://X:6000/d5/3605/NEW/testsanswriteaffinity.dc1"
curl -g -I -XHEAD "http://X:6000/d5/3605/NEW/testsanswriteaffinity.dc1"
curl -g -I -XHEAD "http://X:6000/d3/3605/NEW/testsanswriteaffinity.dc1"
curl -g -I -XHEAD "http://X:6000/d6/3605/NEW/testsanswriteaffinity.dc1" # [Handoff]
curl -g -I -XHEAD "http://X:6000/d4/3605/NEW/testsanswriteaffinity.dc1" # [Handoff]
curl -g -I -XHEAD "http://X:6000/d4/3605/NEW/testsanswriteaffinity.dc1" # [Handoff]

Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh X "ls -lah ${DEVICE:-/srv/node*}/d5/containers/3605/311/70aaa9106791d35ef322d6f82df65311"
ssh X "ls -lah ${DEVICE:-/srv/node*}/d5/containers/3605/311/70aaa9106791d35ef322d6f82df65311"
ssh X "ls -lah ${DEVICE:-/srv/node*}/d3/containers/3605/311/70aaa9106791d35ef322d6f82df65311"
ssh X "ls -lah ${DEVICE:-/srv/node*}/d6/containers/3605/311/70aaa9106791d35ef322d6f82df65311" # [Handoff]
ssh X "ls -lah ${DEVICE:-/srv/node*}/d4/containers/3605/311/70aaa9106791d35ef322d6f82df65311" # [Handoff]
ssh X "ls -lah ${DEVICE:-/srv/node*}/d4/containers/3605/311/70aaa9106791d35ef322d6f82df65311" # [Handoff]

note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.