We did set the write affinity to "none" instead of removing it from our configuration, which was perhaps causing the same issue, we probably did a mistake in the configuration when we were testing your workaround.
Now that we have removed it from our configuration, it's all good.
As you can see from the below logs, we are seeing the correct behaviour now.
But it's quite impacting for us, as we are using the region feature in order to provider a better latency/response time to our internal clients to manage their object on different DCs.
It would be great indeed to have this functionnality back, with the write_affinity, which was working perfectly with Kilo version.
As for now, it's a regression for us to not have this post_as_copy feature anymore.
Do you know if it will be difficult to develop a working fix for this ? Or to reimplement it by any chance ?
We would be happy to help and try the fix you can provide to "beta-test" it.
In the meantime, we will probably remove the write_affinity on our production environnement due to this bug, which will cause some latency for some of our clients.
Thanks in advance for your answer.
----------
PUT objet
./chkjmeter.pl -s dc1 -e preprod -p TESTCONTAINER -c NEW -o testsanswriteaffinity -D 3 -a 1000
Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh X "ls -lah ${DEVICE:-/srv/node*}/d5/containers/3605/311/70aaa9106791d35ef322d6f82df65311"
ssh X "ls -lah ${DEVICE:-/srv/node*}/d5/containers/3605/311/70aaa9106791d35ef322d6f82df65311"
ssh X "ls -lah ${DEVICE:-/srv/node*}/d3/containers/3605/311/70aaa9106791d35ef322d6f82df65311"
ssh X "ls -lah ${DEVICE:-/srv/node*}/d6/containers/3605/311/70aaa9106791d35ef322d6f82df65311" # [Handoff]
ssh X "ls -lah ${DEVICE:-/srv/node*}/d4/containers/3605/311/70aaa9106791d35ef322d6f82df65311" # [Handoff]
ssh X "ls -lah ${DEVICE:-/srv/node*}/d4/containers/3605/311/70aaa9106791d35ef322d6f82df65311" # [Handoff]
note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.
Hello !
My bad, you are correct :)
We did set the write affinity to "none" instead of removing it from our configuration, which was perhaps causing the same issue, we probably did a mistake in the configuration when we were testing your workaround.
Now that we have removed it from our configuration, it's all good.
As you can see from the below logs, we are seeing the correct behaviour now.
But it's quite impacting for us, as we are using the region feature in order to provider a better latency/response time to our internal clients to manage their object on different DCs.
It would be great indeed to have this functionnality back, with the write_affinity, which was working perfectly with Kilo version.
As for now, it's a regression for us to not have this post_as_copy feature anymore.
Do you know if it will be difficult to develop a working fix for this ? Or to reimplement it by any chance ?
We would be happy to help and try the fix you can provide to "beta-test" it.
In the meantime, we will probably remove the write_affinity on our production environnement due to this bug, which will cause some latency for some of our clients.
Thanks in advance for your answer.
----------
PUT objet
./chkjmeter.pl -s dc1 -e preprod -p TESTCONTAINER -c NEW -o testsanswriteaf finity -D 3 -a 1000
------- ------- ------- ------- ------- ------- ------- - iteaffinity. dc1 api.swift. com/v1/ AUTH_TESTCONTAI NER ------- ------- ------- ------- ------- ------- - api.swift. com/v1/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1
ENV :preprod
SITE :dc1
RESOURCE :NEW/testsanswr
URL :http://
-------
OBJECT CREATE OK CHECK OK http://
META ADD OK CHECK OK expired on Tue Aug 11 14:21:14 2020, NOW Tue Aug 11 14:04:35 2020
Proxy DC 1 2020/12/ 04/30 PUT /v1/AUTH_ TESTCONTAINER/ NEW/testsanswri teaffinity. dc1 HTTP/1.0 201 - HTTP-Tiny/0.070 gAAAAABfMokIHh2x... 93 - - tx9f716ff5a39a4 47eb3d03- 005f32894d - 0.3986 - - 1597147469. 603407621 1597147470. 001974583 0 2020/12/ 04/30 GET /v1/AUTH_ TESTCONTAINER/ NEW/testsanswri teaffinity. dc1 HTTP/1.0 200 - HTTP-Tiny/0.070 gAAAAABfMokIHh2x... - 93 - tx4a6f045b844e4 f12bd15d- 005f32894e - 0.0176 - - 1597147470. 022356033 1597147470. 039961100 0 2020/12/ 04/34 POST /v1/AUTH_ TESTCONTAINER/ NEW/testsanswri teaffinity. dc1 HTTP/1.0 202 - HTTP-Tiny/0.070 gAAAAABfMokIHh2x... - 76 - txc4d7538e50394 b9a93a46- 005f328951 - 0.5977 - x-delete- at:1597148474 1597147474. 152446032 1597147474. 750192642 0 2020/12/ 04/35 HEAD /v1/AUTH_ TESTCONTAINER/ NEW/testsanswri teaffinity. dc1 HTTP/1.0 200 - HTTP-Tiny/0.070 gAAAAABfMokIHh2x... - - - tx213479caa1464 bfa8c15d- 005f328952 - 0.0184 - - 1597147475. 246030569 1597147475. 264401913 0
Aug 11 14:04:30 localhost swift-proxy: 127.0.0.1 127.0.0.1 11/Aug/
Aug 11 14:04:30 localhost swift-proxy: 127.0.0.1 127.0.0.1 11/Aug/
Aug 11 14:04:34 localhost swift-proxy: 127.0.0.1 127.0.0.1 11/Aug/
Aug 11 14:04:35 localhost swift-proxy: 127.0.0.1 127.0.0.1 11/Aug/
Storage DC 1
Aug 11 14:04:29 storage01.dc1 swift-object: X - - [11/Aug/ 2020:12: 04:29 +0000] "PUT /d6/3633/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" 201 - "PUT http:// api.swift. com/v1/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" "tx9f716ff5a39a 447eb3d03- 005f32894d" "proxy-server 294" 0.1916 "-" 111 0 2020:12: 04:30 +0000] "GET /d6/3633/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" 200 93 "GET http:// api.swift. com/v1/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" "tx4a6f045b844e 4f12bd15d- 005f32894e" "proxy-server 294" 0.0020 "-" 111 0 2020:12: 04:34 +0000] "POST /d6/3633/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" 202 76 "POST http:// api.swift. com/v1/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" "txc4d7538e5039 4b9a93a46- 005f328951" "proxy-server 292" 0.3339 "-" 113 0 2020:12: 04:35 +0000] "HEAD /d6/3633/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" 200 93 "HEAD http:// api.swift. com/v1/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" "tx213479caa146 4bfa8c15d- 005f328952" "proxy-server 285" 0.0014 "-" 113 0
Aug 11 14:04:30 storage01.dc1 swift-object: X - - [11/Aug/
ug 11 14:04:34 storage01.dc1 swift-object: X - - [11/Aug/
Aug 11 14:04:35 storage01.dc1 swift-object: X - - [11/Aug/
Storage DC 2 2020:12: 04:29 +0000] "PUT /d4/3633/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" 201 - "PUT http:// api.swift. com/v1/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" "tx9f716ff5a39a 447eb3d03- 005f32894d" "proxy-server 294" 0.1458 "-" 110 0 2020:12: 04:34 +0000] "POST /d4/3633/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" 202 76 "POST http:// api.swift. com/v1/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" "txc4d7538e5039 4b9a93a46- 005f328951" "proxy-server 292" 0.5770 "-" 111 0
Aug 11 14:04:29 storage01.dc2 swift-object: X - - [11/Aug/
Aug 11 14:04:34 storage01.dc2 swift-object: X - - [11/Aug/
Storage DC 3 2020:12: 04:29 +0000] "PUT /d5/3633/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" 201 - "PUT http:// api.swift. com/v1/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" "tx9f716ff5a39a 447eb3d03- 005f32894d" "proxy-server 294" 0.2208 "-" 113 0 2020:12: 04:34 +0000] "POST /d5/3633/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" 202 76 "POST http:// api.swift. com/v1/ AUTH_TESTCONTAI NER/NEW/ testsanswriteaf finity. dc1" "txc4d7538e5039 4b9a93a46- 005f328951" "proxy-server 292" 0.3796 "-" 111 0
Aug 11 14:04:29 storage01.dc3 swift-object: X - - [11/Aug/
ug 11 14:04:34 storage01.dc3 swift-object: X - - [11/Aug/
----------
/# swift-get-nodes /etc/swift/ object. ring.gz NEW "testsanswritea ffinity. dc1"
Warning: account/container specified but ring not named "container"
Account NEW finity. dc1
Container testsanswriteaf
Object None
Partition 3605 ef322d6f82df653 11
Hash 70aaa9106791d35
Server:Port Device X:6000 d5
Server:Port Device X:6000 d5
Server:Port Device X:6000 d3
Server:Port Device X:6000 d6 [Handoff]
Server:Port Device X:6000 d4 [Handoff]
Server:Port Device X:6000 d4 [Handoff]
curl -g -I -XHEAD "http:// X:6000/ d5/3605/ NEW/testsanswri teaffinity. dc1" X:6000/ d5/3605/ NEW/testsanswri teaffinity. dc1" X:6000/ d3/3605/ NEW/testsanswri teaffinity. dc1" X:6000/ d6/3605/ NEW/testsanswri teaffinity. dc1" # [Handoff] X:6000/ d4/3605/ NEW/testsanswri teaffinity. dc1" # [Handoff] X:6000/ d4/3605/ NEW/testsanswri teaffinity. dc1" # [Handoff]
curl -g -I -XHEAD "http://
curl -g -I -XHEAD "http://
curl -g -I -XHEAD "http://
curl -g -I -XHEAD "http://
curl -g -I -XHEAD "http://
Use your own device location of servers: -/srv/node* }/d5/containers /3605/311/ 70aaa9106791d35 ef322d6f82df653 11" -/srv/node* }/d5/containers /3605/311/ 70aaa9106791d35 ef322d6f82df653 11" -/srv/node* }/d3/containers /3605/311/ 70aaa9106791d35 ef322d6f82df653 11" -/srv/node* }/d6/containers /3605/311/ 70aaa9106791d35 ef322d6f82df653 11" # [Handoff] -/srv/node* }/d4/containers /3605/311/ 70aaa9106791d35 ef322d6f82df653 11" # [Handoff] -/srv/node* }/d4/containers /3605/311/ 70aaa9106791d35 ef322d6f82df653 11" # [Handoff]
such as "export DEVICE=/srv/node"
ssh X "ls -lah ${DEVICE:
ssh X "ls -lah ${DEVICE:
ssh X "ls -lah ${DEVICE:
ssh X "ls -lah ${DEVICE:
ssh X "ls -lah ${DEVICE:
ssh X "ls -lah ${DEVICE:
note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.