| 2014-10-22 20:49:13 |
Aditya Sawhney |
bug |
|
|
added bug |
| 2014-10-22 22:01:27 |
Aditya Sawhney |
description |
Doing a container HEAD with X-Newest header immediately after POST on container sometimes returns stale metadata even though all the container servers are working as normal.
The reason for this behavior is that the operation to update the put-timestamp and metadata in container-stat database isn't atomic.
swift\container\server.py Line 275
=====================================
{code}
def PUT(self, req):
...
else: # put container
(1) created = self._update_or_create(req, broker, timestamp)
metadata = {}
(2) metadata.update(
(key, (value, timestamp))
for key, value in req.headers.iteritems()
if key.lower() in self.save_headers or
is_sys_or_user_meta('container', key))
{code}
So, in above code a race condition exists where one thread is trying to update the container metadata while another thread is trying to read the latest metadata. Now, if the read request executes after (1) but before (2) is executed then it gets back stale data with latest timestamp.
Now, the proxy looks at the put-timestamps which will be the same for all replicas and fallbacks to created-at timestamp (x-timestamp). If the replica with stale data was created later than other replicas (when we did container put) then it will have a x-timestamp greater than the others which will make proxy pick the replica with stale data.
swift\proxy\controllers\base.py Line 857
==========================================
{code}
if sources:
sources.sort(key=lambda s: source_key(s[0]))
source, node = sources.pop()
def source_key(resp):
return float(resp.getheader('x-put-timestamp') or
resp.getheader('x-timestamp') or 0)
{code}
The code in container\server.py should either atomically update the metadata and put-timestamp or update the put-timestamp AFTER it has updated the metadata.
This is serious defect which can lead to consistencies in our system which isn't acceptable. |
Doing a container HEAD with X-Newest header immediately after POST on container sometimes returns stale metadata even though all the container servers are working as normal.
The reason for this behavior is that the operation to update the put-timestamp and metadata in container-stat database isn't atomic.
swift\container\server.py Line 275
=====================================
{code}
def PUT(self, req):
...
else: # put container
(1) created = self._update_or_create(req, broker, timestamp)
metadata = {}
(2) metadata.update(
(key, (value, timestamp))
for key, value in req.headers.iteritems()
if key.lower() in self.save_headers or
is_sys_or_user_meta('container', key))
{code}
So, in above code a race condition exists where one thread is trying to update the container metadata while another thread is trying to read the latest metadata. Now, if the read request executes after (1) but before (2) is executed then it gets back stale data with latest timestamp.
Now, the proxy looks at the put-timestamps which will be the same for all replicas and fallbacks to created-at timestamp (x-timestamp). If the replica with stale data was created later than other replicas (when we did container put) then it will have a x-timestamp greater than the others which will make proxy pick the replica with stale data.
swift\proxy\controllers\base.py Line 857
==========================================
{code}
if sources:
sources.sort(key=lambda s: source_key(s[0]))
source, node = sources.pop()
def source_key(resp):
return float(resp.getheader('x-put-timestamp') or
resp.getheader('x-timestamp') or 0)
{code}
The code in container\server.py should either atomically update the metadata and put-timestamp or update the put-timestamp AFTER it has updated the metadata.
This is serious defect which can lead to inconsistencies in our system which isn't acceptable. |
|
| 2014-10-23 17:33:43 |
Aditya Sawhney |
description |
Doing a container HEAD with X-Newest header immediately after POST on container sometimes returns stale metadata even though all the container servers are working as normal.
The reason for this behavior is that the operation to update the put-timestamp and metadata in container-stat database isn't atomic.
swift\container\server.py Line 275
=====================================
{code}
def PUT(self, req):
...
else: # put container
(1) created = self._update_or_create(req, broker, timestamp)
metadata = {}
(2) metadata.update(
(key, (value, timestamp))
for key, value in req.headers.iteritems()
if key.lower() in self.save_headers or
is_sys_or_user_meta('container', key))
{code}
So, in above code a race condition exists where one thread is trying to update the container metadata while another thread is trying to read the latest metadata. Now, if the read request executes after (1) but before (2) is executed then it gets back stale data with latest timestamp.
Now, the proxy looks at the put-timestamps which will be the same for all replicas and fallbacks to created-at timestamp (x-timestamp). If the replica with stale data was created later than other replicas (when we did container put) then it will have a x-timestamp greater than the others which will make proxy pick the replica with stale data.
swift\proxy\controllers\base.py Line 857
==========================================
{code}
if sources:
sources.sort(key=lambda s: source_key(s[0]))
source, node = sources.pop()
def source_key(resp):
return float(resp.getheader('x-put-timestamp') or
resp.getheader('x-timestamp') or 0)
{code}
The code in container\server.py should either atomically update the metadata and put-timestamp or update the put-timestamp AFTER it has updated the metadata.
This is serious defect which can lead to inconsistencies in our system which isn't acceptable. |
Doing a container HEAD with X-Newest header immediately after POST on container sometimes returns stale metadata even though all the container servers are working as normal.
The reason for this behavior is that the operation to update the put-timestamp and metadata in container-stat database isn't atomic.
swift\container\server.py Line 275
=====================================
{code}
def PUT(self, req):
...
else: # put container
(1) created = self._update_or_create(req, broker, timestamp)
metadata = {}
(2) metadata.update(
(key, (value, timestamp))
for key, value in req.headers.iteritems()
if key.lower() in self.save_headers or
is_sys_or_user_meta('container', key))
{code}
So, in above code a race condition exists where one thread is trying to update the container metadata while another thread is trying to read the latest metadata. If the read request executes after (1) but before (2) is executed then it gets back stale data with latest timestamp.
Now, the proxy looks at the put-timestamps which will be the same for all replicas and fallbacks to created-at timestamp (x-timestamp). If the replica with stale data was created later than other replicas (when we did container put) then it will have a x-timestamp greater than the others which will make proxy pick the replica with stale data.
swift\proxy\controllers\base.py Line 857
==========================================
{code}
if sources:
sources.sort(key=lambda s: source_key(s[0]))
source, node = sources.pop()
def source_key(resp):
return float(resp.getheader('x-put-timestamp') or
resp.getheader('x-timestamp') or 0)
{code}
The code in container\server.py should either atomically update the metadata and put-timestamp or update the put-timestamp AFTER it has updated the metadata.
This is serious defect which can lead to inconsistencies in our system which isn't acceptable. |
|
| 2017-07-26 23:11:54 |
clayg |
summary |
x-newest doesnt return newest data under certain race conditions |
x-newest doesnt return newest data for containers |
|
| 2017-07-26 23:14:13 |
clayg |
summary |
x-newest doesnt return newest data for containers |
x-newest doesnt return newest metadata for containers |
|
| 2017-07-26 23:18:59 |
clayg |
summary |
x-newest doesnt return newest metadata for containers |
x-newest doesnt return newest metadata for containers updated with POST |
|
| 2017-07-26 23:23:52 |
clayg |
summary |
x-newest doesnt return newest metadata for containers updated with POST |
x-newest doesnt return newest metadata for containers |
|
| 2017-07-26 23:40:44 |
clayg |
swift: importance |
Undecided |
Low |
|
| 2017-07-26 23:40:47 |
clayg |
swift: status |
New |
Confirmed |
|
| 2017-07-26 23:40:55 |
clayg |
tags |
|
container-server |
|
| 2017-07-26 23:41:02 |
clayg |
tags |
container-server |
container-server low-hanging-fruit |
|
