Ring builder files need a consistent format
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
New
|
Undecided
|
Unassigned |
Bug Description
Currently, ring builder files are pickle'd dicts. Nested in them as various types like ints or floats or strings; even lists or other dicts. Unfortunately, we also pickle arrays [1], which means that if we ever add support for py3, operators won't be able to read any of their py2-generated builder files once they transition to py3 [2].
We definitely *don't* want to just toss the builder and reconstruct it from scratch; this would result in massive data movement in the cluster. Instead, we should create a new format that can be read on both py2 and py3 (falling back to trying the old format if reading it fails), and notify customers that they should perform at least one more rebalance (or *something* that would result in us writing a new ring builder file) under py2 before transitioning to py3.
Note that we've actually done something like this before for the ring file [3], although for slightly different reasons.
[1] https:/
[2] Try running, for example:
for py in python python3; do
echo
echo "testing $py..."
for c in u b; do
$py -c "import array; print(array.
done
done
Correction: operators won't be able to read py3-generated builders under (some versions of) py2.
$ python2.7.10 -c 'import array, pickle, os, sys; pickle. dump(array. array(" I", [0, 0, 0]), os.fdopen(1, "wb"), protocol=2)' | python3.6.0 -c 'import pickle, os, sys; print(pickle. load(os. fdopen( 0, "rb")))'
array('I', [0, 0, 0])
$ python3.6.0 -c 'import array, pickle, os, sys; pickle. dump(array. array(" I", [0, 0, 0]), os.fdopen(1, "wb"), protocol=2)' | python2.7.10 -c 'import pickle, os, sys; print(pickle. load(os. fdopen( 0, "rb")))' Library/ Frameworks/ Python. framework/ Versions/ 2.7/lib/ python2. 7/pickle. py", line 1378, in load file).load( ) Library/ Frameworks/ Python. framework/ Versions/ 2.7/lib/ python2. 7/pickle. py", line 858, in load key](self) Library/ Frameworks/ Python. framework/ Versions/ 2.7/lib/ python2. 7/pickle. py", line 1133, in load_reduce
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/System/
return Unpickler(
File "/System/
dispatch[
File "/System/
value = func(*args)
TypeError: must be char, not unicode
$ python3.6.0 -c 'import array, pickle, os, sys; pickle. dump(array. array(" I", [0, 0, 0]), os.fdopen(1, "wb"), protocol=2)' | python2.7.13 -c 'import pickle, os, sys; print(pickle. load(os. fdopen( 0, "rb")))'
array('I', [0L, 0L, 0L])