Ring builder files need a consistent format

[Tim Burke]

Currently, ring builder files are pickle'd dicts. Nested in them as various types like ints or floats or strings; even lists or other dicts. Unfortunately, we also pickle arrays [1], which means that if we ever add support for py3, operators won't be able to read any of their py2-generated builder files once they transition to py3 [2].

We definitely *don't* want to just toss the builder and reconstruct it from scratch; this would result in massive data movement in the cluster. Instead, we should create a new format that can be read on both py2 and py3 (falling back to trying the old format if reading it fails), and notify customers that they should perform at least one more rebalance (or *something* that would result in us writing a new ring builder file) under py2 before transitioning to py3.

Note that we've actually done something like this before for the ring file [3], although for slightly different reasons.

[1] https://docs.python.org/2/library/array.html
[2] Try running, for example:

    for py in python python3; do
        echo
        echo "testing $py..."
        for c in u b; do
            $py -c "import array; print(array.array($c'H', [0, 1, 2]))"
        done
    done

[3] https://bugs.launchpad.net/swift/+bug/1031954

[Tim Burke (1-tim-z)]

Correction: operators won't be able to read py3-generated builders under (some versions of) py2.

$ python2.7.10 -c 'import array, pickle, os, sys; pickle.dump(array.array("I", [0, 0, 0]), os.fdopen(1, "wb"), protocol=2)' | python3.6.0 -c 'import pickle, os, sys; print(pickle.load(os.fdopen(0, "rb")))'
array('I', [0, 0, 0])

$ python3.6.0 -c 'import array, pickle, os, sys; pickle.dump(array.array("I", [0, 0, 0]), os.fdopen(1, "wb"), protocol=2)' | python2.7.10 -c 'import pickle, os, sys; print(pickle.load(os.fdopen(0, "rb")))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
TypeError: must be char, not unicode

$ python3.6.0 -c 'import array, pickle, os, sys; pickle.dump(array.array("I", [0, 0, 0]), os.fdopen(1, "wb"), protocol=2)' | python2.7.13 -c 'import pickle, os, sys; print(pickle.load(os.fdopen(0, "rb")))'
array('I', [0L, 0L, 0L])

[Samuel Merritt (torgomatic)]

I agree that this would be nice to fix, but operators do have a workaround: install a newer Python 2.7. ("Newer Python 2.7": bet you didn't expect to see those words in a sentence in 2018.)

If we do go make our own, let's be sure and allow 32 or 64 bits for the device ID. We're using array('H') now, which is 16, but there are clusters where that limit is annoyingly small.

[Dr. Jens Harbott (j-harbott)]

Python 2.7 will be EOL in less than 2 years. I'd expect operators to be given at least one year to migrate, so I don't understand why this isn't given a high priority to resolve.

[Dimitri John Ledkov (xnox)]

xenial is on 2.7.12, but we could bump it to 2.7.13 or cherrypick these patches there, if that helps.

bionic has 2.7.15.