Implementation of proposed protocol tweak

[Jason Gerard DeRose]

After a lot of thought over the last six moths or so, there is one final tweak I'd like to make to the hashing protocol, and it has to do with how we cryptographically tie the leaf-hash to the leaf-index, and the root-hash to the file-size. My proposed change has two advantages:

1) Simplicity. It makes the protocol simpler to describe, simpler to implement, and simpler to validate.

2) Higher security confidence by using the standard Skein *key* parameter rather than our own form of cryptographic tying

Our protocol tying problem is similar to the MAC problem, although not as demanding because key-recovery isn't a concern in our case. But any mechanism suitable for MAC should be suitable for us. Currently we do our cryptographic tying like this:

leaf_hash = hash(hash(leaf_index) + leaf_data)
root_hash = hash(hash(file_size) + leaf_hashes)

Although the above is fairly well designed and avoids the major problems of a naive MAC built using a hash function, it's also non-standard and not as carefully designed as HMAC. In retrospect, using standard HMAC with Skein would have been a better choice.

However, an even better choice is to use the Skein *key* parameter and to do the tying like this:

leaf_hash = hash(leaf_data, key=leaf_index)
root_hash = hash(leaf_hashes, key=file_size)

The Skein parameter system was designed to be used in place of HMAC, and provides a very generic cryptographic tying mechanism. This batteries-included approach of Skein is fantastic, and we might as well use it.

In order to better facilitate discussion about this change, I'm providing an implementation of the proposed hash_leaf() and hash_root() functions as:

hash_leaf2(leaf_index, leaf_data)
hash_root2(file_size, leaf_hashes)

Because of the "2" suffix, this causes no API break, and no behavior change for the existing code.

[Jason Gerard DeRose (jderose)]

Note I've now nicely documented the proposed protocol change:

http://docs.novacut.com/filestore/protocol2.html