we need tests about handling of potential duplicate entries in indexes

[Samuele Pedroni]

right now it seems we dedupe duplicates index keys in indexes at the sql level at least, that's probably the behavior we want but there are no tests specifying and checking for this

[Stuart Langridge (sil)]

We need to decide whether a document can exist twice in the same index with the same index key, or not, and then enforce that behaviour with tests.

Ways a document might show up twice:

create_index("words", "splitwords(colours)")
document: { colours: "red red red green blue" }
Should this document be in the index three times (one for each of "red", "green", "blue") or five?

create_index("name", "names")
document: { names: [ "aaron", "aaron", "andrew" ] }
Should this document be in the index twice (one for each of "aaron" and "andrew" ) or three times?

I personally think that indexes should be deduped: there should not be two identical entries (that is, two identical (indexkey, doc) pairs) in an index.