Whitespace is not preserved during collation

Bug #567212 reported by Gregor Middell
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
collatex
New
Undecided
Unassigned

Bug Description

The current default tokenizer is greedily consuming whitespace. Instead of consuming it at the tokenizer level, whitespace should be preserved in the token and only stripped during token normalization.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.