Comment 1 for bug 898074

Revision history for this message
Matthias Brantner (matthias-brantner) wrote :

Making tokenize stream would be really difficult. The reason being that the pattern can be an arbitrary regular expression which is a lot of cases would require to materialize the entire stream anyway.

Hence, I'm suggestion that we define our own string:tokenize function which accepts only separators instead of regular expressions and make this work in a streaming fashion. Would that also do the job?