fn:tokenize() doesn't stream

[William Candillon]

The following query:
let $content := file:read-text("huge_file.txt")
return tokenize($content, "\s")

doesn't stream the result.

[Matthias Brantner (matthias-brantner)]

Making tokenize stream would be really difficult. The reason being that the pattern can be an arbitrary regular expression which is a lot of cases would require to materialize the entire stream anyway.

Hence, I'm suggestion that we define our own string:tokenize function which accepts only separators instead of regular expressions and make this work in a streaming fashion. Would that also do the job?

[William Candillon (wcandillon)]

Sound's good for me.