fn:tokenize() doesn't stream

Bug #898074 reported by William Candillon on 2011-11-30
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Matthias Brantner

Bug Description

The following query:
let $content := file:read-text("huge_file.txt")
return tokenize($content, "\s")

doesn't stream the result.

Related branches

Changed in zorba:
importance: Undecided → Critical

Making tokenize stream would be really difficult. The reason being that the pattern can be an arbitrary regular expression which is a lot of cases would require to materialize the entire stream anyway.

Hence, I'm suggestion that we define our own string:tokenize function which accepts only separators instead of regular expressions and make this work in a streaming fashion. Would that also do the job?

Changed in zorba:
milestone: none → 2.2
William Candillon (wcandillon) wrote :

Sound's good for me.

Changed in zorba:
assignee: nobody → Matthias Brantner (matthias-brantner)
Changed in zorba:
status: New → Fix Committed
Changed in zorba:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers