Comment 2 for bug 921458

Paul, Could you please investigate why William's program doesn't stream after applying the patch below?
I understand why it didn't stream in the trunk (i.e. with fn:tokenize) but using the string:split function, there should be nothing that prevents it from streaming. I guess the problem is somewhere in the implementation of the split function (in src/runtime/strings/strings_impl.cpp).

=== modified file 'modules/org/expath/ns/file.xq'
--- modules/org/expath/ns/file.xq 2011-10-19 05:09:31 +0000
+++ modules/org/expath/ns/file.xq 2012-02-01 20:39:39 +0000
@@ -23,6 +23,8 @@
  :)
 module namespace file = "http://expath.org/ns/file";

+import module namespace string = "http://www.zorba-xquery.com/modules/string";
+
 import schema namespace output = "http://www.w3.org/2010/xslt-xquery-serialization";
 declare namespace ann = "http://www.zorba-xquery.com/annotations";
 declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
@@ -424,8 +426,7 @@
   $encoding as xs:string
 ) as xs:string*
 {
- let $content := file:read-text($file, $encoding)
- return fn:tokenize($content, "\n")
+ string:split(file:read-text($file, $encoding), "f")
 };

 (:~