open filestream prevents file deletion on windows

[Dennis Knochenwefel]

the bug fix for bug #996084 seems to be a bit more tricky on windows. If you execute the attached example on a windows machine you will get:

user-defined error [file:FOFL9999]: An unknown error occured: "C:\data.txt": I/O error: 32 failed (error The process cannot access the file because it is being used by another process): Can not delete file: C:\data.txt; raised at zorba\modules\org\expath\ns\file.xq.src\file_function.cpp:82

The problem is that the file cannot be deleted as long as the file stream is open. This example query works on linux.

This makes the 2 tests test/rbkt/zorba/file/copy_* fail on windows. Any idea how to solve this?

[Chris Hillery (ceejatec)]

I'm tempted to call this "user error"... I agree it looks like it should work, but that assumption relies on file:read-text() materializing the file contents in memory. We don't want that, and I think we need to accept that Windows will make streaming difficult in this sense.

However, we SHOULD probably offer some function somewhere that would make it possible for a query author to do the right thing - some kind of "close" mechanism. It's possible that scoping $data in this query would be sufficient to solve this problem; Dennis, could you try that out?

[Dennis Knochenwefel (dennis-knochenwefel)]

Even this doesn't work:

  import module namespace file = "http://expath.org/ns/file";
  declare variable $file := "C:\data.txt";
  let $data := file:read-text($file)
  return $data;
  file:delete($file);

I think that even if scoping worked it wouldn't be sufficient. A user should be able to read data into a global variable and delete a file, no? In this case the user wants materialization in memory not only on windows but also on any other OS, right?

This example throws an error even on linux:

  import module namespace file = "http://expath.org/ns/file";
  declare variable $file := "/home/dknochen/test.txt";
  declare variable $data := ();
  $data := file:read-text($file);
  file:delete($file);
  $data

As the streaming part is hidden from the user, the materialization part could be hidden as well, don't you think? For this we need to identify the operations that force materialization. Would that make sense? Is it feasible?

[Chris Hillery (ceejatec)]

The second example works fine on Linux, in fact. At least it does for me. Linux is happy to delete a file which somebody has opened, and it will arrange for the file to go away once everybody is done with it.

I (think I) agree in principle that the right answer is for Zorba to materialize streams when the underlying data source goes away, as in file:delete(). However, I have to admit I can't imagine how to implement that. The Zorba compiler almost certainly can't do it by itself in this case, since the thing being deleted ($file) isn't even the same variable that needs to be materialized ($data). Fixing this might be a major undertaking.

I'm assigning to Markos to see if he has any thoughts about whether this is feasible and/or should be attempted.

[Dennis Knochenwefel (dennis-knochenwefel)]

true, the second example does work on linux (my fault).