fn:match fails if the string is non-utf8

[Daniel Turcanu]

I have a query that reads a lot of files and apply fn:match on them.
Some files have non-utf8 characters, and file:read-text reads with no problem.
fn:matches calls to_string to convert to ICU string, but that fails. So fn:matches returns false, although I think it should raise an error. Actually to_string should raise an error, otherwise the non-utf8 problem gets unnoticed.

[Paul J. Lucas (paul-lucas)]

We (meaning several Zorba team members) had a long discussion about what to do for invalid UTF-8 byte sequences a while ago and the consensus reached was that the validity of UTF-8 byte sequence should be checked only on entry into Zorba and not after it's been read in. So if the byte sequence is to be checked at all, it should be checked in read-text and any error should be raised there.

[Daniel Turcanu (danielturcanu)]

But here is about ICU.
It is doing the utf8 check once again, and returns error. I think to_string should throw utf8 error, not return false.
Maybe ICU returns some other errors. It is not right for fn:match to just return false, as if the pattern wasn't matched.

[Paul J. Lucas (paul-lucas)]

I can't control what ICU does. If I were to change to_string() to throw an exception now, there are probably plenty of places in the code that would not catch it, so the exception would go all the way to the top and crash Zorba.

You didn't specify a release ("Group") for this bug, so it's not clear whether you expect anything to be done by 2.0. Given what I said above, I think it's too dangerous a change so close to a release.

If the bad UTF-8 sequence were caught when it was read-in in the first place and an exception thrown then, this problem would be moot since it would never get to my code in this case.

[Daniel Turcanu (danielturcanu)]

Gabriel, then it's your bug.
file:read-text should validate for utf8 characters.

[Gabriel Petrovay (gabipetrovay)]

There is a problem with this. read-text will generate a streamable string and the job of read-text is over. So the wrong string will probably be consumed somewhere later in the runtime.

Not sure how to solve this so let-s escalate it to zorba-coders.

[Chris Hillery (ceejatec)]

IMHO, the check needs to be added in the StreamableString implementation. The "rule" is that Zorba doesn't check for UTF8 validity internally, only at the entry points. This is a new entry point, so it needs the check.

[Gabriel Petrovay (gabipetrovay)]

I am not sure if I am the right person to do this. Removing myself from this bug.

[Matthias Brantner (matthias-brantner)]

Paul, can you please mark this bug as fix committed as soon as the transcoding_streambuffer branch has been merged? If I understood correctly, these changes should resolve the bug.