Misleading Error: Invalid character '"'

[Dirk Zimoch]

When using a tab character in a field, EPICS 7 prints a misleading error message, not hinting at the tab but at the closing(?) quote.

Example 1: field(DESC, "a b")
Example 2: info(TAB, "a b")

Error: Invalid character '"'
 at or before """ in path "." file "..." line ...

BTW: Is there a reason why I cannot use a tab in my (info) fields?

[Dirk Zimoch (dirk.zimoch)]

There is a tab, not a space between a and b in the examples above.

[Dirk Zimoch (dirk.zimoch)]

Tabs at the beginning or end of the string give even stranger messages:

info(TAB, "a ")
info(TAB, " b")
Error: Invalid character '"'
 at or before """ in path "." file "test.db" line 3
Error: Newline in string, closing quote missing
Error: syntax error

info(TAB, " ")
Error: Invalid character '"'
 at or before """ in path "." file "test.db" line 3
Error: Invalid character '"'
Error: Invalid character ')'
Error: syntax error

[Dirk Zimoch (dirk.zimoch)]

With other "funny" characters, the result is somewhat different.
For example the "bell" char 0x07.

Error: Invalid character '"'
 at or before """ in path "." file "test.db" line 3
Error: Invalid character 0x07
Error: Invalid character '"'
Error: Invalid character ')'
Error: syntax error

At least 0x07 is mentioned, but the message is still misleading.

[Andrew Johnson (anj)]

Why? Because dbLex.l has bugs in the JSON string lexing patterns. Congratulations, you're the first person to have noticed since JSON support was introduced to the database!

[mdavidsaver (mdavidsaver)]

In dbLex.l, I guess the isprint() could change to something else? isprint() && !isspace(). Or maybe isgraph() if this is portable.

> if (isprint((int) yytext[0])) {
> sprintf(message, "Invalid character '%c'", yytext[0]);
> }
> else {
> sprintf(message, "Invalid character 0x%2.2x", yytext[0]);
> }

Offset tracking would be nice, but may be a more complex change.

[Andrew Johnson (anj)]

I take that back, this isn't a bug so much as a request that we be more liberal than the JSON-5 spec permits. I might be open to that, but first we should make the error easier to understand. Here's what's happening:

The values of info() and field() entries in a database file are supposed to be valid JSON-5 values, and are currently parsed as such.

Looking at the JSON spec https://datatracker.ietf.org/doc/html/rfc7159 and the diagrams on https://www.json.org/json-en.html that spec doesn't actually allow *any* unescaped control characters inside string values. The spec says:

   A string begins and ends with
   quotation marks. All Unicode characters may be placed within the
   quotation marks, except for the characters that must be escaped:
   quotation mark, reverse solidus, and the control characters (U+0000
   through U+001F).

Thus all control characters are supposed to be escaped according to the older JSON rules.

The JSON-5 spec at https://spec.json5.org/#strings has similar language, although its BNF also confusingly allows an unescaped SourceCharacter from the ECMAScript language spec at https://262.ecma-international.org/5.1/#sec-6 specification, which is defined as "any Unicode code unit". I think that can be ignored though.

Both of our JSON parser lexers (dbLex.l and yajl_lex.c) follow those strict specifications when it comes to the set of characters allowed inside strings. I could change them to allow unescaped tab characters inside strings (please comment if you have an opinion about that either way), but I will have to fix both parsers to do that.

I agree that the error messages you got aren't particularly helpful. The character being complained about is the initial double-quote at the start of the string – the lexer couldn't match the whole string because of the illegal character between the quotes, so it back-tracked to the very start of the it and complained about the quote itself. This also explains the "funny" result with the BEL character, which isn't currently legal anywhere in a .db file.

To give a more friendly error message here I can add error-matching patterns that recognize anything that looks like a string to a human but doesn't to the strict lexer, then tell the user what's wrong with their string. The first part should be relatively straightforward to code, although I'm not sure I want to write code that could analyze any kind of broken string and explain why it isn't a legal JSON string.

[Andrew Johnson (anj)]

This change

diff --git a/modules/database/src/ioc/dbStatic/dbLex.l b/modules/database/src/ioc/dbStatic/dbLex.l
index 33185133c..19812c9da 100644
--- a/modules/database/src/ioc/dbStatic/dbLex.l
+++ b/modules/database/src/ioc/dbStatic/dbLex.l
@@ -130,6 +130,14 @@ static int yyreset(void)

 <INITIAL,JSON>{whitespace} ;

+<JSON>{doublequote}({stringchar}|{escape})*{doublequote} { /* bad JSON */
+ yyerrorAbort("Bad character in JSON string");
+}
+
+<JSON>{singlequote}({stringchar}|{escape})*{singlequote} { /* bad JSON */
+ yyerrorAbort("Bad character in JSON string");
+}
+
 <INITIAL,JSON>. {
     char message[40];
     YY_BUFFER_STATE *dummy=0;

results in this error message:

woz$ softIoc -d ~/db/test.db
Error: Bad character in JSON string
 at or before ""a b"" in file "/Users/anj/db/test.db" line 5
Error: Invalid character ')'
Error: syntax error
dbLoadRecords: failed to load '/Users/anj/db/test.db'
Error: Failed to load: /Users/anj/db/test.db

which is better, although not perfect because of the messages afterwards. Instead of just calling yyerrorAbort() it could take a look at the string (found in yytext) and explain what's wrong with it. Contributions welcomed...

[Dirk Zimoch (dirk.zimoch)]

JSON also requires UTF-8 (which is not really supported anywhere else in EPICS, AFAIK). Thus using non-ASCII Latin-1 chars would be illegal. But it does not complain about Latin-1 encoded äöü which may have their place in DESC for example, or μ and ° in EGU.

[Dirk Zimoch (dirk.zimoch)]

To clarify: I’m not complaining that the parser accepts Latin-1. Please keep it that way.
Rather our understanding of a string should not be limited by JSON.

[Dirk Zimoch (dirk.zimoch)]

I like https://spec.json5.org/#parsers „A JSON5 parser may accept non-JSON5 forms or extensions“

[Andrew Johnson (anj)]

The dbLex.l parser doesn't try to understand UTF-8 and will accept any characters from \x80-\xff in the string for the second parameter of a field() or info() entry. If you want us to keep that behavior you should generate a PR adding tests for that capability, with comments in the test giving the reasoning for them.

What happens to a string after it has been saved by dbPutString() or dbPutInfo() may be more challenging though. If a field is a JSON link it's parsed during iocInit() by the yajl parser, and that probably does require the string to be valid UTF-8. Similarly the QSRV code which parses info tags named "Q:group" uses the yajl parser. I think your update to the dbpf command calls the yajl parser so value strings there may have to be UTF-8, etc.