Incorrect JSON serialization of supplementary plane code points
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Zorba |
Fix Released
|
Undecided
|
Paul J. Lucas |
Bug Description
this bug is a follow-up of bug #1024448
Currently, the result of the following JSONiq query:
let $message := "👊"
return { "message": $message }
is serialized into incorrect JSON:
{ "message" : "\ufffffff0\
the correct result would be:
{ "message" : "\ud83d\udc4a" }
Explanation:
Characters from the supplementory plane are usually represented in utf-16 surrogate pairs within JSON results. The above result is in particular incorrect because JSON allows only 4 hex digits after '\u'. utf-16 surrogate pairs alway fit into a 4 hex digit or 2 x 4 hex digit window which is most probably the reason why utf-16 is used.
This has been greatly fixed in the JSON parser by Paul (see mp: https:/
@Paul: I'm not sure if you are the right person to assign this bug to?
thanks
Related branches
- Chris Hillery: Approve
- Dennis Knochenwefel: Approve
- Paul J. Lucas: Approve
-
Diff: 198 lines (+89/-24)9 files modifiedChangeLog (+1/-0)
src/api/serialization/serializer.cpp (+42/-22)
src/api/serialization/serializer.h (+1/-1)
src/util/ascii_util.h (+38/-0)
test/rbkt/ExpQueryResults/zorba/jsoniq/coll_dyn_03.xml.res (+1/-1)
test/rbkt/ExpQueryResults/zorba/jsoniq/string_escapes.xml.res (+1/-0)
test/rbkt/ExpQueryResults/zorba/jsoniq/supplemental_plane.xml.res (+1/-0)
test/rbkt/Queries/zorba/jsoniq/string_escapes.xq (+2/-0)
test/rbkt/Queries/zorba/jsoniq/supplemental_plane.xq (+2/-0)
Changed in zorba: | |
status: | Incomplete → Confirmed |
Changed in zorba: | |
status: | Confirmed → In Progress |
summary: |
- incorrect JSON serialization of supplementory plane code points + Incorrect JSON serialization of supplementory plane code points |
summary: |
- Incorrect JSON serialization of supplementory plane code points + Incorrect JSON serialization of supplementary plane code points |
Changed in zorba: | |
status: | In Progress → Fix Committed |
Changed in zorba: | |
status: | Fix Committed → Fix Released |
First, how does one execute a JSoniq query? If I put the above query into a file and do:
bin/zorba -f -i -r --trailing-nl -q /tmp/foo.xq
I get:
</tmp/ foo.xq> :2,8: static error [err:XPST0003]: invalid expression; raised at .../src/ compiler/ translator/ translator. cpp:11081