Encode.pm 2.54 breaks database functions (naco_normalize, maintain_control_numbers, others)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Evergreen |
Fix Released
|
High
|
Unassigned | ||
2.5 |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
* Evergreen master
* PostgreSQL 9.2.4
* Fedora 19
Fedora recently upgraded its version of the Perl Encode.pm module to 2.54. Presumably other distributions will begin to run into this problem as well.
After that upgrade, I noticed that the --load-all-sample flag for eg_db_config started to fail with the error:
ERROR: Cannot decode string with wide characters at /usr/lib64/
... when trying to load the authority sample records.
It turns out that the behaviour of Encode.pm changed with respect to the UTF8 flag for strings when decode_utf8() is called - probably https:/
Changing the decode_utf8() calls to NFD() -- per Tom Christiansen's cookbook recipe that you should normally normalize unicode input to decomposed format on the way in (http://
However, after that, "authority_
Record # 133 : Exception: OpenSRF::EX::ERROR 2013-10-21T22:35:48 main /openils/
Exception: OpenSRF::EX::ERROR 2013-10-21T22:35:48 OpenSRF:
Cannot decode string with wide characters at /usr/lib64/
Wide character in warn at /usr/local/
no mapping found at position 4 in [シュプリンガ
MARC::Charset is currently packaged at version 1.33 on Fedora, so it is possible a newer version mitigates the problem.
My work in progress branch, for what it's worth, is at http://
Changed in evergreen: | |
milestone: | none → 2.5.0 |
importance: | Undecided → High |
status: | New → Confirmed |
Changed in evergreen: | |
milestone: | 2.5.0 → none |
Changed in evergreen: | |
assignee: | nobody → Dan Wells (dbw2) |
Changed in evergreen: | |
status: | Fix Committed → Fix Released |
no longer affects: | evergreen/2.4 |
Oh hey, noticed that "prove -l lib t" in the perlmods directory turns up:
t/14-OpenILS- Utils.t ....... ....... ....... ..... 14/30 Utils.t line 89. � � � � � ' ö& #xE7;&# xC7;È À' Utils.t ....... ....... ....... ..... Dubious, test returned 1 (wstat 256, 0x100)
# Failed test 'clean_marc: diacritics'
# at t/14-OpenILS-
# got: '�
# expected: 'è
# Looks like you failed 1 test of 30.
t/14-OpenILS-
Failed 1/30 subtests
So it looks like the functions outside of pl/Perl get hit by this too. Great!