libpst / readpst incorrectly decodes latin1 contacts, etc.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
LIBPST |
New
|
Undecided
|
Unassigned | ||
libpst (Ubuntu) |
New
|
Undecided
|
Unassigned |
Bug Description
After a client of ours moved from Exchange 2003 to Office 365 we had to get some data out of PST-files, which mostly worked well, but apparently Contacts and some Tasks have a tendency på be incorrectly decoded into gibberish.
As far as I can tell, the problem is that the data is interpreted to be UTF16 that needs to be converted to UTF8 and the charset I defined on the commandline for readpst is not consulted in this transaction.
When inspecting the debug log, it is clear to human eyes that this conversion is incorrect and if anything should have been from the charset I specified to UTF8 and not from UTF16 to UTF8.
As far as I can tell, the problem occurs in the 'pst_vb_utf16to8', which seems to be called indescrimately, and it seems that the charset I specify to readpst is rarely used, if ever.
I wonder if it would be possible to have a switch to present the user with the unconverted version and possibly a couple of encoding and let the user decide the proper one. There are several contacts that are fine, but over 200 that suffer from this garbling of the data. Unfortunately it is more or less impossible to get from the utf8 version of the non-utf16 data back to latin1, as far as I can tell.
This is a sample contact that has the issue (Most are totally illegible, but a few had some text I could search for):
FN:Ballerup Politi
N:汋獯整�;潊湨祮;;;
EMAIL:慂汬牥灵倠汯瑩<
ADR;TYPE=
LABEL;TYPE=
TEL;TYPE=
TEL;TYPE=
TITLE:楖散潰楬楴潫浭獩狦
NOTE:Gladsaxe Politi (kredsen) 3969 1448\n
VERSION: 3.0
END:VCARD
Attached is debug version of the parsing of this contact.
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: pst-utils 0.6.59-1build1
ProcVersionSign
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.11
Architecture: amd64
CurrentDesktop: X-Cinnamon
Date: Tue Jun 30 11:05:51 2015
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-07-27 (337 days ago)
InstallationMedia: Linux Mint 17 "Qiana" - Release amd64 20140624
ProcEnviron:
SHELL=/bin/bash
TERM=xterm
PATH=(custom, no user)
LANG=da_DK.UTF-8
XDG_RUNTIME_
SourcePackage: libpst
UpgradeStatus: No upgrade log present (probably fresh install)
PS:
This is the full commandline I used to get the above result (and all the other contacts now shown here):
readpst -C iso-8859-1 -cv -tc -o /tmp/TAN/ -d /tmp/TAN/ TAN-debug. txt Postkasse\ -\ Torben\ Andersen.pst
Also, Office 2013 act precisely the same way as libpst seems to do, but wouldn't it be nice to have a leg up on Microsoft here?
Clearly, the data is in there and if we can only avoid it getting garbled then it is easy enough to convert the Latin1 entries later to UTF-8, if need be.