IMAP folder names need UTF-7

[Uday Reddy]

Konrad Hinsen reports (viewmail-info, 2011-12-29)

I noticed that IMAP folder names containing non-ASCII characters are
represented incorrectly in VM. They work perfectly well, but the name
that is displayed (and required for input) is messed up. For the
German and French names I have in my account that's not so bad, but I
wouldn't want to deal with Greek or Chinese names.

I explored the issue a bit and found that the folder names are not
decoded from UTF-7 (which is what IMAP specifieds). The raw UTF-7
strings are used as Emacs strings. Inserting a simple utf7-decode
produces a perfectly fine folder list, but then all actions on those
folders fail because the folder names are not properly re-encoded in
UTF-7.

[Uday Reddy (reddyuday)]

I think the easiest way of encoding folder names would be to decide that

- the "maildrop spec" would use UTF-7 names and

- the human-readable account:mailbox specifications would have decoded
characters.

In that case, `vm-imap-parse-spec-to-list' should decode the folder names
from UTF-7 into Emacs internal character set.

And, we would need to define a converse `vm-imap-encode-list-to-spec'
function that does the encoding. I will add this function in the trunk, and
you can then modify it.

When you get it to work, please send me a patch and I can add it in.

If you are going to be playing with code, it would be best to register as a
user on launchpad.net so that you can file and participate in bug reports
etc.

Cheers,
Uday

[Uday Reddy (reddyuday)]

Hi Konrad,

Attached is a patch for vm-imap.el, which creates new functions that would
need UTF-7 encoding/decoding. (These are also in revision 1318 of the
trunk.)

Please get back to me if you have any questions.

Cheers,
Uday

[Uday Reddy (reddyuday)]

Patch from Konrad

[Uday Reddy (reddyuday)]

Comments from Konrad:

I explored this a bit, and came up with something that works though it
isn't perfect. With the attached patch (relative to revno 1323), the
mailbox names are correctly shown in the folder list, and can be
entered correctly when visiting folders or saving to folders, in both
cases with working completion.

What's still wrong is the folder name shown as part of the buffer name
or as part of a prompt in the minibuffer. To fix that I'd have to put
a utf7-decode into vm-imap-parse-spec-to-list (it's there in a
comment), but then visiting IMAP folders breaks because
vm-imap-parse-spec-to-list also gets called from
vm-establish-new-folder-imap-session. Another solution would be to
insert utf7 decoding in a few places in vm.el, but I am not sure
that's a good solution either.

I am tempted to try an arrangement in which mailbox names and specs
are stored as decoded strings everywhere and UTF7 encoding/decoding
happens as close as possible to the IMAP protocol level. But I don't
know the code well enough to say if that is workable approach.

Konrad.

[Uday Reddy (reddyuday)]

Good suggestions, Konrad. I have implemented them. Please find them in the attached patch.

Cheers,
Uday

[Uday Reddy (reddyuday)]

File name completions and default folders still show UTF7 encoded characters.

[Uday Reddy (reddyuday)]

Retracting the last comment (#6). It is working fine.

[khinsen (konrad-hinsen)]

I have tested your last patch, it works partially.

When I visit a folder by caling (vm "IMAP.folder ...) or any of the vm-visit-imap-... functions, supplying a standard Emacs string for the name, everything works fine. However, when I type "v" from a summary view and enter the folder name interactively, the IMAP access fails because the folder name gets passed through utf7-encode twice.

I'll try to trace where exactly this happens.

[Uday Reddy (reddyuday)]

khinsen writes:

> I have tested your last patch, it works partially.
>
> When I visit a folder by caling (vm "IMAP.folder ...) or any of the vm-
> visit-imap-... functions, supplying a standard Emacs string for the
> name, everything works fine. However, when I type "v" from a summary
> view and enter the folder name interactively, the IMAP access fails
> because the folder name gets passed through utf7-encode twice.

There is no double encoding. However, it is possible that minibuffer
reading doesn't work right when there are foreign characters. It would be
good to narrow it down.

Cheers,
Uday

[khinsen (konrad-hinsen)]

There was a double encoding: vm-read-imap-folder-name calls vm-imap-encode-list-to-spec, which calls utf7-encode and thus returns an already encoded string to a function that normally takes and unencoded string.

The attached diff (relative to revision 1333) produces a version that works fine for all the folder operations I tend to use.

[Uday Reddy (reddyuday)]

khinsen writes:

> There was a double encoding: vm-read-imap-folder-name calls vm-imap-
> encode-list-to-spec, which calls utf7-encode and thus returns an already
> encoded string to a function that normally takes and unencoded string.

Then I guess my patch didn't apply fully. The patch you sent me has more
utf7-encode's than mine, not less.

I have now pushed the changes into revision 1335 on the trunk. Let us see
if you are happy with that.

Cheers,
Udaay

[khinsen (konrad-hinsen)]

Uday Reddy writes:

 > I have now pushed the changes into revision 1335 on the trunk. Let us see
 > if you are happy with that.

It works fine!

Konrad.