chinese email display problem

[Robert Brown]

VM 7.19, but the code is the same in 8.0.X.
Received MIME email with part encoded text/plain, charset GB2312.
Running Emacs 23.0.92.4 in a plain text terminal window that understands utf-8.
Variable vm-mime-charset-converter-alist is set so that gb2312 is converted to utf-8.

When I display the email message, VM converts it to utf-8, which my terminal
program is set up to display. I see the correct Chinese characters when I place a
(sit-for 5) at the beginning of vm-mime-charset-decode-region, but the buffer is later
messed up by this function. I think the order of the COND tests is wrong. The function
should do nothing if display of multiple fonts is impossible or the charset (utf-8) is
my VM MIME default face. The following patch fixes the problem for me.

*** vm-mime.el.~1~ Fri Jun 11 21:59:27 2004
--- vm-mime.el Wed Aug 12 14:22:05 2009
***************
*** 233,239 ****

  (defun vm-mime-charset-decode-region (charset start end)
    (or (markerp end) (setq end (vm-marker end)))
! (cond ((or vm-xemacs-mule-p vm-fsfemacs-mule-p)
         (if (or (and vm-xemacs-p (memq (device-type) '(x gtk mswindows)))
                 vm-fsfemacs-p
                 nil)
--- 233,241 ----

  (defun vm-mime-charset-decode-region (charset start end)
    (or (markerp end) (setq end (vm-marker end)))
! (cond ((not (vm-multiple-fonts-possible-p)) nil)
! ((vm-mime-default-face-charset-p charset) nil)
! ((or vm-xemacs-mule-p vm-fsfemacs-mule-p)
         (if (or (and vm-xemacs-p (memq (device-type) '(x gtk mswindows)))
                 vm-fsfemacs-p
                 nil)
***************
*** 252,259 ****
                     (put-text-property start end 'vm-coding (car cell))))
               ;; In XEmacs 20.0 beta93 decode-coding-region moves point.
               (goto-char opoint))))
- ((not (vm-multiple-fonts-possible-p)) nil)
- ((vm-mime-default-face-charset-p charset) nil)
        (t
         (let ((font (cdr (vm-string-assoc
                           charset
--- 254,259 ----

[Ulrich Müller (ulm)]

> The function should do nothing if display of multiple fonts is
> impossible or the charset (utf-8) is my VM MIME default face.

From this I conclude that you have utf-8 in the list of
vm-mime-default-face-charsets?

What happens if you leave this variable at its default?

[Robert Brown (robert-brown)]

I believe my earlier patch was wrong and that I had misconfigured VM. I'm using
Emacs 23 with VM 7.19 in a terminal window that's configured to display utf-8.
Everything is now working properly for me. I am no longer using iconv
to translate
some character sets to utf-8. I set vm-mime-default-face-charsets to
T. I no longer
set vm-mime-charset-converter-alist.

However, I did have to make two small changes to vm-mime.el. First, the code
was using the wrong function to get raw bytes from the buffer. Second, VM sets
text properties that are strings. The default comparison function that Emacs
uses for text properties is EQ, which can return NIL for equivalent
strings. Here
is the patch that I now believe is correct for VM 7.19 on Emacs 23. Some or
all of it may also be needed for VM 8.

*** vm-mime.el.~1~ Fri Jun 11 21:59:27 2004
--- vm-mime.el Tue Sep 29 13:38:42 2009
***************
*** 595,607 ****
                        (delete-char -1)))))
            (setq inputpos start)
            (while (< inputpos end)
! (setq char (char-after inputpos))
              (cond ((= char ?\n)
                     (vm-insert-char char 1 nil work-buffer)
                     (setq cols 0))
                    ((and (= char 32)
                          (not (= (1+ inputpos) end))
! (not (= ?\n (char-after (1+ inputpos)))))
                     (vm-insert-char char 1 nil work-buffer)
                     (vm-increment cols))
                    ((or (< char 33) (> char 126)
--- 595,607 ----
                        (delete-char -1)))))
            (setq inputpos start)
            (while (< inputpos end)
! (setq char (get-byte inputpos))
              (cond ((= char ?\n)
                     (vm-insert-char char 1 nil work-buffer)
                     (setq cols 0))
                    ((and (= char 32)
                          (not (= (1+ inputpos) end))
! (not (= ?\n (get-byte (1+ inputpos)))))
                     (vm-insert-char char 1 nil work-buffer)
                     (vm-increment cols))
                    ((or (< char 33) (> char 126)
***************
*** 777,782 ****
--- 777,793 ----
        (vm-with-string-as-temp-buffer string 'vm-decode-mime-encoded-words)
      string ))

+ (defun vm-next-single-property-change (start property)
+ "Reimplement NEXT-SINGLE-PROPERTY-CHANGE, but compare properties with
+ EQUAL instead of EQ. This function is necessary because VM uses text
+ properties that are strings."
+ (let ((value (get-text-property start property))
+ (next (next-single-property-change start property)))
+ (if (or (null next)
+ (not (equal value (get-text-property next property))))
+ next
+ (vm-next-single-property-change next property))))
+
  (defun vm-reencode-mime-encoded-words ()
    (let ((charset nil)
        start coding pos q-encoding
***************
*** 787,793 ****
        (setq start (point-min))
        (while (not done)
        (setq charset (get-text-property start 'vm-charset))
! (setq pos (next-single-property-change start 'vm-charset))
        (or pos (setq pos (point-max) done t))
        (i...

[Ulrich Müller (ulm)]

Unfortunately, function "get-byte" exists neither in GNU Emacs 22 nor in XEmacs.

[Robert Brown (robert-brown)]

Yes, that sounds right. The code may need to be conditionalized on Emacs
version. If you'd like I can spend a bit of time experimenting with "legacy"
Emacs releases.

bob

==========

On Mon, Dec 7, 2009 at 1:57 PM, Ulrich Müller <email address hidden> wrote:
> Unfortunately, function "get-byte" exists neither in GNU Emacs 22 nor in
> XEmacs.
>
> --
> chinese email display problem
> https://bugs.launchpad.net/bugs/412670
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in VM - Goto http://launchpad.net/vm for active development: Incomplete
> Status in VM (View Mail) for Emacs: Incomplete
>
> Bug description:
> VM 7.19, but the code is the same in 8.0.X.
> Received MIME email with part encoded text/plain, charset GB2312.
> Running Emacs 23.0.92.4 in a plain text terminal window that understands utf-8.
> Variable vm-mime-charset-converter-alist is set so that gb2312 is converted to utf-8.
>
> When I display the email message, VM converts it to utf-8, which my terminal
> program is set up to display.  I see the correct Chinese characters when I place a
> (sit-for 5) at the beginning of vm-mime-charset-decode-region, but the buffer is later
> messed up by this function.  I think the order of the COND tests is wrong.  The function
> should do nothing if display of multiple fonts is impossible or the charset (utf-8) is
> my VM MIME default face.  The following patch fixes the problem for me.
>
>
> *** vm-mime.el.~1~      Fri Jun 11 21:59:27 2004
> --- vm-mime.el  Wed Aug 12 14:22:05 2009
> ***************
> *** 233,239 ****
>
>  (defun vm-mime-charset-decode-region (charset start end)
>    (or (markerp end) (setq end (vm-marker end)))
> !   (cond ((or vm-xemacs-mule-p vm-fsfemacs-mule-p)
>         (if (or (and vm-xemacs-p (memq (device-type) '(x gtk mswindows)))
>                 vm-fsfemacs-p
>                 nil)
> --- 233,241 ----
>
>  (defun vm-mime-charset-decode-region (charset start end)
>    (or (markerp end) (setq end (vm-marker end)))
> !   (cond ((not (vm-multiple-fonts-possible-p)) nil)
> !         ((vm-mime-default-face-charset-p charset) nil)
> !         ((or vm-xemacs-mule-p vm-fsfemacs-mule-p)
>         (if (or (and vm-xemacs-p (memq (device-type) '(x gtk mswindows)))
>                 vm-fsfemacs-p
>                 nil)
> ***************
> *** 252,259 ****
>                     (put-text-property start end 'vm-coding (car cell))))
>               ;; In XEmacs 20.0 beta93 decode-coding-region moves point.
>               (goto-char opoint))))
> -       ((not (vm-multiple-fonts-possible-p)) nil)
> -       ((vm-mime-default-face-charset-p charset) nil)
>        (t
>         (let ((font (cdr (vm-string-assoc
>                           charset
> --- 254,259 ----
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/viewmail/+bug/412670/+subscribe
>

[Uday Reddy (reddyuday)]

I am bumping this because it was set to expire a couple of weeks ago. If there is no interest in investigating this further, please change it to "won't fix" so that it will get off the table.