thunderbird does not warn anymore about unencodable glyphs in recipient name (regression, UTF-8)

Bug #145429 reported by Rolf Leggewie on 2007-09-27
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Enigmail
Won't Fix
Medium
Mozilla Thunderbird
Invalid
High
enigmail (Ubuntu)
Medium
Unassigned

Bug Description

Binary package hint: mozilla-thunderbird

Version 1.5 of Thunderbird in feisty used to warn me when I tried to send a mail to my friend 武, for example, that the character cannot be encoded in my standard ISO-8859-1 encoding and whether I want to send the mail in UTF-8. The latest version in gutsy (version 2.0.0.6 (20070924)) fails to do that and just sends the message which results in the name being replaced by question mark(s).

This is still present in hardy.

Attach mail to this bag thru "Add an attachment" link, please.
1) Save mail to ".eml" file. 2) Attach the file (mime-type=text/plain)

> 2. Use Firefox and go to http://foxkeh.jp/
Source of this page.
  <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="ja" lang="ja">
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<title> text is written in Japanese characters.

Created an attachment (id=262876)
This is the received mail, ???? as subject, since it is not UTF encoded.

(In reply to comment #0)
> Subject: =?ISO-8859-1?Q?=3F=3F=3F=3F=3F*=3F=3F=3F?=

> 4. Now a mail compose window opens in Thunderbird, the subject is in Japanese,
> the Body is only the URL
What will happen when next steps are added after step 4?
4.1) Update subject field manually. (Insert a space, then remove the space)
4.2) Save as Draft

Created an attachment (id=263011)
This happens when I add and remove a space character and save in Drafts, same result: ????

Created an attachment (id=263012)
This happens, when I add some Latin text to the Japanese subject and save as Draft, again: ????

(In reply to comment #4)
> This happens when I add and remove a space character and save in Drafts
(In reply to comment #5)
> This happens, when I add some Latin text to the Japanese subject

Thanks for quick test.
Can I ask you one more test to clarify?
 1. Open the web site by Firefox. (http://foxkeh.jp/)
 2. View page source, and select text between <title> and </title>(title text),
    and Command+C(copy to clip board).
 3. Open mail compose window of Thunderbird, and Command+V to subject(paste),
    and fill To:, and fill mail body text(ascii char only, such as "aaa"),
    then save as Draft.
Problem is recreated?

Created an attachment (id=263039)
I copied Japanese title from Firefox source in the TB subject line, same result: ????

(In reply to comment #7)
> I copied Japanese title from Firefox source in the TB subject line, same
> result: ????

Thanks for quick test. Your test results say that Firefox has no relation.
When Tb 2.0.0.0(en-US version, Japanese MS Win-XP SP2) and new profile and default encoding of compose=iso-8859-1, Tb issued warning as expected in this test case.
Mac OS X only problem? Or PPC Mac OS X only problem?
Or problem when locale=unicode? (If so, may occur on Win/Linux of locale=unicode too.)

(In reply to comment #7)
> Created an attachment (id=263039) [details]
> I copied Japanese title from Firefox source in the TB subject line
Oh, forgot important question.
When the Japanese text is copied to subject, displayed characters in subject is same as one in source view of Firefox?
(Binary representation of unicode for the text may varies while operations, for example, composed <-> decomposed, and it can occur on last 3 characters.)
If you are uncertain on "same characters", please attach screen shot.

Created an attachment (id=263152)
Copy and paste of Japanese characters between Firefox and Thunderbird works without problems, display is correct

(In reply to comment #7)
Another question(and test request) to clarify.
> I copied Japanese title from Firefox source in the TB subject line,
Depends on "default character encoding" setting for compose?
 1.other iso-8859-x such as iso-8859-15, 2.windows-1252, 3.koi8-r

Created an attachment (id=263200)
ISO-8859-15 encoding, does not work

Created an attachment (id=263201)
KOI8-R encoding, does not work

Created an attachment (id=263202)
ISO-8859-2 encoding (Central European), does not work

Created an attachment (id=263203)
ISO-2022-JP encoding (Japanese), works correctly

Created an attachment (id=263204)
UTF-8 encoding (Unicode), works correctly

Your requested Windows-1252 encoding doesn't exist on Thunderbird 2.0.0.0 (Mac, PPC). There is only Windows-1251 (Cyrillic) and Windows-1255 (Hebrew) of those Windows encodings.

Thanks for test. Last test request from me to clarify.
Is problem recreated with Tb 1.5 and latest-trunk?
(New prof, Dummy POP3 account, No Global Inbox, "Compose & Save Drafts" only and ignore other bugs, iso-8859-1 only is sufficient)

Can someone(or you) test with different locale of Mac OS X?
(It's difficult for me to test with other locale than current on Japanese Win...)

I see the same problem, using Thunderbird 1.5.0.10 on Fedora Core 5. It doesn't sound like it is fixed, so perhaps I can give another data point, and also give weight to the opinion that it is a serious bug.

On a mailing list I replied to a message that had a subject, containing Japanese, encoded as:

Re: =?UTF-8?B?5pel4oaS6Iux44CA57+76Kiz44O76YCa6Kiz44CAY291cnNlcyA=?=
 =?UTF-8?B?b3IgY2VydGlmaWNhdGlvbnM=?=

I clicked reply, and in the mail when I composed it I could see the Japanese. But when I sent it the mail used:

Re: =?ISO-8859-1?Q?=3F=3F=3F=3F=3F=3F=3F=3Fcourses_or_certif?= =?ISO-8859-1?Q?ications?=

I.e. The Japanese has turned into lots of question marks. I see the same in my saved folder.

I have iso-8859-1 as my default email language. When I hit reply, then change the mail encoding to UTF-8, and save/send it, it is fine, no corruption.

Usually when I have UTF-8 characters in the *body* of the email it will prompt me, saying "would you like to send this email as UTF-8?".

So, I see two solutions:

SOLUTION ONE: Whatever code scans the body to detect characters different from the current encoding, in order to give the user that warning, must also scan all the headers as well.

SOLUTION TWO: If a header was utf-8 encoded, and the mail is replied-to/forwarded then the header should stay in utf-8 encoding, and not be converted to default encoding. (I.e. the header may be a different encoding to the body.)

Only solution one can work, since it's also possible that the user added some UTF-8 characters into Subject field when composing the reply. You can't rely on the original Subject field for encoding.

(In reply to comment #19)
> Re: =?UTF-8?B?5pel4oaS6Iux44CA57+76Kiz44O76YCa6Kiz44CAY291cnNlcyA=?=
> =?UTF-8?B?b3IgY2VydGlmaWNhdGlvbnM=?=
==>
> Re: =?ISO-8859-1?Q?=3F=3F=3F=3F=3F=3F=3F=3Fcourses_or_certif?=
> =?ISO-8859-1?Q?ications?=

Same result as bug opener's case.
Confirming and changing to OS=All.

WADA, did you intend to leave HW=mac and bug unconfirmed?

Sorry, I seems to have failed to click "Confirm bug".

Binary package hint: mozilla-thunderbird

Version 1.5 of Thunderbird used to warn me when I tried to send a mail to my friend 武, for example, that the character cannot be encoded in my standard ISO-8859-1 encoding and whether I want to send the mail in UTF-8. The latest version in gutsy (version 2.0.0.6 (20070924)) fails to do that and just sends the message which results in the name being replaced by question mark(s).

Updating the title to better reflect the problem - when mail is being sent, we check the body of the message to make sure all the characters in the body can be represented using the selected character encoding; the problem is that no such check is done for the subject.

This works fine for me (TB 2.0.0.12 and later).

When I add a single character not available in selected encoding to the Subject line, TB asks "Send in UTF-8"

Could you please recheck and eventually close this bug?

I can confirm it still happens as of thunderbird 2.0.0.14 (ubuntu 7.10)
I just created a new email with subject set to "テスト" (Japanese, meaning "test"), and only ascii in the body. When I send it doesn't prompt about encoding, and it arrives with the subject changed to "???".

Strange. Just tested with TB 2.0.0.14 (Fedora7). Encoding set to Western (8859-1).

Created new email with Subject: "テスト" and only a single word "test" in the body.

Clicking on Send immediately brings up "Send in UTF-8" dialog. The same happens
on Windows version.

Must be something wrong in your installation or preferences.

Default encoding for me is also iso-8859-1.
Any suggestions for what preference might be affecting it? I've had a hunt through and couldn't see anything that might be related. I'm sending text-only email.
I had "For messages that contain 8-bit characters, use 'quoted-printable' MIME encoding" unchecked. I tried checking that and sending again, but it made no difference.

In environmental variables, I have:
  LANG=en_GB.UTF-8

I have the enigmail plugin installed.

Let me know if you can think of any other setting you want me to check.

Have you tried it without enigmail plugin?

Rolf Leggewie (r0lf) on 2008-08-01
description: updated
Rolf Leggewie (r0lf) wrote :

still the same in hardy TB 2.0

*** Bug 369067 has been marked as a duplicate of this bug. ***

I confirm this problem and it applies to all headers, not just the subject. The name of your dear Japanese friend will be all ????, too. Changed the title accordingly. It is also a regression. TB in feisty used to work correctly. It must have been something 1.5 or earlier.

Quoting
https://bugs.launchpad.net/ubuntu/+source/mozilla-thunderbird/+bug/145429

"Version 1.5 of Thunderbird in feisty used to warn me when I tried to send a
mail to my friend 武, for example, that the character cannot be encoded in my
standard ISO-8859-1 encoding and whether I want to send the mail in UTF-8. The
latest version in gutsy (version 2.0.0.6 (20070924)) fails to do that and just
sends the message which results in the name being replaced by question
mark(s)."

Indee(In reply to comment #30)
> Have you tried it without enigmail plugin?

Indeed, enigmail seems to be the culprit. With enigmail turned off, everything is fine. With the plugin turned the detection seems to go either wrong or not be called at all.

Changed in thunderbird:
status: Unknown → Confirmed
Rolf Leggewie (r0lf) wrote :

problem is known and discussed upstream. confirming.

Changed in mozilla-thunderbird:
importance: Undecided → Medium
status: New → Confirmed
Changed in thunderbird:
status: Unknown → Confirmed
In , Mcow (mcow) wrote :

<email address hidden>: do you also have Enigmail installed?

Rolf Leggewie (r0lf) wrote :

turns out that enigmail is the culprit

*** Bug 346657 has been marked as a duplicate of this bug. ***

Rolf Leggewie (r0lf) wrote :

still present as of today with uptodate karmic packages.

If it's really Enigmail, then it's fixed in current trunk for Thunderbird 3.0.

Changed in enigmail:
status: Unknown → Confirmed

That's fixed for Thunderbird 3.0

Changed in enigmail:
status: Confirmed → Won't Fix
Rolf Leggewie (r0lf) wrote :

yes, it's fixed in lucid. thanks.

Changed in enigmail (Ubuntu):
status: Confirmed → Fix Released

yes, it's indeed fixed. thank you, guys.

Closing

not fixed without identifying bug#. sorry. it's either WFM or invalid

Changed in thunderbird:
status: Confirmed → Invalid
Changed in thunderbird:
importance: Unknown → High
Changed in enigmail:
importance: Unknown → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.