ToDigest.py i18n subject

[tkikuchi]

ToDigestl.py (v2.23) was considerably improved but
there remain some oddities in digested message subject
representation. Specificaly, MIME subject is not
wrapped with the lheader() propperly.
Please examine the files I am going to attach.

1. Test program to examine the behaviour of lheader()
and improbed (but lengthy) header_in_a_line() which
removes excessive CRLF and adjust folding white spaces.
 It simulates the subject-prefix which will be added in
CookHeaders.py.
2. Result of test program. You will notice
incompatiblity when the subject is once MIME encoded.
3. New patch which use header_in_a_line() and Utils.wrap()

[tkikuchi (tkikuchi-users)]

The file headtest.py.txt was added: headtest.py ... Code test program

[tkikuchi (tkikuchi-users)]

The file testresult.txt was added: testresult.txt ... Code test result

[tkikuchi (tkikuchi-users)]

The file ToDigest.py.diff.txt was added: ToDigest.py

[tkikuchi (tkikuchi-users)]

Logged In: YES
user_id=67709

Forget to note: This patch is revise of #668819 which was
closed and differently applied in recent CVS.

[bwarsaw (bwarsaw)]

Logged In: YES
user_id=12800

Tell me what you think of the hial() function in the
attached file.

[bwarsaw (bwarsaw)]

The file hial.py was added: None

[tkikuchi (tkikuchi-users)]

Logged In: YES
user_id=67709

Looks like good for english text (may be for western) but
folding white space should be treated as null string ('') in
iso-2022-jp encoded japanese (and other RFC 2047 encoded
MIME subject, I believe). u' '.join() must be u''.join() in
these languages. You must alway check if the part of the
header is mime encoded or not when joining. :-(

[tkikuchi (tkikuchi-users)]

Logged In: YES
user_id=67709

OK, Barry. I will compromise. Use u''.join() not u'
'.join(). This eliminate extra space added when joining.
Remember that all-ASCII header will get double space after
the prefix for English text while the spaces after the
prefix is removed for Japanese text.
I think __unicode__() joining in the email package should
take care the difference in RFC2822/2047 headers.

[tkikuchi (tkikuchi-users)]

Logged In: YES
user_id=67709

This is my final patch.
And, I like no blank line in TOC.

[tkikuchi (tkikuchi-users)]

The file ToDigest.py.diff.txt was added: Todigest.py.diff

[tkikuchi (tkikuchi-users)]

Logged In: YES
user_id=67709

updating patch for fixing new unicode-related error reported
by Dan Mick.

[tkikuchi (tkikuchi-users)]

Logged In: YES
user_id=67709

Sorry, I have uploaded intermediate version. This is final,
I hope.

[tkikuchi (tkikuchi-users)]

The file ToDigest.py.diff2.txt was added: ToDigest.py.diff 2nd

[bwarsaw (bwarsaw)]

Logged In: YES
user_id=12800

I'm accepting and applying this patch -- with two
differences. One the line that says "return
oneline.encode()" I want to pass the argument "replace" so
that there's more chance that much of the header decoding
can actually happen. Without this, iso-8859-1 encoded
Subject: headers posted to an English list leave the RFC
2047 encodings in the header, which looks ugly.

The second change is to replace the bare except (bad! :) in
oneline() with a qualified except. I believe only
UnicodeError and LookupError can occur here, although with
the first change above, maybe not even UnicodeError.

Now I want to see if this fixes the problem I've been having
with the spambayes list. Thanks! :)

[bwarsaw (bwarsaw)]

Logged In: YES
user_id=12800

Looks like this is working for python.org at least, so I'm
closing the issue.