decoding unicode (still) not supported bug

Bug #328353 reported by elaine ashton
6
Affects Status Importance Assigned to Milestone
GNU Mailman
Fix Released
Undecided
Mark Sapiro

Bug Description

Running 2.1.12rc2 I'm getting a lot of shunted messages when it hits the archiver. I saw an earlier bug report with a patch that appears to have made it into this release but..I'm still getting a lot of email shunted without any apparent reason. The error and message dump are as follows. List language pref is english. Site language pref is english.

**Update*** This problem disappeared when I upgraded from Python 2.4.4 to 2.5.1 - So the solution is to use a more recent version of python if seeing this error.

Feb 11 18:11:46 2009 (3144) SHUNTING: 1234401373.246341+325698c6ec7fb59408285b92d3e7664c51258d95
Feb 11 18:13:16 2009 (3147) send_digests() failed: decoding Unicode is not supported
Feb 11 18:13:16 2009 (3144) Uncaught runner exception: decoding Unicode is not supported
Feb 11 18:13:16 2009 (3144) Traceback (most recent call last):
  File "/opt/opensolaris.org/mailman/Mailman/Queue/Runner.py", line 120, in _oneloop
    self._onefile(msg, msgdata)
  File "/opt/opensolaris.org/mailman/Mailman/Queue/Runner.py", line 191, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/opt/opensolaris.org/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose
    mlist.ArchiveMail(msg)
  File "/opt/opensolaris.org/mailman/Mailman/Archiver/Archiver.py", line 216, in ArchiveMail
    h.processUnixMailbox(f)
  File "/opt/opensolaris.org/mailman/Mailman/Archiver/pipermail.py", line 578, in processUnixMailbox
    a = self._makeArticle(m, self.sequence)
  File "/opt/opensolaris.org/mailman/Mailman/Archiver/HyperArch.py", line 681, in _makeArticle
    mlist=self.maillist)
  File "/opt/opensolaris.org/mailman/Mailman/Archiver/HyperArch.py", line 305, in __init__
    charset = message.get_content_charset(cset_out)
  File "/usr/lib/python2.4/email/Message.py", line 800, in get_content_charset
    charset = unicode(charset, 'us-ascii').encode('us-ascii')
TypeError: decoding Unicode is not supported

And the corresponding email with the email addresses trimmed.

[----- start pickle file -----]
<----- start object 1 ----->
From <email address hidden> Wed Feb 11 17:16:12 2009
Return-Path: <email address hidden>
X-Original-To: <email address hidden>
Delivered-To: <email address hidden>
Received: by mail.opensolaris.org (Postfix, from userid 501)
        id BB9B18D3CA; Wed, 11 Feb 2009 17:16:12 -0800 (PST)
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
        oss-mail1.opensolaris.org
X-Spam-Level:
X-Spam-Status: No, score=-6.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED
        autolearn=ham version=3.2.5
Received: from sca-es-mail-1.sun.com (sca-es-mail-1.Sun.COM [192.18.43.132])
        by mail.opensolaris.org (Postfix) with ESMTP id C95E08D3A9
        for <email address hidden>; Wed, 11 Feb 2009 17:15:12 -0800 (PST)
Received: from fe-sfbay-09.sun.com ([192.18.43.129])
        by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id
        n1C1FCTY026531
        for <email address hidden>; Wed, 11 Feb 2009 17:15:12 -0800 (PST)
MIME-version: 1.0
Content-type: multipart/mixed; boundary="Boundary_(ID_r1P2Tq7K/ReXv3hja2KS8g)"
Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com
        (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 23
        2008))
        id <email address hidden> for <email address hidden>;
        Wed, 11 Feb 2009 17:15:12 -0800 (PST)
Received: from [192.168.1.65] ([unknown] [76.254.2.176])
        by fe-sfbay-09.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01
        64bit (built Dec 23 2008)) with ESMTPSA id
        <email address hidden>;
        Wed, 11 Feb 2009 17:15:11 -0800 (PST)
Date: Wed, 11 Feb 2009 17:11:52 -0800
From: Rafael Vanoni <email address hidden>
In-reply-to: <email address hidden>
Sender: <email address hidden>
To: "Li, Aubrey" <email address hidden>
Message-id: <email address hidden>
References: <email address hidden>
User-Agent: Thunderbird 2.0.0.18 (X11/20081216)
Cc: "<email address hidden>" <email address hidden>
Subject: Re: [tesla-dev] powertop> c/p-state messages on sparc
X-BeenThere: <email address hidden>
X-Mailman-Version: 2.1.12rc2
Precedence: list
List-Id: "Project Tesla: Solaris Enhanced Power Management"
        <tesla-dev.opensolaris.org>
List-Unsubscribe: <http://mail.opensolaris.org/mailman/options/tesla-dev>,
        <mailto:<email address hidden>?subject=unsubscribe>
List-Archive: <http://mail.opensolaris.org/pipermail/tesla-dev>
List-Post: <mailto:<email address hidden>>
List-Help: <mailto:<email address hidden>?subject=help>
List-Subscribe: <http://mail.opensolaris.org/mailman/listinfo/tesla-dev>,
        <mailto:<email address hidden>?subject=subscribe>
X-List-Received-Date: Thu, 12 Feb 2009 01:16:13 -0000
X-List-Received-Date: Thu, 12 Feb 2009 01:16:13 -0000
X-List-Received-Date: Thu, 12 Feb 2009 01:16:13 -0000
X-List-Received-Date: Thu, 12 Feb 2009 01:16:13 -0000
X-List-Received-Date: Thu, 12 Feb 2009 01:16:13 -0000
X-List-Received-Date: Thu, 12 Feb 2009 01:16:13 -0000

This is a multi-part message in MIME format.

--Boundary_(ID_r1P2Tq7K/ReXv3hja2KS8g)
Content-type: text/plain; format=flowed; charset=ISO-8859-1
Content-transfer-encoding: 7BIT

Li, Aubrey wrote:
> Rafael Vanoni wrote:
>
>> Here's a patch to show different c/p-state messages on sparc. I'm also
>> suggesting adding "(idleness)" to the c-state title on x86.
>>
>> Current:
>> C-states Avg residency P-states (frequencies)
>>
>> New x86:
>> C-states (idleness) Avg residency P-states (frequencies)
>>
>> New sparc:
>> Idle states Avg residency Frequency levels
>>
>> Let me know what you think.
>>
>> Thanks,
>> Rafael
>
> Where is message.c?
>
> -Aubrey

Good thing no one is keeping count of how many times I forgot to hg add
new files :)

Rafael

--Boundary_(ID_r1P2Tq7K/ReXv3hja2KS8g)
Content-type: text/plain; name=messages.diff
Content-transfer-encoding: 7BIT
Content-disposition: inline; filename=messages.diff

diff -r b1289ac55099 usr/src/cmd/powertop/amd64/Makefile
--- a/usr/src/cmd/powertop/amd64/Makefile Tue Jan 06 20:10:19 2009 -0800
+++ b/usr/src/cmd/powertop/amd64/Makefile Wed Feb 11 17:09:55 2009 -0800
@@ -25,7 +25,7 @@
 include ../Makefile.com
 include ../../Makefile.cmd.64

-MACH_OBJS = dtp_events.o
+MACH_OBJS = dtp_events.o messages.o
 SRCS += $(MACH_OBJS:%.o=%.c)

 .KEEP_STATE:
diff -r b1289ac55099 usr/src/cmd/powertop/amd64/messages.c
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/usr/src/cmd/powertop/amd64/messages.c Wed Feb 11 17:09:55 2009 -0800
@@ -0,0 +1,45 @@
+/*
+ * Copyright 2008, Intel Corporation
+ * Copyright 2008, Sun Microsystems, Inc
+ *
+ * This file is part of PowerTOP
+ *
+ * This program file is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program in a file named COPYING; if not, write to the
+ * Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor,
+ * Boston, MA 02110-1301 USA
+ *
+ * Authors:
+ * Arjan van de Ven <email address hidden>
+ * Eric C Saxe <email address hidden>
+ * Aubrey Li <email address hidden>
+ */
+
+/*
+ * GPL Disclaimer
+ *
+ * For the avoidance of doubt, except that if any license choice other
+ * than GPL or LGPL is available it will apply instead, Sun elects to
+ * use only the General Public License version 2 (GPLv2) at this time
+ * for any software where a choice of GPL license versions is made
+ * available with the language indicating that GPLv2 or any later
+ * version may be used, or where a choice of which version of the GPL
+ * is applied is otherwise unspecified.
+ */
+
+/*
+ * amd64 platform specific display messages
+ */
+
+const char *g_msg_idle_state = "C-states (idleness)";
+const char *g_msg_freq_state = "P-states (frequencies)";
+
diff -r b1289ac55099 usr/src/cmd/powertop/common/display.c
--- a/usr/src/cmd/powertop/common/display.c Tue Jan 06 20:10:19 2009 -0800
+++ b/usr/src/cmd/powertop/common/display.c Wed Feb 11 17:09:55 2009 -0800
@@ -238,8 +238,7 @@
                (void) wbkgd(cstate_window, COLOR_PAIR(PT_COLOR_DEFAULT));
        }

- print(cstate_window, 0, 0, "%s", "Cn\t\t\tAvg residency\n");
-
+ print(cstate_window, 0, 0, "%s\t\tAvg residency\n", g_msg_idle_state);
        res = (((double)g_cstate_info[0].total_time / g_total_c_time)) * 100;
        (void) sprintf(c, "C0 (cpu running)\t\t(%.1f%%)\n", (float)res);
        print(cstate_window, 1, 0, "%s", c);
@@ -263,7 +262,7 @@
                print(cstate_window, i + 1, 0, "%s", c);
        }

- print(cstate_window, 0, 48, "%s", "P-states (frequencies)\n");
+ print(cstate_window, 0, 48, "%s", g_msg_freq_state);

        if (g_npstates < 2) {
                (void) sprintf(c, "%4lu Mhz\t%.1f%%",
diff -r b1289ac55099 usr/src/cmd/powertop/common/powertop.c
--- a/usr/src/cmd/powertop/common/powertop.c Tue Jan 06 20:10:19 2009 -0800
+++ b/usr/src/cmd/powertop/common/powertop.c Wed Feb 11 17:09:55 2009 -0800
@@ -150,7 +150,7 @@
                                usage();
                        break;
                case 'v':
- if (PTOP_ON_VERBOSE)
+ if (PTOP_ON_CPU || PTOP_ON_VERBOSE)
                                usage();
                        else
                                g_op_mode = PTOP_MODE_VERBOSE;
diff -r b1289ac55099 usr/src/cmd/powertop/common/powertop.h
--- a/usr/src/cmd/powertop/common/powertop.h Tue Jan 06 20:10:19 2009 -0800
+++ b/usr/src/cmd/powertop/common/powertop.h Wed Feb 11 17:09:55 2009 -0800
@@ -231,6 +231,11 @@
 extern char **g_argv;

 /*
+ * Platform specific messages
+ */
+extern const char *g_msg_idle_state;
+extern const char *g_msg_freq_state;
+/*
  * Suggestions related
  */
 extern void suggest_p_state(void);
diff -r b1289ac55099 usr/src/cmd/powertop/i386/Makefile
--- a/usr/src/cmd/powertop/i386/Makefile Tue Jan 06 20:10:19 2009 -0800
+++ b/usr/src/cmd/powertop/i386/Makefile Wed Feb 11 17:09:55 2009 -0800
@@ -24,7 +24,7 @@

 include ../Makefile.com

-MACH_OBJS = dtp_events.o
+MACH_OBJS = dtp_events.o messages.o
 SRCS += $(MACH_OBJS:%.o=%.c)

 .KEEP_STATE:
diff -r b1289ac55099 usr/src/cmd/powertop/i386/messages.c
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/usr/src/cmd/powertop/i386/messages.c Wed Feb 11 17:09:55 2009 -0800
@@ -0,0 +1,45 @@
+/*
+ * Copyright 2008, Intel Corporation
+ * Copyright 2008, Sun Microsystems, Inc
+ *
+ * This file is part of PowerTOP
+ *
+ * This program file is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program in a file named COPYING; if not, write to the
+ * Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor,
+ * Boston, MA 02110-1301 USA
+ *
+ * Authors:
+ * Arjan van de Ven <email address hidden>
+ * Eric C Saxe <email address hidden>
+ * Aubrey Li <email address hidden>
+ */
+
+/*
+ * GPL Disclaimer
+ *
+ * For the avoidance of doubt, except that if any license choice other
+ * than GPL or LGPL is available it will apply instead, Sun elects to
+ * use only the General Public License version 2 (GPLv2) at this time
+ * for any software where a choice of GPL license versions is made
+ * available with the language indicating that GPLv2 or any later
+ * version may be used, or where a choice of which version of the GPL
+ * is applied is otherwise unspecified.
+ */
+
+/*
+ * i386 platform specific display messages
+ */
+
+const char *g_msg_idle_state = "C-states (idleness)";
+const char *g_msg_freq_state = "P-states (frequencies)";
+
diff -r b1289ac55099 usr/src/cmd/powertop/sparcv9/Makefile
--- a/usr/src/cmd/powertop/sparcv9/Makefile Tue Jan 06 20:10:19 2009 -0800
+++ b/usr/src/cmd/powertop/sparcv9/Makefile Wed Feb 11 17:09:55 2009 -0800
@@ -25,7 +25,7 @@
 include ../Makefile.com
 include ../../Makefile.cmd.64

-MACH_OBJS = dtp_events.o
+MACH_OBJS = dtp_events.o messages.o
 SRCS += $(MACH_OBJS:%.o=%.c)

 .KEEP_STATE:
diff -r b1289ac55099 usr/src/cmd/powertop/sparcv9/messages.c
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/usr/src/cmd/powertop/sparcv9/messages.c Wed Feb 11 17:09:56 2009 -0800
@@ -0,0 +1,45 @@
+/*
+ * Copyright 2008, Intel Corporation
+ * Copyright 2008, Sun Microsystems, Inc
+ *
+ * This file is part of PowerTOP
+ *
+ * This program file is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program in a file named COPYING; if not, write to the
+ * Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor,
+ * Boston, MA 02110-1301 USA
+ *
+ * Authors:
+ * Arjan van de Ven <email address hidden>
+ * Eric C Saxe <email address hidden>
+ * Aubrey Li <email address hidden>
+ */
+
+/*
+ * GPL Disclaimer
+ *
+ * For the avoidance of doubt, except that if any license choice other
+ * than GPL or LGPL is available it will apply instead, Sun elects to
+ * use only the General Public License version 2 (GPLv2) at this time
+ * for any software where a choice of GPL license versions is made
+ * available with the language indicating that GPLv2 or any later
+ * version may be used, or where a choice of which version of the GPL
+ * is applied is otherwise unspecified.
+ */
+
+/*
+ * amd64 platform specific display messages
+ */
+
+const char *g_msg_idle_state = "Idle states\t";
+const char *g_msg_freq_state = "Frequency levels";
+

--Boundary_(ID_r1P2Tq7K/ReXv3hja2KS8g)--

Related branches

description: updated
Revision history for this message
Mark Sapiro (msapiro) wrote :

I am unable to duplicate this error with your included message, Mailman 2.1.12rc2, Python 2.4.3 and Python email 3.0.1.

Can you give me anything more specific about the configuration under which this occurred. This would appear to be a serious problem, and we claim that Python 2.4.x and Mailman 2.1.12 is a viable combination so I would like to fix it

Revision history for this message
Mark Sapiro (msapiro) wrote :

Even though I was unable to duplicate the exception in my Python 2.4 environment, I see how the error could occur and I have implemented a workaround.

Thanks for the report.

Changed in mailman:
assignee: nobody → msapiro
status: New → Fix Committed
Revision history for this message
aduritz (s-angelov) wrote :

i have experienced the same problem with my 2.1.12rc2 installation. after reading the comments above i upgraded to 2.1.12 (final), however digests are stuck and not being processed and i still get the following error in my error log:

Mar 08 17:42:14 2009 (23828) send_digests() failed: decoding Unicode is not supported
Mar 08 23:29:11 2009 (23828) send_digests() failed: decoding Unicode is not supported
Mar 08 23:52:05 2009 (23828) send_digests() failed: decoding Unicode is not supported

i am using opensolaris snv_105 and python 2.4.4.

Revision history for this message
Mark Sapiro (msapiro) wrote :

If possible, can you post the lists/LISTNAME/digest.mbox file, or email it to me at <email address hidden>.

Unfortunately, in trying to protect against this kind of error stopping the list altogether, we catch the exception and don't log the traceback. The underlying problem is almost certainly in Scrubber.py, but the only issue of this nature that I'm aware of in Scrubber.py was fixed in 2.1.12rc2.

So I would like to get your digest.mbox to try and duplicate the problem. If you are unwilling to send or post the digest.mbox, please try the attached ToDigest.patch and report the traceback.

Revision history for this message
elaine ashton (elaine-ashton) wrote : Re: [Bug 328353] Re: decoding unicode (still) not supported bug

On Mar 8, 2009, at 6:52 PM, aduritz wrote:

> i have experienced the same problem with my 2.1.12rc2 installation.
> after reading the comments above i upgraded to 2.1.12 (final), however
> digests are stuck and not being processed and i still get the
> following
> error in my error log:
>
> Mar 08 17:42:14 2009 (23828) send_digests() failed: decoding Unicode
> is not supported
> Mar 08 23:29:11 2009 (23828) send_digests() failed: decoding Unicode
> is not supported
> Mar 08 23:52:05 2009 (23828) send_digests() failed: decoding Unicode
> is not supported
>
> i am using opensolaris snv_105 and python 2.4.4.

Interesting. My env is/was snv_88 and python 2.4.4 as well. Try the
upgrade to python 2.5.1. I know it should work with 2.4.4 but, perhaps
there's something unusual going on with the opensolaris python package
and I don't see any updates in 105 or better for python.

e.

Revision history for this message
Mark Sapiro (msapiro) wrote :

I think there may be an issue with the opensolaris Python 2.4.4. I received this traceback from the patched ToDigest.py from aduritz.

Mar 09 12:27:56 2009 (6235) send_digests() failed: decoding Unicode is not supported
Mar 09 12:27:56 2009 qrunner(6235): Traceback (most recent call last):
Mar 09 12:27:56 2009 qrunner(6235): File "/usr/local/mailman/Mailman/Handlers/ToDigest.py", line 98, in process
Mar 09 12:27:56 2009 qrunner(6235): send_digests(mlist, mboxfp)
Mar 09 12:27:56 2009 qrunner(6235): File "/usr/local/mailman/Mailman/Handlers/ToDigest.py", line 144, in send_digests
Mar 09 12:27:56 2009 qrunner(6235): send_i18n_digests(mlist, mboxfp)
Mar 09 12:27:56 2009 qrunner(6235): File "/usr/local/mailman/Mailman/Handlers/ToDigest.py", line 341, in send_i18n_digests
Mar 09 12:27:56 2009 qrunner(6235): mcset = msg.get_content_charset('')
Mar 09 12:27:56 2009 qrunner(6235): File "/usr/lib/python2.4/email/Message.py", line 800, in get_content_charset
Mar 09 12:27:56 2009 qrunner(6235): charset = unicode(charset, 'us-ascii').encode('us-ascii')
Mar 09 12:27:56 2009 qrunner(6235): TypeError: decoding Unicode is not supported

Python 2.4 shipped with email 3.0.1 and the line " charset = unicode(charset, 'us-ascii').encode('us-ascii')" is not in Message.py in email 3.0.1. It is in email 2.5.8 which used to ship with Mailman, but as of Mailman 2.1.12 is no longer installed in Mailman's pythonlib, but in email 2.5.8 it is line 846 of Message.py, not line 800.

First, is there a /usr/local/mailman/pythonlib/email/ directory. If there is, move it aside or remove it and restart Mailman, as it should not be there in Mailman 2.1.12. If there was an email/ directory in /usr/local/mailman/pythonlib/, and moving it aside does not fix the problem, try the following interactive Python session

$ python
[...]
>>> import email
>>> email.__version__

to see what email version this is. It should be 3.0.1. If not, there is something wrong with the opensolaris Python install.

Revision history for this message
Mark Sapiro (msapiro) wrote :

I see in the mailman-users post at <http://mail.python.org/pipermail/mailman-users/2009-March/065398.html> that this is an upgrade from Mailman 2.1.1. Thus, I think it is quite possible that the problem is caused by a residual email package in /usr/local/mailman/pythonlib/email/. This should have been removed when Mailman 2.1.2rc2 was installed. If there is a /usr/local/mailman/pythonlib/email/ package, how was 2.1.12 installed? The 2.1.12 "make install" should have removed it.

Revision history for this message
elaine ashton (elaine-ashton) wrote :

On Mar 9, 2009, at 11:45 AM, Mark Sapiro wrote:

> I think there may be an issue with the opensolaris Python 2.4.4. I
> received this traceback from the patched ToDigest.py from aduritz.
>
> Mar 09 12:27:56 2009 (6235) send_digests() failed: decoding Unicode
> is not supported
> Mar 09 12:27:56 2009 qrunner(6235): Traceback (most recent call last):
> Mar 09 12:27:56 2009 qrunner(6235): File "/usr/local/mailman/
> Mailman/Handlers/ToDigest.py", line 98, in process
> Mar 09 12:27:56 2009 qrunner(6235): send_digests(mlist, mboxfp)
> Mar 09 12:27:56 2009 qrunner(6235): File "/usr/local/mailman/
> Mailman/Handlers/ToDigest.py", line 144, in send_digests
> Mar 09 12:27:56 2009 qrunner(6235): send_i18n_digests(mlist,
> mboxfp)
> Mar 09 12:27:56 2009 qrunner(6235): File "/usr/local/mailman/
> Mailman/Handlers/ToDigest.py", line 341, in send_i18n_digests
> Mar 09 12:27:56 2009 qrunner(6235): mcset =
> msg.get_content_charset('')
> Mar 09 12:27:56 2009 qrunner(6235): File "/usr/lib/python2.4/email/
> Message.py", line 800, in get_content_charset
> Mar 09 12:27:56 2009 qrunner(6235): charset = unicode(charset,
> 'us-ascii').encode('us-ascii')
> Mar 09 12:27:56 2009 qrunner(6235): TypeError: decoding Unicode is
> not supported
>
> Python 2.4 shipped with email 3.0.1 and the line " charset =
> unicode(charset, 'us-ascii').encode('us-ascii')" is not in
> Message.py in
> email 3.0.1. It is in email 2.5.8 which used to ship with Mailman, but
> as of Mailman 2.1.12 is no longer installed in Mailman's pythonlib,
> but
> in email 2.5.8 it is line 846 of Message.py, not line 800.
>
> First, is there a /usr/local/mailman/pythonlib/email/ directory. If
> there is, move it aside or remove it and restart Mailman, as it should
> not be there in Mailman 2.1.12. If there was an email/ directory in
> /usr/local/mailman/pythonlib/, and moving it aside does not fix the
> problem, try the following interactive Python session
>
> $ python
> [...]
>>>> import email
>>>> email.__version__
>
> to see what email version this is. It should be 3.0.1. If not, there
> is
> something wrong with the opensolaris Python install.

I don't see the $basedir/pythonlib/email dir

It would appear to be 3.0.2...

/opt/opensolaris.org/mailman> /usr/bin/python2.4
Python 2.4.4 (#1, Apr 10 2008, 07:06:40) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
 >>> import email
 >>> email.__version__
'3.0.2'

The 2.5.1 python I installed from sfw is 4.0.1

/opt/opensolaris.org/mailman> /usr/local/bin/python2.5
Python 2.5.1 (r251:54863, May 16 2007, 19:39:00)
[GCC 3.4.6] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
 >>> import email
 >>> email.__version__
'4.0.1'

e.

Revision history for this message
aduritz (s-angelov) wrote :

there is no email directory in /usr/local/mailman/pythonlib/. checking the email version shows that it is 3.0.2:

#python
Python 2.4.4 (#1, Dec 1 2008, 02:36:01) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import email
>>> email.__version__
'3.0.2'
>>> ^D

i have checked /usr/lib/python2.4/email/Message.py and it contains "charset = unicode(charset, 'us-ascii').encode('us-ascii')" on line 800.
the opensolaris buld that i am using also comes with python 2.5.2 (Python 2.5.2 (r252:60911, Dec 2 2008, 01:05:52) [C] on sunos5). email for the python 2.5.2 installation in version 4.0.2.

Revision history for this message
Mark Sapiro (msapiro) wrote :

This is apparently an issue with the 3.0.2 email package. You could try upgrading to Python 2.5.2. This apparently worked for elaine ashton's issue.

Or you could try the attached patch to the 3.0.2 email/Message.py. Note that while I'm confident that the patch will fix this specific issue, there may be other inconsistencies in email 3.0.2.

Revision history for this message
Mark Sapiro (msapiro) wrote :

After some experimentation, I think I have found the underlying problem in email 3.0.2 and the attached patch to Charset.py will fix it.

With this patch installed, the prior patch to Message.py is unnecessary although it won't actually hurt anything.

Revision history for this message
aduritz (s-angelov) wrote :

hi Mark and elaine,

i applied the patch for Charset.py and was able to successfully send the digest for one of the mailing lists.
i am concerned about the shunted messages. according to Marks's post at <http://mail.python.org/pipermail/mailman-users/2009-March/065399.html> (excuse me for cross-posting), messages should be unshunted in order to be archived. is it correct to assume that they will only be archived and not resend again to members of the list, thus duplicated ?

Revision history for this message
Mark Sapiro (msapiro) wrote :

> is it correct to assume that they will only be archived and not resend again to members of the list, thus duplicated ?

Yes, the messages that were shunted during archiving will be unshunted to the archive queue and only archived, not resent to the list. You can verify this as follows. Do

 bin/dumpdb qfiles/shunt/xxx.pck

This will dump two objects - the message and the message metadata. The metadata will have an attribute 'whichq' which is the original queu to which the message will be unshunted. If this is 'archive', the unshunted message will only be archived. If it is 'out' the unshunted message will be sent to the 'recipients' in the metadata. If this is 'in', the unshunted message will be passed through those handlers remaining in 'pipeline'.

Revision history for this message
aduritz (s-angelov) wrote :

strange... from 48 shunted messages only the last one has the 'whichq' attribute (set to /usr/local/mailman/qfiles/bounces') all the other messages have metadata similar to the one below:

<----- start object 2 ----->
{'received_time': 1236449516.963038, 'version': 3, '_parsemsg': False}
[----- end pickle file -----]

Revision history for this message
aduritz (s-angelov) wrote :

i tried unshunting only one of the shunted messages, it does not go to the archive queue, instead i get the following in my logs/vette file:

Mar 10 00:55:52 2009 (26791) Mailman post from <email address hidden> held, message-id=<email address hidden>: Post by non-member to a members-only list

Revision history for this message
aduritz (s-angelov) wrote :

it looks like the upgrade fron 2.1.12rc2 to 2.1.12 final somehow altered the shunted files. i checked my backup copy of the qfiles dir prior to upgrading to 2.1.12 final and the .pck files look different. i will restore there and try to unshunt them.

Revision history for this message
Mark Sapiro (msapiro) wrote :

Beginning with Mailman 2.1.11, there is a cull_bad_shunt cron which by default deletes any shunt queue entries older than a week.

You have to examine each of the shunted messages with bin/dumpdb to know what the message is and to where it will be unshunted. The particular message that you unshunted that wound up being held as a non-member post was probably shunted for a different reason. You can check mailman's error log for the 'base name' (the file name without the .pck extension) to see why it was shunted.

If there is no 'whichq' the message will be unshunted to the 'in' queue by default. But something is wrong. Can you find the error log entries for those shunted messages. It is a different issue.

Note that the problem you had originally with messages shunted during archiving may have been fixed between 2.1.12rc2 and 2.1.12, so it is not surprising that those 'archive' queue shunted messages aren't there.

Revision history for this message
aduritz (s-angelov) wrote :

thanks Mark! i seem to have healthy lists once again and the archives got all the missing messages. it turned out that not only the .pck files were altered somehow but the names were totally different.

thank you!

St.

Revision history for this message
Mark Sapiro (msapiro) wrote :

Actually, the only place that would create a shunt queue entry without a 'whichq' as you quote above should also log a 'Dequeuing message destined for missing list: LISTNAME' message in the error log. Are you seeing those? This is really a "can't happen" kind of error.

Revision history for this message
Mark Sapiro (msapiro) wrote :

The timestamps were probably different too. The original messages shunted during archiving were probably deleted by cron/cull_bad_shunt, and the shunt queue entries you were seeing were more recent for another reason.

Revision history for this message
Mark Sapiro (msapiro) wrote :

I have changed the status back to In Progress since there is an issue in the Python email library version 3.0.2 which shipped with python 2.4.4 through Python 2.4.6. The patch to email/Charset.py attached here will fix it, but it isn't otherwise fixed.

Changed in mailman:
status: Fix Committed → In Progress
Revision history for this message
aduritz (s-angelov) wrote :

i don't see the 'Dequeuing message destined...' error. you are right that timestamps are different and as i said names of the files are different, however examinig the content i see the same messages inside... only the metadata has been changed.

Revision history for this message
Mark Sapiro (msapiro) wrote :

You are correct. There is apparently a bug in bin/update which tries to convert pre Mailman 2.1.5 queue entries to the current format, but it loses the metadata from 2.1.5 and later entries. I'll have to fix that.

Revision history for this message
ovadbar (ovadbar) wrote :

I tried to add both patches and the same error shows up.
I also don't want to use a different version of python because of another error that shows up.

Before I ran the fix I was not able to run bin/arch now I am. My current solution is to run a cronjob to run bin/arch since it seemed to be a partial solution.

Revision history for this message
Mark Sapiro (msapiro) wrote :

ovadbar: I assume "both patches" are those in comments #10 and #11. Did you restart Mailman after applying the patches? If so, please post one of the tracebacks from the archiving error. Also, please post the output from an interactive bin/withlist session similar to the following:

[msapiro@msapiro ...f/test-mailman]$ bin/withlist -i
No list name supplied.
Python 2.6.5 (r265:79063, May 1 2010, 12:04:38)
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import email
>>> email.__version__
'4.0.1'
>>>

Revision history for this message
ovadbar (ovadbar) wrote :

Thanks I had not restarted mailman.
Both patches were the ones in 10 and 11.
What's strange is that the owner of the mailman qrunner process was not mailman me or root. In fact it was a person that had been added to the project recently in order to use samba but doesn't use uniix.

Thank you for the help.

Revision history for this message
Mark Sapiro (msapiro) wrote :

If 'mailmanctl start' is run as root as it should be, the owner of all the processes should be the mailman user, so yes, that's strange.

Revision history for this message
Mark Sapiro (msapiro) wrote :

Changing status back to Fix Released as the underlying issue is fixed in current releases of the Python email package.

Changed in mailman:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.