Message held for moderation crashes mailman when used with MySQL

[Alex King]

There is a known problem when the version of mailman in 18.04LTS is used with a MySQL database.

Resulting in a 500 error in postorius when going to "Held Messages" and a traceback in the logs, ending with:

json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0)

According to Mark Sapiro:
> The issue is in storing a pickled representation of the json encoding of
> 'rule_misses'. This gets very long if you have lots of header filters,
> and with at least the mysql backend gets truncated in the database which
> causes the Unterminated string error.

> This is <https://gitlab.com/mailman/mailman/issues/385> which should be
> fixed by <https://gitlab.com/mailman/mailman/merge_requests/333>. The
> fix in in Mailman core version 3.2.0

See the thread at https://<email address hidden>/thread/33UEST4ANA3NVCGQMD6NVFZAQFKCT7ET/

Also the related bug report at https://gitlab.com/mailman/mailman/issues/576 which shows the traceback from postorius.

I note from /usr/share/doc/mailman3/README.Debian:

 For productive setups, SQLite3 is *not* recommended as database backend.
PostgreSQL or MySQL should be used instead. The database configuration can
be automated by installing 'dbconfig-pgsql' or 'dbconfig-mysql' and running
'dpkg-reconfigure mailman3'. A respective local or remote database server
has to be available in this case.

Perhaps the easiest "fix" for 18.04 (certainly not my preference, I'd prefer an actual bug fix) is to put somewhere a recommendation to only use PostreSQL for production setups.

Or I'm wondering if a DB model change to boost the size of the relevant varchar column in the DB would be a possible workaround with minimal impact?

Thanks,
Alex

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: mailman3 3.1.1-9
ProcVersionSignature: Ubuntu 4.15.0-47.50-generic 4.15.18
Uname: Linux 4.15.0-47-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.6
Architecture: amd64
Date: Sun Apr 7 10:54:31 2019
InstallationDate: Installed on 2018-02-10 (420 days ago)
InstallationMedia: Ubuntu-Server 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801)
PackageArchitecture: all
SourcePackage: mailman3
UpgradeStatus: Upgraded to bionic on 2018-08-22 (227 days ago)

[Christian Ehrhardt  (paelzer)]

Thank you for the report, this really is an ugly error.
I'll tag up the bug as fixed in Cosmic and later already, but need to discuss how to proceed in this case. The upstream discussion mentioned having a lot of "header filters".
You might help me understand, is that something the people commonly have or something that creeps up over time?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Bryce Harrington (bryce)]

Dropping the ubuntu-server subscription, since there's still not enough information to proceed, and since the server team has not adopted mailman3 for active maintenance in main.

It may be better practice to install mailman3 using upstream's recommended practices so you can more easily pull down fixes from them as they become available. That said, if you identify specific patches worth inclusion in the Ubuntu 18.04 package, you're welcome to follow the SRU policy to get them reviewed and incorporated:

https://wiki.ubuntu.com/StableReleaseUpdates