[MIR] mailman3 to replace mailman, or drop mailman to universe and off server iso

[Dimitri John Ledkov]

[Availability]
The package is already universe for quite a while and build/works fine so far.
It is for example already used for https://lists.canonical.com/mailman3/postorius/lists/

This source builds only docs and mailman3 itself (which provides mailman3-core).

[Rationale]
This is part of the MIR activity for all dependencies of mailman3
The "main" MIR of it is at bug 1775427:

Mailman (2) has only python2 support, but we strive for python3,
therefore Mailman3 which has python3 support should be promoted to main.

This is an evolution of the mailman2 core delivery engine of the system, but I can't guess how much it deviated so we need to recheck security (IMHO)

[Security]

No known CVEs found.
A few old issues can be found against mailman2 and one (but long fixed) for mailman 3
=> https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=mailman

[Quality assurance]

The mailman3 stacks as of now (Disco) installs fine and provides a base
config. But due to the nature of the package that needs further modification
to be of real use.

The package does in error cases ask a debconf-high question, but it is not going to be installed by default - so that is no MIR problem.

There are a few (2) Ubuntu bugs outstanding, but none of them is critical and stopping this MIR request.
There are a few (2) Debian bugs outstanding all seems to "just work" still, but when we main this in 19.10 we should at least evaluate and maybe adress this one:
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921445

There are regular upstream releases and Debian packages them and fixes regularly.

No exotic HW involved.

The package ships a test suite, but it is not run at build time for upstream bug #400.
It tests things in a dep8 test thou to provide coverage.

d/watch is set up and ok.

No Lintian warning except fairly recent newer Standards version.

The package does not rely on demoted or obsolete packages.

[UI standards]

Comes with 7 translations in .po files
No End-user applications that needs a standard conformant desktop file.

[Dependencies]

Some dependencies are not in main, but we drive MIR for all related packages
that are not in main at the same time.
Please check the list of bugs from the main Mailman3 MIR in bug 1775427 to get an overview.

[Standards compliance]
The package meets the FHS and Debian Policy standards.
The packaging itself is very straight forward and uses dh_* as much as possible - the d/rules fits on one screen.

[Maintenance]

The Server team will subscribe for the package for maintenance

[Background]
The package description explains the general purpose and context of the package well.

---

References to all the related MIR bug - one per affected package.

Status (simplified version of https://wiki.ubuntu.com/MIRTeam#Process_states)
   (nothing) means we have not even prepared the MIR yet
 - MIR not needed (e.g. dependency will be dropped)
 * all of the template fields for the MIR are complete
 ! waiting for MIR review
 ? cpaelzer needs experienced MIR team members to continue
 S needs security review to be accepted
 + MIR accepted
 x MIR nacked

Extra Flags which team members or on review feedback can be added:
 T extra work for the team that should be done before 20.04 (state todo after package)
 M extra work for the team before getting MIR to be acceptable (read as asap)
 C needed if only promoting mailman3 (core)
 H needed if only promoting mailman3 (core) plus Hyperkitty
 J no more needed for mailman3-web if we can cut node-less out

FLAGS | MIR-Bug | Package (Notes)
 TC
 MH
----------------------------------------
+ | bug 1820179 | coreapi
+ | bug 1820180 | coreschema
S | bug 1820181 | django-allauth
+ | bug 1820182 | django-filter
S | bug 1820183 | django-guardian
+ | bug 1820185 | django-haystack
S | bug 1820186 | django-mailman3
- | | django-paintstore ((no dep hyperkitty >=1.2.2-1)
+ | bug 1820188 | django-picklefield
+ | bug 1820189 | django-q
S | bug 1820190 | djangorestframework
S C | bug 1820191 | flufl.bounce
+ C | bug 1820192 | flufl.i18n
+ C | bug 1820193 | flufl.lock
M | bug 1820195 | glewlwyd (need to break dep [font corruption] or break packaging)
S | bug 1820196 | hyperkitty (avoid type error https://bugs.launchpad.net/ubuntu/+source/robot-detection/+bug/1820225/comments/2)
+ | bug 1820197 | itypes
- | n/a | jqueryui (no dep hyperkitty >=1.2.2-1)
+ C | bug 1820199 | lazr.config
+ C | bug 1820200 | lazr.delegates
-MJ | bug 1820201 | less.js (Dep will be cut out in Debian bug 924961)
- | n/a | libjs-jquery-colorpicker (no hyperkitty >=1.2.2-1)
S | bug 1820202 | libmatheval
S | bug 1820203 | libpgm
S C | bug 1775427 | mailman3
S | bug 1820204 | mailmanclient
S H | bug 1820205 | mailman-hyperkitty
+ | bug 1820206 | mailman-suite
-MJ | bug 1820207 | node-source-map (wait for node-less dep to be cut)
-MJ | bug 1820208 | nodejs (wait for node-less dep to be cut)
S+? | bug 1820209 | norm (check if embedded protolib is an issue)
+ | bug 1820982 | nose
S | bug 1820210 | postorius
M | bug 1820211 | python3-openid (change to python-openid)
M C | bug 1820212 | python-aiosmtpd (fix setuid, main loop and user nobody usage)
+T | bug 1820213 | python-arrow (non fatal build error on dateutil, tests)
+ | bug 1820214 | python-blessed
S+? | bug 1820215 | python-django-extensions (many functions in one package)
+ | bug 1820216 | python-django-gravatar2
STC | bug 1820217 | python-falcon (opt: update version before 20.04)
+ | bug 686045 | python-lockfile
+TC | bug 1820220 | python-public (opt: update version before 20.04)
+ | bug 1820221 | python-requests-oauthlib
+T | bug 1820223 | python-uritemplate (be a sync, better new upstream)
S | bug 1820224 | python-whoosh
+ | bug 1820225 | robot-detection
-M | n/a | ruby-sass (deprecated, use sassc - Debian 924629)
-M | TBD | sassc (Dep will be cut by Debian bug 924961)
-M | TBD | libsass (Dep will be cut by Debian bug 924961)
S | bug 1820226 | twitter-bootstrap3 (need JS expert tom complete SEC-Review)
S | bug 1820227 | uwsgi (FTBFS: #1820095) (SEC ~Nack, needs simpler replacement)
+ | bug 1820229 | wcwidth
+ | bug 1597439 | zeromq3
STC | bug 1820233 | zope.component (version update)
STC | bug 1820234 | zope.configuration (version update)
STC | bug 1820236 | zope.event (version update)
STC | bug 1820237 | zope.hookable (version update)
+TC | bug 1820238 | zope.i18nmessageid (version update)
STC | bug 1820239 | zope.schema (version update)

List Debian bugs we depend on being resolved as expected before
the dependency tree can completely be MIRed:
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924629
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924961

This might sometimes be slightly out of date, to cross check and for history on all of them check git at:
https://git.launchpad.net/~ubuntu-server/+git/mailman3-MIR-party/tree/package-list.txt

---

Past Decisions in regard to this:
mailman is python2, mailman3 is python3, but also it has new dependency stack due to adopting django but also splitting or rewriting a lot of its components.

Q: Do we want bin:mailman3 or bin:mailman3-full in main? (the latter pulls
   in src:mailman-suite as well). src:mailman-suite has potentially more
   things - django, hyperkitty, ruby-sass, uwsgi...
A: We analyzed this and we will need the full set of dependencies, even
   meaning MIRs are needed for all of the above listed (and more).

Q: Before proceeding further with assignment/doing technical work in
   driving above MIRs, a product level yes/no answers should probably be
   answered for above two questions.
A: Discussed in several sprints now, as uncomfortable as it is we will go
   through the mass-MIR processing

[Dimitri John Ledkov (xnox)]

@powersj @dpb1 - ping? i thought there was consensus the other day...

[Dimitri John Ledkov (xnox)]

mailman-suite / mailman3-web should change packaging to launch as python3, not python2 webapp.

[Joshua Powers (powersj)]

There is consensus to do this, we need the work broken down to determine if we can get it all done next cycle or over 2 cycles.

[Dimitri John Ledkov (xnox)]

@powersj I think the biggest blocker would be security reviews of all of these things.

[Joshua Powers (powersj)]

Assigning to Christian for investigation

[Christian Ehrhardt  (paelzer)]

Time to get it started, I assume this will accompany us for a while at least, but we have to start at some point.

Thanks xnox for the initial list of affected packages, but I assume it is much worse.
I agree that mass-MIR handling is probably the biggest step to take here.

[Christian Ehrhardt  (paelzer)]

I wanted to eventually file MIRs for all of these and reference the bugs here.
I wondered to drive all in this MIR, but that will break discussions as e.g. python-flufl and zope are so damn different that doing that in just this bug will make us loose track too much.
OTOH now that I did the deep check on dependencies, I'm afraid this just is too much either way.

I began with re-tracing those dependencies to be sure they are still valid.
It was time to tune my dependency check helper anyway and my fear of "more than we thought" was confirmed depending on what we select.

If you'd just (and only) look at the mailman3 binary from the mailman3 source, then the current dependency list is almost correct, just zope.i18nmessageid needs to be added which is a second level dependency.

But I think users would actually expect mailman3-full being the main package we'd want.
Read the description of mailman3:

 Description: Mailing list management system
  This is GNU Mailman version 3, a mailing list management system. This package
  provides the core delivery engine of the system, which handles the mailing
  lists data, receives messages, handles the moderation and processing of these
  messages and delivers them to the mailing lists subscribers. It communicates
  with the other components through a private administrative REST API.
  .
  Default database backend is SQLite3 in order to not break automated
  installations. For productive setups, PostgreSQL or MySQL are much better
  options though. See README.Debian for further information.
  .
  In order to get the full Mailman3 system, the metapackage 'mailman3-full'
  should be installed.

But using mailman3-full pulls in way too many dependencies by mailman3-web and python3-mailman-hyperkitty. Then we would explode from 15 to 49 dependencies including very non-server'ish ones like fonts and way more zope.

But as I said coming from mailman (2) you'd expect just that - as formerly the web integration and all of it was part of mailman (2).

We'd also want the lazr related packages be owned by foundations I guess who own the rest of lazr already.

I'll attach the full list of dependencies, so everyone interested can read into it.
My recommendation would be to declare only the core mailman3 for main inclusion, not mailman3-full.
I'll bring it up for team discussion later today.

[Christian Ehrhardt  (paelzer)]

Summary (mostly as a working theory for now):
Seeding/Unseeding:
- we would only select and fully support the core mailman3 (to keep things reviewable and supportable)
- all the web bits pulled in by mailman3-full via mailman3-web and hyperkitty would not get into main
- once migrated mailman2 would be demoted (another nail in the python2 coffin)
Ownership:
- flufl*, lazr.*, python-*, nose as listed are generic python libraries which I'd ask foundations to own
- zope.* as listed would be on Server Team
- mailman3 itself would be on Server Team

next: I will talk to various people to get some acks on that approach

[Christian Ehrhardt  (paelzer)]

To help the discussions to come I isolated the dependencies of mailman3-web
mailman-hyperkitty is very similar.

[Christian Ehrhardt  (paelzer)]

Discussed in the Team, and the conclusion is that we have to go the big route of mailman3-full
Maybe breaking out hyperkitty to a suggests, but that is not helping too much.

I'll update the bug tasks tomorrow and prep a summarized Dependency list of the planned approach. I will then send a few mails about taking over ownership (as mentioned not all we need is server'ish).

Once those are acknowledged we can start to do the mass filing of per package MIR requests.

[Christian Ehrhardt  (paelzer)]

Existing option left, let mailman[23] go - not my decision thou - might bring it up on the sprint thou.

[Dimitri John Ledkov (xnox)]

Well #is has now deployed mailman3, i think it all should go into main.

[Christian Ehrhardt  (paelzer)]

On Mon, Feb 18, 2019 at 11:41 PM Dimitri John Ledkov
<email address hidden> wrote:
> Well #is has now deployed mailman3, i think it all should go into main.

Oh yeah Dimitri - decisions are actually made already, there will be a
mass MIR filing (and then processing) hopefully starting this week to
spread the load.
Tthe intention as of now is to complete this in 19.10

[Christian Ehrhardt  (paelzer)]

While preparing the MIR bugs for the involved soruces I found that some we initially detected are already in main pulled in by python3-django-openstack-auth:
- python3-csscompressor
- python3-django-compressor
- python3-rcssmin
- python3-rjssmin

Dropping those from our list we have to work on for mailman3.

Furthermore I have realized (thanks wgrant for explaining) that LP had problems with the huge number of tasks (only when there are many subscribers). Therefore I ahve dropped them to allow updates to the bug.
Instead I'll add the bug references (once the MIRs are created) to the Description of this bug.
From there one can as well easily get to the related MIRs.

[Christian Ehrhardt  (paelzer)]

Work to cut out more of the dependencies we really don't think to be approproate (ruby/nodejs) was discussed upstream and started with Debian:
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924629
- https://bugs.debian.org/cgi-bin/gbugreport.cgi?bug=924961

The conclusion of this might have to wait until Buster is released.
We will carry on with all the other MIRs as they seem to need to happen as-is.
Once node/rub/sass is resolved we can re-generate the dependency list and add/remove extra packages as needed then.

[Christian Ehrhardt  (paelzer)]

MIR review on zope.* done (I start with the easier bits first, to have the feeling of hope :-) )
Due to their integration into a web stack I marked them in the description and the respective bugs as waiting for security review.

[Christian Ehrhardt  (paelzer)]

Updated after completion of flufl.* and lazr.* MIR review.

FYI: after discussion with Doko we will discuss in todays MIR team meeting if/how to distribute further reviews (especially the less trivial cases) among the team to spread the load as usual, but also not carry too much of the "self-ack" feeling that one might think.

[Christian Ehrhardt  (paelzer)]

Updated description after the review of django.* and marking some packages for Team review already

[Christian Ehrhardt  (paelzer)]

zeromq ftbfs is fixed and there was no explicit reason to demote it, so we can use the old MIR and consider that one done. Updated the description.

[Christian Ehrhardt  (paelzer)]

Updated for MIR review of coreapi coreschema robot-detection and wcwidth

[Christian Ehrhardt  (paelzer)]

[Duplication]
This is part of the six core packages of mailman3 that pull in further components as needed.
Since this represents mailman doing mailing list processing there is a duplication to mailman2.
But the intention is to stop seeding mailman2 as soon as mailman3 got promoted.

[Embedded sources and static linking]
This package does not contain embedded library sources.
This package doe not statically link to libraries.
No Go package

[Security]
I can confirm that there seems to be no CVE/Security history for this package.
But there is enough for mailman2 (and a bit for 3) that we should expect not (much) less in the future when it becomes more widely used.
=> https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=mailman

It Does not:
- uses old webkit
- uses lib*v8 directly
- integrates arbitrary javascript into the desktop
- deals with system authentication
- uses centralized online accounts

But it does:
- processes arbitrary web (actually mail) content
- parse data formats
- open a port (Rest API through python3-httplib2)
- runs a daemon as root

This is core mail forwarding code.
A security review is recommended on this package.

[Common blockers]
- builds fine at the moment
- server Team committed to subscribe once this gets promoted (enough for now)
- code is not user visible, no translation needed
- no build time tests, but tests run as autopkgtest (as they need services up)

[Packaging red flags]
- no library with classic symbol tracking
- watch file is present
- Lintian warnings are present bug ok
- debian/rules is rather clean
- no usage of Built-Using
- no golang package that would make things harder

Ubuntu Delta is present but minimal and ok.
We just drop the lynx dep to a suggest) which is fine.
Maybe we can one day convince Debian to do so as well.

[Upstream red flags]
- no suspicious errors during build (a few warnings, but nothing serious)
- it is pure python, so no incautious use of malloc/sprintf
- no use of sudo, gksu
- no use of pkexec
- no use of LD_LIBRARY_PATH
- no important open bugs
- no Dependency on webkit, qtwebkit, libgoa-*
- no embedded copies in upstream either

[Summary]
Ack from the MIR-Teams POV, but as outlined above a security review is recommended.
Assigning the security Team.

[Christian Ehrhardt  (paelzer)]

Updated for the complete of the MIR review on main mailman3 components: hyperkitty, mailman* and postorious

[Christian Ehrhardt  (paelzer)]

Updates for completing python-arrow python-blessed python-public and python-uritemplate
Some new non blocking but recommended tasks added, as usual marked in the overview and details can be found in the respective bugs.

[Christian Ehrhardt  (paelzer)]

Completed the review on the remaining pytohn libs, overall the remaining TODOs are some tasks for the server Team (breakign dependencies), a lot of security reviews, but also 5 tasks for the MIR team.
Those split into 3 discussion and 2 package reviews.
As discussed I'll bring those up on next weeks MIR Team meeting.

[Mathieu Trudel-Lapierre (cyphermox)]

Setting mailman to Triaged; let's get this off the "New MIRs" list -- it's not new, just not completed yet. mailman3's status nicely captures this.

[Joshua Powers (powersj)]

After dozens of reviews by the security team, we are ending the MIR for mailman3 and the enormous list of dependencies due to the complexity of the package, difficulties maintaining, and security concerns. mailman3 will live in universe.