On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:

>On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
>
>> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
>
>> Need a script to upgrade from MM2 to MM3
>> https://bugs.launchpad.net/bugs/965532
>
>Here are some thoughts on a possible migration technique.
>I would request discussion and suggestions.
>
>In particular, what about the idea of converting the configuration file to
>HTML as an intermediate file format?  Selectable css could easily render it
>as a viewable report. It could still be edited by hand without too much
>difficulty.

It's an interesting idea.  As you observed, mm2 can export to XML, so it's not
such a big stretch.

w>Steps to migrate from MM2 to MM3
>
>1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
>include testing to assure that things are ready to create new lists.

Right, and it should be doable even while mm2 is still functional.

>2) Translate list configurations
>
>  a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this
>  to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy
>  would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would
>  populate configurations at the List level. It might also reformat selected
>  parameters. In particular, various <option type="radio" > entries might be
>  transformed into enumerations such as "Yes"/"No" or
>  "Hidden"/"Private"/"Public" rather than numerical values. This would
>  enhance readability.
>
>  b) TOOL3 would populate the World level with the MM2 defaults and
>  recursively promote common values up the tree, leaving only those entries
>  which would need to override their parent to derive the current
>  value. Values which match the parent would be flagged. (The inheritance
>  flag should be tri-state. "Differs from parent", "Same as parent",
>  "Inherited from parent")
>
>  c) At this point, the user might edit some of the configurations and rerun
>  TOOL3 adjusting the inheritance flag as appropriate.

I think the trickiest part will be what to do about subscriber information.
In mm2, this is always list-centric, but in mm3, you need to collate and
globalize all the membership information into the user database.  You can
probably do the same kind of up-promotion there, but it would be from
member->address->user.  IOW, if you see an address subscribed to a mailing
list with the same values across all those lists, put the preferences in the
user.  What happens if you see <email address hidden> subscribed to three different
lists with three different passwords?  That's a tough one because there's no
way to express that in mm3 (nor probably should there be).

So I think you will occasionally have to just resolve some conflicts by
flipping a coin.  In the case of passwords, perhaps you'd always make the user
do a password reset.

>  d) Now, we begin translation to MM3 configuration options. For each MM3
>  option, TOOL4 computes the equivalent value from the MM2 values. Each
>  computed value also gets the corresponding inheritance flag. Values that
>  cannot be computed from the available information get the "Inherited from
>  parent" flag. MM2 values used in computations are marked as "translated".

The user herself could probably write a script for this pipeline you're
proposing, that would allow her to do bulk transformations of configuration
variable.

>  e) After a chance to edit the MM3 configurations, TOOL5 would recompute
>  inheritance flags, report any MM2 values that have not been translated and
>  produce a copy of the configuration file simplified by removing all
>  inherited entries.
>
>  f) After a final inspection TOOL7 would actually import the configurations,
>  committing entries to the MM3 database.
>
>3) For the migration of rosters, we should be able to do it one subscription
>at a time through a pipeline that permits pre- and post- hooks.  A --dry-run
>option would be appropriate.

Almost definitely.  The --dry-run step which would produce an output of those
conflicts, and impossible situations.  The user would then have a chance to
re-edit the intermediate file so that the values can be better mapped to mm3.
It's probably worth doing for both the rosters and list configurations.

>  a) We can assume that each email address is a distinct person.  The
>  subscribers can utilize the UI to merge email addresses into a common
>  persona.

We'll probably need a "claim and merge" operation in the system.  And a way to
purge unclaimed addresses after a while.

>  b) We can also assume that each subscription overrides its parent in the
>  Persona-->EMailAddress-->Subscription hierarchy. The individual users can
>  use the UI to consolidate their selections.

A good challenge for the Postoriusians :)

> All of the tools should be written in Python, hopefully in a dialect that is
> common to all of the versions supported by MM3.

+1.  Today that would be 2.6 and 2.7.

> TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes
> as they come in. Similarly, in step 3, we can use TOOL1 and discard the list
> configuration information.

> TOOL2 can reformat the XML as HTML, thus making the input data into a
> viewable report. The inheritance flag would become a class attribute on the
> option. Would it make sense to go a step further and generate html forms and
> run a trivial http server on localhost? It might be easier to do this in
> django, but I think that requiring that level of installation is probably
> too much for the current situation.

It wouldn't be hard to do in standard Python.  OTOH, I'm not sure we want to
maintain a stack of html templates, forms, and form processing in the
conversion tool.

It's sounding like this suite of conversion tools may need to be a separate
sub-project.

-Barry