Need a script to upgrade from MM2 to MM3

Bug #965532 reported by Barry Warsaw
32
This bug affects 5 people
Affects Status Importance Assigned to Milestone
GNU Mailman
Fix Released
Critical
Richard Wackerbarth

Bug Description

We need a script, documentation, or other procedure to help people migrate from Mailman 2 to Mailman 3.

Tags: mailman3
Barry Warsaw (barry)
Changed in mailman:
status: New → Confirmed
importance: Undecided → High
importance: High → Critical
milestone: none → 3.0.0b2
Barry Warsaw (barry)
Changed in mailman:
assignee: nobody → Richard Wackerbarth (wacky)
Revision history for this message
Richard Wackerbarth (wacky) wrote : Re: [Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3
Download full text (3.7 KiB)

On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:

> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:

> Need a script to upgrade from MM2 to MM3
> https://bugs.launchpad.net/bugs/965532

Here are some thoughts on a possible migration technique.
I would request discussion and suggestions.

In particular, what about the idea of converting the configuration file to HTML as an intermediate file format?
Selectable css could easily render it as a viewable report. It could still be edited by hand without too much difficulty.

Richard "Wacky" Wackerbarth
- - - - - -

Steps to migrate from MM2 to MM3

1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should include testing to assure that things are ready to create new lists.

2) Translate list configurations

  a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would populate configurations at the List level. It might also reformat selected parameters. In particular, various <option type="radio" > entries might be transformed into enumerations such as "Yes"/"No" or "Hidden"/"Private"/"Public" rather than numerical values. This would enhance readability.

  b) TOOL3 would populate the World level with the MM2 defaults and recursively promote common values up the tree, leaving only those entries which would need to override their parent to derive the current value. Values which match the parent would be flagged. (The inheritance flag should be tri-state. "Differs from parent", "Same as parent", "Inherited from parent")

  c) At this point, the user might edit some of the configurations and rerun TOOL3 adjusting the inheritance flag as appropriate.

  d) Now, we begin translation to MM3 configuration options. For each MM3 option, TOOL4 computes the equivalent value from the MM2 values. Each computed value also gets the corresponding inheritance flag. Values that cannot be computed from the available information get the "Inherited from parent" flag. MM2 values used in computations are marked as "translated".
  e) After a chance to edit the MM3 configurations, TOOL5 would recompute inheritance flags, report any MM2 values that have not been translated and produce a copy of the configuration file simplified by removing all inherited entries.

  f) After a final inspection TOOL7 would actually import the configurations, committing entries to the MM3 database.

3) For the migration of rosters, we should be able to do it one subscription at a time through a pipeline that permits pre- and post- hooks. A --dry-run option would be appropriate.

  a) We can assume that each email address is a distinct person. The subscribers can utilize the UI to merge email addresses into a common persona.

  b) We can also assume that each subscription overrides its parent in the Persona-->EMailAddress-->Subscription hierarchy. The individual users can use the UI to consolidate their selections.

Some additional thoughts:

 All of the tools should be written in Python, hopefully in a dialect that is common to all of t...

Read more...

Revision history for this message
Barry Warsaw (barry) wrote : Re: [Mailman-Developers] [Bug 965532] [NEW] Need a script to upgrade from MM2 to MM3
Download full text (5.7 KiB)

On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:

>On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
>
>> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
>
>> Need a script to upgrade from MM2 to MM3
>> https://bugs.launchpad.net/bugs/965532
>
>Here are some thoughts on a possible migration technique.
>I would request discussion and suggestions.
>
>In particular, what about the idea of converting the configuration file to
>HTML as an intermediate file format? Selectable css could easily render it
>as a viewable report. It could still be edited by hand without too much
>difficulty.

It's an interesting idea. As you observed, mm2 can export to XML, so it's not
such a big stretch.

w>Steps to migrate from MM2 to MM3
>
>1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
>include testing to assure that things are ready to create new lists.

Right, and it should be doable even while mm2 is still functional.

>2) Translate list configurations
>
> a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this
> to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy
> would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would
> populate configurations at the List level. It might also reformat selected
> parameters. In particular, various <option type="radio" > entries might be
> transformed into enumerations such as "Yes"/"No" or
> "Hidden"/"Private"/"Public" rather than numerical values. This would
> enhance readability.
>
> b) TOOL3 would populate the World level with the MM2 defaults and
> recursively promote common values up the tree, leaving only those entries
> which would need to override their parent to derive the current
> value. Values which match the parent would be flagged. (The inheritance
> flag should be tri-state. "Differs from parent", "Same as parent",
> "Inherited from parent")
>
> c) At this point, the user might edit some of the configurations and rerun
> TOOL3 adjusting the inheritance flag as appropriate.

I think the trickiest part will be what to do about subscriber information.
In mm2, this is always list-centric, but in mm3, you need to collate and
globalize all the membership information into the user database. You can
probably do the same kind of up-promotion there, but it would be from
member->address->user. IOW, if you see an address subscribed to a mailing
list with the same values across all those lists, put the preferences in the
user. What happens if you see <email address hidden> subscribed to three different
lists with three different passwords? That's a tough one because there's no
way to express that in mm3 (nor probably should there be).

So I think you will occasionally have to just resolve some conflicts by
flipping a coin. In the case of passwords, perhaps you'd always make the user
do a password reset.

> d) Now, we begin translation to MM3 configuration options. For each MM3
> option, TOOL4 computes the equivalent value from the MM2 values. Each
> computed value also gets the corresponding inheritance flag. Values that
> cannot be computed from the available information get the "Inh...

Read more...

Revision history for this message
Richard Wackerbarth (wacky) wrote :
Download full text (6.8 KiB)

Many MM admins are NOT programmers. I could do the transforms in either Python or XSL.

Which would be more acceptable to our users who might want to customize the transform? In either case, the transforms would be kept in a separate file from the rest of the engine and pipeline so that "where to customize" is clear.

Richard

On Apr 10, 2012, at 3:46 PM, Barry Warsaw <email address hidden> wrote:

> On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:
>
>> On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
>>
>>> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
>>
>>> Need a script to upgrade from MM2 to MM3
>>> https://bugs.launchpad.net/bugs/965532
>>
>> Here are some thoughts on a possible migration technique.
>> I would request discussion and suggestions.
>>
>> In particular, what about the idea of converting the configuration file to
>> HTML as an intermediate file format? Selectable css could easily render it
>> as a viewable report. It could still be edited by hand without too much
>> difficulty.
>
> It's an interesting idea. As you observed, mm2 can export to XML, so it's not
> such a big stretch.
>
> w>Steps to migrate from MM2 to MM3
>>
>> 1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
>> include testing to assure that things are ready to create new lists.
>
> Right, and it should be doable even while mm2 is still functional.
>
>> 2) Translate list configurations
>>
>> a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this
>> to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy
>> would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would
>> populate configurations at the List level. It might also reformat selected
>> parameters. In particular, various <option type="radio" > entries might be
>> transformed into enumerations such as "Yes"/"No" or
>> "Hidden"/"Private"/"Public" rather than numerical values. This would
>> enhance readability.
>>
>> b) TOOL3 would populate the World level with the MM2 defaults and
>> recursively promote common values up the tree, leaving only those entries
>> which would need to override their parent to derive the current
>> value. Values which match the parent would be flagged. (The inheritance
>> flag should be tri-state. "Differs from parent", "Same as parent",
>> "Inherited from parent")
>>
>> c) At this point, the user might edit some of the configurations and rerun
>> TOOL3 adjusting the inheritance flag as appropriate.
>
> I think the trickiest part will be what to do about subscriber information.
> In mm2, this is always list-centric, but in mm3, you need to collate and
> globalize all the membership information into the user database. You can
> probably do the same kind of up-promotion there, but it would be from
> member->address->user. IOW, if you see an address subscribed to a mailing
> list with the same values across all those lists, put the preferences in the
> user. What happens if you see <email address hidden> subscribed to three different
> lists with three different passwords? That's a tough one because there's no
> way to express that in mm3 (nor prob...

Read more...

Barry Warsaw (barry)
Changed in mailman:
milestone: 3.0.0b2 → 3.0.0b3
Barry Warsaw (barry)
Changed in mailman:
milestone: 3.0.0b3 → 3.0.0b4
Revision history for this message
galuszkak (galuszkak) wrote :

That is really important thing to make it work.

Revision history for this message
Kẏra (thekyriarchy) wrote :

has any work on this taken place? is there documentation anywhere?

Revision history for this message
Barry Warsaw (barry) wrote :

abompard has a branch, which I've slowly been working through.

Barry Warsaw (barry)
Changed in mailman:
milestone: 3.0.0b4 → none
Revision history for this message
Jack Underwood (ocean-wolf) wrote :

Should this bug have had its milestone removed on the release of b4?

Any help needed? How much work do you envisage remains here? As a python coder and mailman admin, I wonder if I can help.

Revision history for this message
Kẏra (thekyriarchy) wrote :

I can't tell if the code was merged or not. It says "abstain": https://code.launchpad.net/~abompard/mailman/import21/+merge/192146

Revision history for this message
Barry Warsaw (barry) wrote : Re: [Bug 965532] Re: Need a script to upgrade from MM2 to MM3

On Sep 08, 2014, at 12:04 AM, Kẏra wrote:

>I can't tell if the code was merged or not. It says "abstain":
>https://code.launchpad.net/~abompard/mailman/import21/+merge/192146

Yes, that branch was merged.

Revision history for this message
Kẏra (thekyriarchy) wrote :

great, so this bug can be marked as resolved?

Revision history for this message
Kẏra (thekyriarchy) wrote :

is this bug still unresolved despite the code merge?

Revision history for this message
Barry Warsaw (barry) wrote :

On Feb 03, 2015, at 10:21 PM, Kẏra wrote:

>is this bug still unresolved despite the code merge?

It probably needs more testing with real-world data.

Revision history for this message
Abhilash Raj (raj-abhilash1) wrote :

This bug has been moved to the new gitlab repo here: https://gitlab.com/maxking/mailman/issues/4

Revision history for this message
Abhilash Raj (raj-abhilash1) wrote :

This bug has been moved to the new gitlab repo here: https://gitlab.com/maxking/mailman/issues/5

Revision history for this message
Abhilash Raj (raj-abhilash1) wrote :

Please ignore the above two comments, the new url on gitlab is : https://gitlab.com/mailman/mailman/issues/4

Mark Sapiro (msapiro)
Changed in mailman:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.