On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:
>On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote:
>
>> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
>
>> Need a script to upgrade from MM2 to MM3
>> https://bugs.launchpad.net/bugs/965532
>
>Here are some thoughts on a possible migration technique.
>I would request discussion and suggestions.
>
>In particular, what about the idea of converting the configuration file to
>HTML as an intermediate file format? Selectable css could easily render it
>as a viewable report. It could still be edited by hand without too much
>difficulty.
It's an interesting idea. As you observed, mm2 can export to XML, so it's not
such a big stretch.
w>Steps to migrate from MM2 to MM3
>
>1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
>include testing to assure that things are ready to create new lists.
Right, and it should be doable even while mm2 is still functional.
>2) Translate list configurations
>
> a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this
> to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy
> would be Root-->World-->Site-->Domain-->List-->Subscriber. TOOL2 would
> populate configurations at the List level. It might also reformat selected
> parameters. In particular, various <option type="radio" > entries might be
> transformed into enumerations such as "Yes"/"No" or
> "Hidden"/"Private"/"Public" rather than numerical values. This would
> enhance readability.
>
> b) TOOL3 would populate the World level with the MM2 defaults and
> recursively promote common values up the tree, leaving only those entries
> which would need to override their parent to derive the current
> value. Values which match the parent would be flagged. (The inheritance
> flag should be tri-state. "Differs from parent", "Same as parent",
> "Inherited from parent")
>
> c) At this point, the user might edit some of the configurations and rerun
> TOOL3 adjusting the inheritance flag as appropriate.
I think the trickiest part will be what to do about subscriber information.
In mm2, this is always list-centric, but in mm3, you need to collate and
globalize all the membership information into the user database. You can
probably do the same kind of up-promotion there, but it would be from
member->address->user. IOW, if you see an address subscribed to a mailing
list with the same values across all those lists, put the preferences in the
user. What happens if you see <email address hidden> subscribed to three different
lists with three different passwords? That's a tough one because there's no
way to express that in mm3 (nor probably should there be).
So I think you will occasionally have to just resolve some conflicts by
flipping a coin. In the case of passwords, perhaps you'd always make the user
do a password reset.
> d) Now, we begin translation to MM3 configuration options. For each MM3
> option, TOOL4 computes the equivalent value from the MM2 values. Each
> computed value also gets the corresponding inheritance flag. Values that
> cannot be computed from the available information get the "Inherited from
> parent" flag. MM2 values used in computations are marked as "translated".
The user herself could probably write a script for this pipeline you're
proposing, that would allow her to do bulk transformations of configuration
variable.
> e) After a chance to edit the MM3 configurations, TOOL5 would recompute
> inheritance flags, report any MM2 values that have not been translated and
> produce a copy of the configuration file simplified by removing all
> inherited entries.
>
> f) After a final inspection TOOL7 would actually import the configurations,
> committing entries to the MM3 database.
>
>3) For the migration of rosters, we should be able to do it one subscription
>at a time through a pipeline that permits pre- and post- hooks. A --dry-run
>option would be appropriate.
Almost definitely. The --dry-run step which would produce an output of those
conflicts, and impossible situations. The user would then have a chance to
re-edit the intermediate file so that the values can be better mapped to mm3.
It's probably worth doing for both the rosters and list configurations.
> a) We can assume that each email address is a distinct person. The
> subscribers can utilize the UI to merge email addresses into a common
> persona.
We'll probably need a "claim and merge" operation in the system. And a way to
purge unclaimed addresses after a while.
> b) We can also assume that each subscription overrides its parent in the
> Persona-->EMailAddress-->Subscription hierarchy. The individual users can
> use the UI to consolidate their selections.
A good challenge for the Postoriusians :)
> All of the tools should be written in Python, hopefully in a dialect that is
> common to all of the versions supported by MM3.
+1. Today that would be 2.6 and 2.7.
> TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes
> as they come in. Similarly, in step 3, we can use TOOL1 and discard the list
> configuration information.
> TOOL2 can reformat the XML as HTML, thus making the input data into a
> viewable report. The inheritance flag would become a class attribute on the
> option. Would it make sense to go a step further and generate html forms and
> run a trivial http server on localhost? It might be easier to do this in
> django, but I think that requiring that level of installation is probably
> too much for the current situation.
It wouldn't be hard to do in standard Python. OTOH, I'm not sure we want to
maintain a stack of html templates, forms, and form processing in the
conversion tool.
It's sounding like this suite of conversion tools may need to be a separate
sub-project.
On Apr 09, 2012, at 03:35 PM, Richard Wackerbarth wrote:
>On Apr 8, 2012, at 12:43 PM, Launchpad Bug Tracker wrote: /bugs.launchpad .net/bugs/ 965532
>
>> Barry Warsaw (barry) has assigned this bug to you for GNU Mailman:
>
>> Need a script to upgrade from MM2 to MM3
>> https:/
>
>Here are some thoughts on a possible migration technique.
>I would request discussion and suggestions.
>
>In particular, what about the idea of converting the configuration file to
>HTML as an intermediate file format? Selectable css could easily render it
>as a viewable report. It could still be edited by hand without too much
>difficulty.
It's an interesting idea. As you observed, mm2 can export to XML, so it's not
such a big stretch.
w>Steps to migrate from MM2 to MM3
>
>1) Manually install MM3. Hook it up to the MTA, UI, and Archiver. This should
>include testing to assure that things are ready to create new lists.
Right, and it should be doable even while mm2 is still functional.
>2) Translate list configurations World-- >Site-- >Domain- ->List- ->Subscriber. TOOL2 would /"Private" /"Public" rather than numerical values. This would
>
> a) Use TOOL1 to extract the set of list configurations from MM2. Pipe this
> to TOOL2 which generates a tree of MM2 configurations. That tree hierarchy
> would be Root-->
> populate configurations at the List level. It might also reformat selected
> parameters. In particular, various <option type="radio" > entries might be
> transformed into enumerations such as "Yes"/"No" or
> "Hidden"
> enhance readability.
>
> b) TOOL3 would populate the World level with the MM2 defaults and
> recursively promote common values up the tree, leaving only those entries
> which would need to override their parent to derive the current
> value. Values which match the parent would be flagged. (The inheritance
> flag should be tri-state. "Differs from parent", "Same as parent",
> "Inherited from parent")
>
> c) At this point, the user might edit some of the configurations and rerun
> TOOL3 adjusting the inheritance flag as appropriate.
I think the trickiest part will be what to do about subscriber information. >address- >user. IOW, if you see an address subscribed to a mailing
In mm2, this is always list-centric, but in mm3, you need to collate and
globalize all the membership information into the user database. You can
probably do the same kind of up-promotion there, but it would be from
member-
list with the same values across all those lists, put the preferences in the
user. What happens if you see <email address hidden> subscribed to three different
lists with three different passwords? That's a tough one because there's no
way to express that in mm3 (nor probably should there be).
So I think you will occasionally have to just resolve some conflicts by
flipping a coin. In the case of passwords, perhaps you'd always make the user
do a password reset.
> d) Now, we begin translation to MM3 configuration options. For each MM3
> option, TOOL4 computes the equivalent value from the MM2 values. Each
> computed value also gets the corresponding inheritance flag. Values that
> cannot be computed from the available information get the "Inherited from
> parent" flag. MM2 values used in computations are marked as "translated".
The user herself could probably write a script for this pipeline you're
proposing, that would allow her to do bulk transformations of configuration
variable.
> e) After a chance to edit the MM3 configurations, TOOL5 would recompute
> inheritance flags, report any MM2 values that have not been translated and
> produce a copy of the configuration file simplified by removing all
> inherited entries.
>
> f) After a final inspection TOOL7 would actually import the configurations,
> committing entries to the MM3 database.
>
>3) For the migration of rosters, we should be able to do it one subscription
>at a time through a pipeline that permits pre- and post- hooks. A --dry-run
>option would be appropriate.
Almost definitely. The --dry-run step which would produce an output of those
conflicts, and impossible situations. The user would then have a chance to
re-edit the intermediate file so that the values can be better mapped to mm3.
It's probably worth doing for both the rosters and list configurations.
> a) We can assume that each email address is a distinct person. The
> subscribers can utilize the UI to merge email addresses into a common
> persona.
We'll probably need a "claim and merge" operation in the system. And a way to
purge unclaimed addresses after a while.
> b) We can also assume that each subscription overrides its parent in the ->EMailAddress- ->Subscription hierarchy. The individual users can
> Persona-
> use the UI to consolidate their selections.
A good challenge for the Postoriusians :)
> All of the tools should be written in Python, hopefully in a dialect that is
> common to all of the versions supported by MM3.
+1. Today that would be 2.6 and 2.7.
> TOOL1 already exists (`bin/export.py`). TOOL2 can discard the roster nodes
> as they come in. Similarly, in step 3, we can use TOOL1 and discard the list
> configuration information.
> TOOL2 can reformat the XML as HTML, thus making the input data into a
> viewable report. The inheritance flag would become a class attribute on the
> option. Would it make sense to go a step further and generate html forms and
> run a trivial http server on localhost? It might be easier to do this in
> django, but I think that requiring that level of installation is probably
> too much for the current situation.
It wouldn't be hard to do in standard Python. OTOH, I'm not sure we want to
maintain a stack of html templates, forms, and form processing in the
conversion tool.
It's sounding like this suite of conversion tools may need to be a separate
sub-project.
-Barry