diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/INSTALL.mhonarc-mm mailman-2.1.4-mhonarc/INSTALL.mhonarc-mm --- mailman-2.1.4-mod/INSTALL.mhonarc-mm Thu Jan 1 01:00:00 1970 +++ mailman-2.1.4-mhonarc/INSTALL.mhonarc-mm Thu Jan 1 17:08:51 2004 @@ -0,0 +1,290 @@ +Mailman patch #820723: Mailman/pipermail/MHonArc integration patch +------------------------------------------------------------------ + +Objectives and Description +-------------------------- + +This patch tightly integrates the MHonArc mail-to-HTML convertor (Earl Hood's +description, not mine) with Mailman and its internal pipermail archiving code. +The purpose of the patch is to produce a fusion of (hopefully) the best feature +of pipermail and MHonArc for handling Mailman mailing list archives. + +Although pipermail has a number of weaknesses it has some good features, +including structuring mail archives by per-list configurable periods and +supporting private and public archive access. + +MHonArc is represented by its proponents as being superior to pipermail in +handling attachments and multipart MIME messages, and being able to +build/rebuild archives from large UNIX mboxes. + +In contrast, pipermail's "database" creaks and groans with large mboxes and +lists with high levels of traffic, and it collapses into a heap when it is asked +to rebuild from very large mboxes. The latest pipermail handling of attachments +and MIME is OK'ish but appears to be weaker than MHonArc. + +With MHonArc, I could not quickly find an obvious way to get the HTML archives +it is generating into optional Yearly/Monthly/Weekly/Daily periods as pipermail +does. It is clearly possible (for instance, see +http://www.mail-archive.com/mailman-users%40python.org/) but how ... + +List archive privacy can be important for some user communities and the +Mailman/pipermail solution works well. + +I also wanted searchable archives and saw no reason not to continue using my +Mailman/HTdig integration patch for per-list archive searching. + +Finally, I thought it would be neat to make choosing whether MHonArc or +pipermail should generate a list's archive pages a per-list configuration option +and have Mailman's archive builder $prefix/bin/arch work whichever choice was +made for any list. + +The upshot was this patch, which is a hack, but one which works. The code is not +pretty but I defy even the best cosmetic surgeon to produce movie star looks +when he is grafting a wart onto a boil. The code works and can fill my needs +until Mailman version 3 comes along incorporating a wizzy new archiver and +archive search capability ... + +Alternative ways of using MHonArc and HTdig in conjunction with Mailman exist +and doubtless some would argue are superior. There is no compulsion to use this +patch; it is just another option you can choose from. + +Implementation Details +---------------------- + +The implementation operates with pipermail in charge of archiving. That is, the +pipermail code generates the top level archive TOC pages for each list, +organises each list's archive directory structure and sorts out the archving +period stuff. But when it comes to generating the message and index pages of the +HTML archives the use of pipermail or MHonArc depends on the option set for any +list via the Archiving Options page of the admin web GUI. + +For lists set for MHonArc archiving, pipermail uses an instance of MHonArc +instead of its own code to generate the HTML message and index pages. For such +lists, pipermail maintains only a vestigial database so that the problems of +large pipermail databases is avoided. + +The organization of the archives on disk is pretty much the same for both +pipermail'ed and MHonArc'ed lists. The top level list TOC is the same as for +normal Mailman as is the per-period sub-directory structure and per-period text +(mbox) archive. The naming of message files is different for MHonArc as is the +storage of extracted attachment files. For extracted attachments pipermail uses +a separate directory structure while MHonArc puts them in the same directory as +the messages. Actually, this is a win for MHonArc because the URL's on message +pages linking to extracted attachments that it generates are relative URL's. In +contrast, pipermail generates absolute URLs. This means that, for pipermail +generated message pages, if a list archive is changed from private to public or +vice versa, the links on messages pages are wrong and can only be corrected by +rebuilding the entire list archive. With MHonArc'ed archives list privacy +changes are a non-event, with Mailman's creation and deletion of the symlink to +the list archives in $prefix/archives/public/ doing all that is needed. + +$prefix/bin/arch works as normal, regardless of whether pipermail or MHonArc is +generating message and index pages. Indeed, having changed the archiver option +for a list, $prefix/bin/archÊ--wipe must be run to have the change take effect; +this is because of the incompatibility between the per-list database, message +file naming and attachment storage schemes used by the two options. + +Because I like the date/thread/subject/author indexes produced by pipermail, +MHonArc (as used by this patch) does the same thing. The layout of message and +index pages generated by MHonArc is controlled using three MHonArc resource +configuration files (MRCFs), which must reside in Mailman's template directory +structure. The MRCFs are selected for a list in the same way that any other +template file is associated with that list, that is, the template hierarchy is +searched. Before pipermail invokes MHonArc to handle a messages or a group of +messages, it selects the which MRCFs apply and passes these as parameters to +MHonArc. The default MRCFs reside in $prefix/templates/en as mhonarc.mrc, +author.mrc and subject.mrc. The look and feel the default MRCFs produce is +similar to that of the archive and index pages for regular pipermail'ed archive +pages. The only thing that is different with the operation of the templates for +MHonArc is that there is no variable substitution performed by Mailman; instead +some useful values (such as list name and archive name) are passed to MHonArc +using environment variables which can be referred to in the MRCFs to +characterize the pages being generated. + +Whether the MRCFs are passed to MHonArc on the command line when it is invoked +depends on: + +1. Whether the message(s) being passed to MHonArc is the first one for a new + period; if it is then they are, regardless of any other factors. + +2. The value of the MHONARC_SAVE_RESOURCES MM config variable (Default value is + True). This config variable tells pipermail to pass MHonArc either the + -saveresources or the -nosaveresources command line option. If + MHONARC_SAVE_RESOURCES is True then the MRCFs are not passed as command line + options (exluding the situation in [1]), because MHonArc will already have + the template information from the MRCFs in its database. + +The downside to MHONARC_SAVE_RESOURCES being True is that new messages for an +existing archive period will continue to use the existing templates despite +changes of/to the mhonarc.mrc, author.mrc or subject.mrc applicable to the list +concerned, even though mailmancntrl -restart may have been run. To get the +revised templates into use, run bin/arch --wipe for the list. + +Following installation of this patch, the availability of the features it +provides is dependent on the installation of MHonArc (see Necessary Precursors +below) and the setting the value of MHONARC_ARCHIVER_PATH to a non-empty string. + +MHonArc has be installed on the Mailman server. I found installing it into +$prefix/mhonarc worked for me. There is a new Mailman configuration variable, +added to $prefix/Defaults.py by the patch, which tells Mailman where MHonArc is +installed. This was the sum total of setup I had to do for the MHonArc +installation. + +MHONARC_ARCHIVER_PATH = os.path.join(PREFIX, 'mhonarc', 'bin', 'mhonarc') + +The patch adds an option, with radio buttons to choose pipermail or MHonArc as +archiver for a list, to the Archiving Options page of the web admin GUI. This +option is only displayed after a non blank value has been assigned to +MHONARC_ARCHIVER_PATH. The default value for for which archiver to use is set +by a new Mailman configuration variable added to $prefix/Defaults.py by the +patch: + +# Which archiver to use by default to generate archive pages: +# 0 - pipermail +# 1 - mhonarc +DEFAULT_WHICH_ARCHIVER = 0 + +When a new list is created and when/if archiving is enabled for it, it will use +the archiver specified by DEFAULT_WHICH_ARCHIVER at the time of the list's +creation. Lists with existing archives that pre-date adoption of this patch will +continue to be pipermail archives unless their choice of archiver is changed via +the web admin GUI. + +As noted above, when the archiver nominated for a given list is changed the +change will not take effect until $prefix/bin/arch --wipe is run for the list. +This is not done automatically at the time the option is changed using the web +admin GUI for a good reason: if the list has a large mbox then the amount of +processing involved in rebuilding the archive is inappropriate for a process +being run "in-line" as part of a web server transaction. + +With pipermail in charge, MHonArc only gets to see the archive a period at a +time. In practice, each period of a list's archive is what MHonArc sees and it +maintains a database and index pages for each of them quite separately. This +means that thread and date links generated by MHonArc terminate at the boundary +of the archiving period. The only index linking all of the period archives for a +list is the tope level TOC page for a list's archives which is generated by +pipermail in the normal way. Thus far, this characteristic has not been a +problem for me and I've not given much thought to changing it. + +When $prefix/bin/arch is run on a list configured to use MHonArc, and because of +the period archive approach, pipermail passes messages for the same period to +MHonArc for processing in temporary mbox files. The way this is done means that +the memory demand made by Mailman/pipermail in handling a large mbox is no +bigger than the biggest message in the input file being processed. By contrast, +when archiving a single message, for the ArchRunner say, it is passed via stdin +to MHonArc. + +My Mailman-HTdig integration patch works as normal with archives generated by +both pipermail and MHonArc. Just install and use it in the same way as usual. + +Mailman's pipermail features and configuration for obscuring mail address in the +HTML archives to thwart email address harvesters are not automatically applied +to MHonArc generated pages. Features provided by MHonArc can be used by making +modifications to the MHonArc resource configuration files that control archive +index and message page generation. You should take copies of the default MRCFs +installed in $prefix/templates/en, modify them and add them to a site, virtual +host or list specific sub-directory of the template hierarchy. + +The default MRCFs have embedded in them default indexing control directives for +HTdig, in anticipation of HTdig being used for archive search. With pipermail +generated HTML pages the effect of changing the value of Mailman config +variables ARCHIVE_INDEXING_ENABLE and ARCHIVE_INDEXING_DISABLE in +$prefix/Mailman/mm_cfg.py is dynamically incorporated into the pages. With +MHonArc generated pages this must be achieved by copying and modifying the +default MRCFs installed in $prefix/templates/en and adding them to a site, +virtual host or list specific sub-directory of the template hierarchy. + +Things Still To Do (Maybe) +-------------------------- + +1. Integrated support for different languages +2. Obscuring mail address in the HTML archives + + +Applicability +------------- + +This patch is applicable to Mailman MM 2.1.3 + +Necessary Precursors +-------------------- + +The following patch must be applied to Mailman before applying this patch: + +1. Mailman Patch #760567: both this patch and #760567 update the version number +of the Mailman list database in order to add extra attributes to it. +Note that this means that this MHonArc integration patch and #760567 will have +to be updated the next time the standard DATA_FILE_VERSION value is updated in +$prefix/Mailman/Version.py +2. MHonArc has to be installed on the Mailman server machine. My development +and testing has been done with MHonArc 2.6.8. When I installed MHonArc 2.68, I +just followed the instructions and the initial terminal dialogue looked like +this (where $prefix is the value of the Mailman --with-prefix ./configure +option): + + +> install.me +Checking dependencies: + Fcntl ......................... ok + File::Basename ................ ok + Getopt::Long .................. ok + Symbol ........................ ok + Time::Local ................... ok +Pathname of perl executable: ("/usr/bin/perl") +Directory to install executables: ("/usr/bin") $prefix/mhonarc/bin +Directory to install library files: ("/usr/lib/perl5/site_perl/5.6.1") +$prefix/mhonarc/lib/perl5/site_perl/5.6.1 +Directory to install documentation: ("/usr/doc/MHonArc") +$prefix/mhonarc/doc/MHonArc +Directory to install manpages: ("/usr/share/man") $prefix/mhonarc/share/man +You have specified the following: + Perl path: /usr/bin/perl + Bin directory: $prefix/mhonarc/bin + Lib directory: $prefix/mhonarc/lib/perl5/site_perl/5.6.1 + Doc directory: $prefix/mhonarc/doc/MHonArc + Man directory: $prefix/mhonarc/share/man +Is this correct? ['y'] +... + + +Changes Made +------------ + +See the Description and Implementation Details above. + +Applying the patch +------------------ + +Apply the patch from within the Mailman build directory using the command: + + + patch -p1 < path-to-patch-file + +History +------- + +Patch Version Changes +------------- ------- + +2.1.3-0.6 1. Added MHONARC_SAVE_RESOURCES config variabletto Defaults.py. + 2. Associated changes in Mailman/Archiver/pipermail.py + +2.1.3-0.5 1. Fixed minor HTML syntax error in mhonarc.mrc and + author.mrc that affected date and author index pages. + +2.1.3-0.4 1. Changed default value of MHONARC_ARCHIVER_PATH in + Defaults.py to the empty string '' + 2. Changed behaviour so that if MHONARC_ARCHIVER_PATH is the + empty string, the ability to change which archiver to use on + the Archiving Options pages of the web admin GUI of lists is + not displayed. In effect until the configuration variable is + defined the installllation of this patch is not seen. + + +2.1.3-0.3 1. Change to tolerate MHonArc prematurely (and validly) closing + the pipe through which it is receiving a message from + pipermail. + + +2.1.3-0.2 First 'official release' +2.1.3-0.1 Original 'unofficial release' diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/Mailman/Archiver/Archiver.py mailman-2.1.4-mhonarc/Mailman/Archiver/Archiver.py --- mailman-2.1.4-mod/Mailman/Archiver/Archiver.py Mon Dec 1 01:07:26 2003 +++ mailman-2.1.4-mhonarc/Mailman/Archiver/Archiver.py Thu Jan 1 17:08:51 2004 @@ -71,6 +71,8 @@ self.archive_private = mm_cfg.DEFAULT_ARCHIVE_PRIVATE self.archive_volume_frequency = \ mm_cfg.DEFAULT_ARCHIVE_VOLUME_FREQUENCY + self.which_archiver = mm_cfg.DEFAULT_WHICH_ARCHIVER + self.archiver_when_wiped = 0 # The archive file structure by default is: # # archives/ diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/Mailman/Archiver/HyperArch.py mailman-2.1.4-mhonarc/Mailman/Archiver/HyperArch.py --- mailman-2.1.4-mod/Mailman/Archiver/HyperArch.py Fri Dec 26 20:41:30 2003 +++ mailman-2.1.4-mhonarc/Mailman/Archiver/HyperArch.py Thu Jan 1 17:08:51 2004 @@ -224,7 +224,26 @@ # characters. return Utils.uncanonstr(text, lang) - +def what_madetext(templatefile, lang=None, mlist=None): + if mlist: + listname = mlist._internal_name + else: + listname = '' + if lang is None: + if mlist is None: + lang = mm_cfg.DEFAULT_SERVER_LANGUAGE + else: + lang = mlist.preferred_language + cachekey = (templatefile, lang, listname) + filepath = _templatefilepathcache.get(cachekey) + if filepath: + template = _templatecache.get(filepath) + if filepath is None or template is None: + # Use the basic maketext, with defaults to get the raw template + (template, filepath) = Utils.findtext(templatefile, lang=lang, raw=1, mlist=mlist) + _templatefilepathcache[cachekey] = filepath + _templatecache[filepath] = template + return filepath # Note: I'm overriding most, if not all of the pipermail Article class # here -ddm @@ -631,6 +650,13 @@ lang=self.maillist.preferred_language, mlist=self.maillist) + def which_MHonArc_template(self, filename='mhonarc.mrc'): + return what_madetext(filename, mlist=self.maillist) + + + def mhonarc_html_quote(self, astring, lang): + return html_quote(astring, lang) + def html_foot(self): # avoid i18n side-effects mlist = self.maillist @@ -1097,6 +1123,45 @@ newz = gzip.open(gzipfile, 'w') finally: # XXX why is this a finally? + os.umask(ou) + if archz: + newz.write(archz.read()) + archz.close() + os.unlink(oldgzip) + # XXX do we really need all this in a try/except? + try: + newz.write(archt.read()) + newz.close() + archt.close() + except IOError: + pass + os.unlink(txtfile) + + def mhonarc_update_archive(self, archive): + # This is the same as update_archive but without the super call + # only do this if the gzip module was imported globally, and + # gzip'ing was enabled via mm_cfg.GZIP_ARCHIVE_TXT_FILES. See + # above. + if gzip: + archz = None + archt = None + txtfile = os.path.join(self.basedir, '%s.txt' % archive) + gzipfile = os.path.join(self.basedir, '%s.txt.gz' % archive) + oldgzip = os.path.join(self.basedir, '%s.old.txt.gz' % archive) + try: + # open the plain text file + archt = open(txtfile) + except IOError: + return + try: + os.rename(gzipfile, oldgzip) + archz = gzip.open(oldgzip) + except (IOError, RuntimeError, os.error): + pass + try: + ou = os.umask(002) + newz = gzip.open(gzipfile, 'w') + finally: os.umask(ou) if archz: newz.write(archz.read()) diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/Mailman/Archiver/pipermail.py mailman-2.1.4-mhonarc/Mailman/Archiver/pipermail.py --- mailman-2.1.4-mod/Mailman/Archiver/pipermail.py Wed Dec 24 17:09:30 2003 +++ mailman-2.1.4-mhonarc/Mailman/Archiver/pipermail.py Thu Jan 1 17:08:51 2004 @@ -7,6 +7,7 @@ import re import sys import time +import errno from email.Utils import parseaddr, parsedate_tz, mktime_tz import cPickle as pickle from cStringIO import StringIO @@ -16,8 +17,10 @@ VERSION = __version__ CACHESIZE = 100 # Number of slots in the cache +from Mailman import mm_cfg from Mailman import Errors from Mailman.Mailbox import ArchiverMailbox +from Mailman.Mailbox import mhonarcArchiverMailbox from Mailman.Logging.Syslog import syslog from Mailman.i18n import _ @@ -246,6 +249,14 @@ def finished_update_article(self): pass +class mhonarcArticle(Article): + + def __init__(self, message=None, sequence=0, keepHeaders=[], + lang=None, mlist=None): + self._set_date(message) + self._lang = lang + self._mlist = mlist + # Pipermail formatter class class T: @@ -535,6 +546,34 @@ return Article(msg, sequence) def processUnixMailbox(self, input, start=None, end=None): + # kludge to process differently if list is being archived by MHonArc + if self.maillist.which_archiver == 1: + if not mm_cfg.MHONARC_ARCHIVER_PATH: + reason = 'MHONARC_ARCHIVER_PATH not set but list %s set to use MHonArc' % \ + self.maillist.real_name + syslog('error', reason) + if self.maillist.archiver_when_wiped != 0: + raise MMArchiverSelectionError, reason + else: + if self.maillist.archiver_when_wiped != 1: + reason = 'list %s set to use MHonArc but archive not wiped since archiver changed' % \ + self.maillist.real_name + syslog('error', reason) + if self.maillist.archiver_when_wiped != 0: + raise MMArchiverSelectionError, reason + else: + self.mhonarcProcessUnixMailbox(input, start, end) + return + else: + if self.maillist.archiver_when_wiped != self.maillist.which_archiver: + reason = 'list %s set to use pipermail but archive not wiped since archiver changed' % \ + self.maillist.real_name + syslog('error', reason) + if self.maillist.archiver_when_wiped == 1 and mm_cfg.MHONARC_ARCHIVER_PATH: + self.mhonarcProcessUnixMailbox(input, start, end) + return + else: + raise MMArchiverSelectionError, reason mbox = ArchiverMailbox(input, self.maillist) if start is None: start = 0 @@ -570,6 +609,119 @@ if end is not None and counter >= end: break counter += 1 + + def _makeMhonarcArticle(self, message=None): + return mhonarcArticle(message, 0, lang=self.maillist.preferred_language, + mlist=self.maillist) + + def mhonarcProcessUnixMailbox(self, input, start=None, end=None): + mbox = mhonarcArchiverMailbox(self, input, self.maillist, start, end) + # We are doing our own dirty archive handling + dirty_archives = [] + # All message for the same list so find the MHonArc configs that + # are applicable + main_rc = self.which_MHonArc_template('mhonarc.mrc') + author_rc = self.which_MHonArc_template('author.mrc') + subject_rc = self.which_MHonArc_template('subject.mrc') + configs = '-rcfile %s -otherindex %s:%s' % (main_rc, author_rc, + subject_rc) + cmdbits = {'config': configs} + if mm_cfg.MHONARC_SAVE_RESOURCES: + saveres = '-saveresources' + else: + saveres = '-nosaveresources' + # Setup command to run MHonArc but make archive directory, environment + # and input pluggable +# mhonarc_cmd = '%%(envstring)s %s -outdir %%(archivedir)s %s %%(config)s -add %%(mboxpath)s' % \ +# (mm_cfg.MHONARC_ARCHIVER_PATH, saveres) + mhonarc_cmd = '%%(envstring)s %s -outdir %%(archivedir)s %s %%(config)s -add %%(mboxpath)s > /dev/null' % \ + (mm_cfg.MHONARC_ARCHIVER_PATH, saveres) + # Initialise some environment variables which we use to pass parameters + # into the MHonArc archiver pages + evars = {'listname': self.mhonarc_html_quote(self.maillist.real_name, + self.maillist.preferred_language), + 'listinfo': self.maillist.GetScriptURL('listinfo', absolute=1) + } + while 1: + result = mbox.next() + if result is None: + break + rtype, this_archive, msg_or_path = result + mboxpath = '' + if rtype == 2: + # Temprorary mbox returned so we will want to add this to the command + mboxpath = msg_or_path + else: + # Message object returned + msg = msg_or_path + cmdbits['mboxpath'] = mboxpath + # Probably only one archive but we interate in case there are more + for arch in this_archive: + archivedir = os.path.join(self.basedir, arch) + if arch not in self.archives: + # Create the archive directory if it doesn't exist + self.new_archive(arch, archivedir) + # and make sure we provide MHonARc configs + cmdbits['config'] = configs + elif mm_cfg.MHONARC_SAVE_RESOURCES: + # MHonArc configs should be in the MH database + cmdbits['config'] = '' + # Add archive description to what we pass in via environment variables + evars['archive'] = self.volNameToDesc(arch) + # Now build the command to run MHonArc + envstuff = [] + for k, v in evars.items(): + envstuff.append("MHMM_%s='%s'" % (k,v)) + cmdbits['archivedir'] = archivedir + cmdbits['envstring'] = ' '.join(envstuff) + cmd = mhonarc_cmd % cmdbits +# syslog('error', 'MHonArc archiving:1 starting cmd: %s\n', cmd) + # Run MHonArc + mhonarch_process = os.popen(cmd, 'w') + if rtype == 1: + # Single message so send this through the pipe + try: + mhonarch_process.write(msg.as_string() + "\n") + except IOError, e: + # MHonARc seems to close the pipe before receiving + # all the output if the message is already in the + # its archives + if e.errno != errno.EPIPE: + raise + msgid = msg.get('message-id', 'n/a') +# syslog('error', 'MHonArc broken pipe accepted, msgid=%s\n', msgid) +# syslog('error', 'MHonArc archiving:2 ending cmd\n') + # Close the pipe and check the return code + status = mhonarch_process.close() + if status: + syslog('error', 'MHonArc archiver non-zero exit status: %d\n', + (status & 0xff00) >> 8) + # Write the message to the period's text archive. + txt_archive_path = archivedir + '.txt' + try: + omask = os.umask(002) + f = open(txt_archive_path, 'a+') + finally: + os.umask(omask) + if rtype == 1: + # Single message + f.write(msg.as_string(True) + '\n') + else: + # Multiple messages + f.write(open(mboxpath).read()) + f.close() + # Keep track of archives we have modified + if arch not in dirty_archives: + dirty_archives.append(arch) + if rtype == 2: + # Remove any temporary mbox file + os.unlink(mboxpath) + if dirty_archives: + # Update the archives we marked as modified + for arch in dirty_archives: + self.mhonarc_update_archive(arch) + # Avoid triggering standard pipermail dirty archive actions + self._dirty_archives = [] def new_archive(self, archive, archivedir): self.archives.append(archive) diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/Mailman/Defaults.py.in mailman-2.1.4-mhonarc/Mailman/Defaults.py.in --- mailman-2.1.4-mod/Mailman/Defaults.py.in Thu Jan 1 13:57:14 2004 +++ mailman-2.1.4-mhonarc/Mailman/Defaults.py.in Thu Jan 1 17:08:51 2004 @@ -228,6 +228,17 @@ DEFAULT_ARCHIVE_VOLUME_FREQUENCY = 1 DEFAULT_DIGEST_VOLUME_FREQUENCY = 1 +# Added by the MHonArc integration patch +# Which archiver to use by default to generate archive pages: +# 0 - pipermail +# 1 - mhonarc +DEFAULT_WHICH_ARCHIVER = 0 +# By default MHonArc will save resource info in its database(s) +# But thiis means that changes to the page templates will not be +# picked up unless a bin/arch --wipe is done. Set MHONARC_SAVE_RESOURCES +# to False and the templates will be passed on every MHonArc invocation. +MHONARC_SAVE_RESOURCES = 1 + # These variables control the use of an external archiver. Normally if # archiving is turned on (see ARCHIVE_TO_MBOX above and the list's archive* # attributes) the internal Pipermail archiver is used. This is the default if @@ -1252,6 +1263,12 @@ PIDFILE = os.path.join(DATA_DIR, 'master-qrunner.pid') SITE_PW_FILE = os.path.join(DATA_DIR, 'adm.pw') LISTCREATOR_PW_FILE = os.path.join(DATA_DIR, 'creator.pw') + +# Added by the MHonArc integration patch +# Add something like this to your mm_cfg.py once you have installed MHonArc +# MHONARC_ARCHIVER_PATH = os.path.join(PREFIX, 'mhonarc', 'bin', 'mhonarc') +# The default, empty defn suppresses use of MHonArc +MHONARC_ARCHIVER_PATH = '' # Import a bunch of version numbers from Version import * diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/Mailman/Errors.py mailman-2.1.4-mhonarc/Mailman/Errors.py --- mailman-2.1.4-mod/Mailman/Errors.py Mon Dec 1 00:35:32 2003 +++ mailman-2.1.4-mhonarc/Mailman/Errors.py Thu Jan 1 17:08:51 2004 @@ -25,6 +25,7 @@ class MMListNotReadyError(MMListError): pass class MMListAlreadyExistsError(MMListError): pass class BadListNameError(MMListError): pass +class MMArchiverSelectionError(MMListError): pass # Membership exceptions class MMMemberError(Exception): pass diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/Mailman/Gui/Archive.py mailman-2.1.4-mhonarc/Mailman/Gui/Archive.py --- mailman-2.1.4-mod/Mailman/Gui/Archive.py Wed Feb 27 00:01:27 2002 +++ mailman-2.1.4-mhonarc/Mailman/Gui/Archive.py Thu Jan 1 17:08:51 2004 @@ -27,18 +27,26 @@ def GetConfigInfo(self, mlist, category, subcat=None): if category <> 'archive': return None - return [ - _("List traffic archival policies."), + opts = [_("List traffic archival policies."), - ('archive', mm_cfg.Toggle, (_('No'), _('Yes')), 0, + ('archive', mm_cfg.Toggle, (_('No'), _('Yes')), 0, _('Archive messages?')), - ('archive_private', mm_cfg.Radio, (_('public'), _('private')), 0, + ('archive_private', mm_cfg.Radio, (_('public'), _('private')), 0, _('Is archive file source for public or private archival?')), - ('archive_volume_frequency', mm_cfg.Radio, + ('archive_volume_frequency', mm_cfg.Radio, (_('Yearly'), _('Monthly'), _('Quarterly'), _('Weekly'), _('Daily')), 0, _('How often should a new archive volume be started?')), - ] + ] + if mm_cfg.MHONARC_ARCHIVER_PATH: + opts.append(\ + ('which_archiver', mm_cfg.Radio, (_('pipermail'), _('mhonarc')), 0, + _('''Which archive page generator is to be used for this list? + You must run bin/arch --wipe before any change to this will take + effect''')) + ) + return opts + diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/Mailman/Mailbox.py mailman-2.1.4-mhonarc/Mailman/Mailbox.py --- mailman-2.1.4-mod/Mailman/Mailbox.py Sat Apr 19 05:57:14 2003 +++ mailman-2.1.4-mhonarc/Mailman/Mailbox.py Thu Jan 1 17:08:51 2004 @@ -109,3 +109,128 @@ return self._scrubber(self._mlist, msg) else: return msg + +import tempfile +from Mailman.i18n import _ +from Mailman import Errors +from Mailman.Logging.Syslog import syslog + +class mhonarcArchiverMailbox(ArchiverMailbox): + # This class is a hack to support integrated use of MHonArc for generating + # the archive content. + # Primary purpose is to return either a single message object or a path + # to a file containing a sequence of messages if thos messages should + # all go into the same archive. + # Value returned is a triple of: + # type of info returned; 1: single message, 2: multiple messages + # a list of archives the message(s) should go in + # a single message object or the path to a temporary mbox file containing + # multiple messages + + def __init__(self, archiver, fp, mlist, start, end): + ArchiverMailbox.__init__(self, fp, mlist) + self._archiver = archiver + self._input = fp + self._mlist = mlist + if start is None: + start = 0 + self._start = start + self._end = end + self._heldover = [] + self._counter = 0 + self._finished = 0 + # Skip over initial, unwanted messages + while self._counter < start: + try: + msg = ArchiverMailbox.next(self) + except Errors.DiscardMessage: + continue + if msg is None: + self._finished = 1 + break + self._counter += 1 + + # Leave scrubbing to MHonArc + def scrub(self, msg): + return msg + + def next(self): + mbox_path = mbox_fp = None + result = None + try: + while 1: + if self._end is not None and self._counter >= self._end: + # Processed as many messages as required + self._finished = 1 + if self._finished: + # No more messages to be processed + if self._heldover: + # Want to return final tranche of messages + result = self._heldover + self._heldover = [] + else: + # No more messages to return + result = None + break + try: + pos = self._input.tell() + msg = ArchiverMailbox.next(self) + except Errors.DiscardMessage: + continue + except Exception: + syslog('error', 'uncaught archiver exception at filepos: %s', + pos) + if mbox_fp: + mbox_fp.close() + os.unlink(mbox_path) + self._heldover = [] + self._finished = 1 + raise + self._counter += 1 + if msg is None: + # No messages left + self._finished = 1 + continue + if msg == '': + # It was an unparseable message + continue + # We have a message + msgid = msg.get('message-id', 'n/a') + counter = self._counter + self._archiver.message(_('#%(counter)05d %(msgid)s')) + article = self._archiver._makeMhonarcArticle(msg) + # Determine which archive(s) the message should be archived to + this_archive = self._archiver.get_archives(article) + if type(this_archive) == type(''): + this_archive = [this_archive] + # Now depends on if we are holding message(s) already + if not self._heldover: + # Not holding message(s) so hold this one + self._heldover = [1, this_archive, msg] + continue + else: + # We are holding a message + if this_archive != self._heldover[1]: + # New messge is not for the same archive(s) + result = self._heldover + # Hold over the new message + self._heldover = [1, this_archive, msg] + # Exit to the return what was held over + break + else: + # New message for same archive(s) as held over + if self._heldover[0] == 1: + # Just holding a single message so insert it into a + # temporary mbox file + mbox_path = tempfile.mktemp('.mbox') + mbox_fp = open(mbox_path, 'w') + mbox_fp.write(self._heldover[2].as_string(True) + '\n') + self._heldover[2] = mbox_path + self._heldover[0] = 2 + # Add new message to the held over temporary mbox file + mbox_fp.write(msg.as_string(True) + '\n') + finally: + # Always close the temporary mbox file on the way out + if mbox_fp: + mbox_fp.close() + return result diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/Mailman/Version.py mailman-2.1.4-mhonarc/Mailman/Version.py --- mailman-2.1.4-mod/Mailman/Version.py Thu Jan 1 14:04:33 2004 +++ mailman-2.1.4-mhonarc/Mailman/Version.py Thu Jan 1 17:10:38 2004 @@ -37,7 +37,11 @@ # config.pck schema version number # 90.1 to take account of extra moderate_inc_body list attribute -DATA_FILE_VERSION = 90.1 +#DATA_FILE_VERSION = 90.1 +# 90.2 to take account of extra which_archiver and +# archiver_when_wiped list attributes +# must be applied AFTER the modinc-2.1.4-0.1.patch +DATA_FILE_VERSION = 90.2 # qfile/*.db schema version number QFILE_SCHEMA_VERSION = 3 diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/bin/arch mailman-2.1.4-mhonarc/bin/arch --- mailman-2.1.4-mod/bin/arch Mon Apr 7 23:51:17 2003 +++ mailman-2.1.4-mhonarc/bin/arch Thu Jan 1 17:08:51 2004 @@ -164,6 +164,8 @@ # Maybe wipe the old archives if wipe: shutil.rmtree(mlist.archive_dir()) + mlist.archiver_when_wiped = mlist.which_archiver + mlist.Save() try: fp = open(mbox) except IOError, msg: diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/bin/newlist mailman-2.1.4-mhonarc/bin/newlist --- mailman-2.1.4-mod/bin/newlist Mon Dec 2 14:44:19 2002 +++ mailman-2.1.4-mhonarc/bin/newlist Thu Jan 1 17:08:51 2004 @@ -1,6 +1,6 @@ #! @PYTHON@ # -# Copyright (C) 1998,1999,2000,2001,2002 by the Free Software Foundation, Inc. +# Copyright (C) 1998-2004 by the Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License @@ -174,6 +174,10 @@ # And assign the preferred language mlist.preferred_language = lang + + # New lists do not default to pipermail archiver_when_wiped + mlist.archiver_when_wiped = mlist.which_archiver + mlist.Save() finally: diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/templates/Makefile.in mailman-2.1.4-mhonarc/templates/Makefile.in --- mailman-2.1.4-mod/templates/Makefile.in Sat Dec 13 16:32:44 2003 +++ mailman-2.1.4-mhonarc/templates/Makefile.in Thu Jan 1 17:08:51 2004 @@ -62,7 +62,7 @@ for d in $(LANGUAGES); \ do \ $(srcdir)/../mkinstalldirs $(DESTDIR)$(TEMPLATEDIR)/$$d; \ - for f in $(srcdir)/$$d/*.html $(srcdir)/$$d/*.txt; \ + for f in $(srcdir)/$$d/*.html $(srcdir)/$$d/*.txt $(srcdir)/$$d/*.mrc; \ do \ $(INSTALL) -m $(FILEMODE) $$f $(DESTDIR)$(TEMPLATEDIR)/$$d; \ done; \ diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/templates/en/author.mrc mailman-2.1.4-mhonarc/templates/en/author.mrc --- mailman-2.1.4-mod/templates/en/author.mrc Thu Jan 1 01:00:00 1970 +++ mailman-2.1.4-mhonarc/templates/en/author.mrc Thu Jan 1 17:08:51 2004 @@ -0,0 +1,106 @@ + + + +
+ + + + + +author.html + + +author + + + + + + + + The $ENV(MHMM_listname)$ $ENV(MHMM_archive)$ Archive by author + + + +

$ENV(MHMM_listname)$ $ENV(MHMM_archive)$ Archives by author

+
+ + + +
+
    + + +
  • $FROMNAME$, $FROMADDR$ + +
    • + + +
+ + +
+ This archive was generated by a fusion of + Pipermail 0.09 (Mailman edition) and + MHonArc 2.6.8. + + + + + + + + + diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/templates/en/mhonarc.mrc mailman-2.1.4-mhonarc/templates/en/mhonarc.mrc --- mailman-2.1.4-mod/templates/en/mhonarc.mrc Thu Jan 1 01:00:00 1970 +++ mailman-2.1.4-mhonarc/templates/en/mhonarc.mrc Thu Jan 1 17:08:51 2004 @@ -0,0 +1,238 @@ + + + + +list- + + + + + +
+ + + + +date.html + + +date + + + + + + + + + The $ENV(MHMM_listname)$ $ENV(MHMM_archive)$ Archive by date + + + +

$ENV(MHMM_listname)$ $ENV(MHMM_archive)$ Archives by date

+
+ + + +
+ +
    + + +
  • $MSGGMTDATE(CUR;%a, %d %b %Y)$ +
      + + +
    +
    • + + +
  • $SUBJECTNA$ - $FROMNAME$, $FROMADDR$
    • +     + +
+ + + +
+ This archive was generated by a fusion of + Pipermail 0.09 (Mailman edition) and + MHonArc 2.6.8. + + + + + + + + + + +thread.html + + +thread + + + + + + + + The $ENV(MHMM_listname)$ $ENV(MHMM_archive)$ Archive by thread + + + +

$ENV(MHMM_listname)$ $ENV(MHMM_archive)$ Archives by thread

+
+ + + +
+
    + + +
  • $SUBJECT$ , + $FROMNAME$ - $DATE$ +
    • +     + +
  • $SUBJECT$ , + $FROMNAME$ - $DATE$ + + +
    • + + +
  • $SUBJECT$ , + $FROMNAME$ - $DATE$ + + +
    • + + +
+ + +
+ This archive was generated by a fusion of + Pipermail 0.09 (Mailman edition) and + MHonArc 2.6.8. + + + + + + + + + + + + + $SUBJECTNA$ + + + + + + + + + +

$SUBJECTNA$

+ +
+ + +
+ + + + +
+ + + + + + + + +
  • Previous by Date: $SUBJECT(PREV)$
    • +     + +
  • Next by Date: $SUBJECT(NEXT)$
    • +     + +
  • Previous by Thread: $SUBJECT(TPREV)$
    • +     + +
  • Next by Thread: $SUBJECT(TNEXT)$
    • +     + +
    + This archive was generated by a fusion of + Pipermail 0.09 (Mailman edition) and + MHonArc 2.6.8. + + + +     diff -r -u -P --exclude=.DS_Store mailman-2.1.4-mod/templates/en/subject.mrc mailman-2.1.4-mhonarc/templates/en/subject.mrc --- mailman-2.1.4-mod/templates/en/subject.mrc Thu Jan 1 01:00:00 1970 +++ mailman-2.1.4-mhonarc/templates/en/subject.mrc Thu Jan 1 17:08:51 2004 @@ -0,0 +1,109 @@ + + + +
    + + + + + +subject.html + + +subject + + +auth + + + + + + + + The $ENV(MHMM_listname)$ $ENV(MHMM_archive)$ Archive by subject + + + +

    $ENV(MHMM_listname)$ $ENV(MHMM_archive)$ Archives by subject

    +
    + +     + +
    +
      + + +
    • $SUBJECTNA$ +
        + + +
      +
      • + + +
    • $FROMNAME$, $FROMADDR$ - $DATE$
      • +       + +
    + + +
    + This archive was generated by a fusion of + Pipermail 0.09 (Mailman edition) and + MHonArc 2.6.8. + + + +     + + + + +