memory error archiving list

[Potstickr]

I concatenated 12 months (Jan-Dec '97) of majordomo
archives into a
test.mbox, deleted archives/private/test, then did
'bin/arch test'. The
test.mbox file is 32982242 bytes, and the archive
contains approximately
27706 messages. It got as far as creating the html for
August when it
crashed with a memory error:

 Updating HTML for article 15832
Updating index files for archive [1997-August]
Traceback (most recent call last):
  File "./bin/arch", line 129, in ?
    main()
  File "./bin/arch", line 119, in main
    archiver.close()
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 304, in close
    self.update_dirty_archives()
  File "/home/mailman/Mailman/Archiver/HyperArch.py",
line 1195, in update_dirty
_archives
    self.__super_update_dirty_archives()
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 513, in update_dirty_
archives
    self.update_archive(i)
  File "/home/mailman/Mailman/Archiver/HyperArch.py",
line 1198, in update_archi
ve
    self.__super_update_archive(archive)
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 417, in update_archiv
e
    self.__set_parameters(archive)
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 332, in __set_paramet
ers
    firstdate = self.database.firstdate(archive)
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
207, in firstdate
    self.__openIndices(archive)
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
238, in __openInd
ices
    self.__closeIndices()
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
259, in __closeIn
dices
    index.close()
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
186, in close
    fp.write(marshal.dumps(self.dict))
MemoryError

I am running mailman 2.0.8 with patches:

mailman-2.0.8-plaintext-0.16.patch
indexing-2.0.6.patch
htdig-2.0.8-0.1.patch

...on NetBSD 1.5.1, python 2.0. Machine is a P3/550
wiuth 512MB.

Let me know if you need the test.mbox file, or if I can
get you any more
information.

Morgan

[http://sourceforge.net/tracker/index.php?func=detail&aid=519708&group_id=103&atid=100103]

[Barry Warsaw (barry)]

This is a known problem w/ Pipermail. While this bug
specifically won't be fixed, the bin/arch script will grow
--start and --end options which will let you rebuild your
large archives in manageable chunks (although Mailman can't
figure out those chunks for you).

[Barry Warsaw (barry)]

I'm closing this as (somewhat) fixed for MM2.1. While the
pipermail code makes it impossible to avoid MemoryErrors,
you can now use bin/arch to "chunkify" your archive. Thus
you should be able to run arch over messages in your .mbox
in 10000 msg chunks (or whatever is appropriate for your
system).