memory error archiving list

Bug #265611 reported by Potstickr
2
Affects Status Importance Assigned to Milestone
GNU Mailman
Fix Released
Medium
Unassigned

Bug Description

I concatenated 12 months (Jan-Dec '97) of majordomo
archives into a
test.mbox, deleted archives/private/test, then did
'bin/arch test'. The
test.mbox file is 32982242 bytes, and the archive
contains approximately
27706 messages. It got as far as creating the html for
August when it
crashed with a memory error:

 Updating HTML for article 15832
Updating index files for archive [1997-August]
Traceback (most recent call last):
  File "./bin/arch", line 129, in ?
    main()
  File "./bin/arch", line 119, in main
    archiver.close()
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 304, in close
    self.update_dirty_archives()
  File "/home/mailman/Mailman/Archiver/HyperArch.py",
line 1195, in update_dirty
_archives
    self.__super_update_dirty_archives()
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 513, in update_dirty_
archives
    self.update_archive(i)
  File "/home/mailman/Mailman/Archiver/HyperArch.py",
line 1198, in update_archi
ve
    self.__super_update_archive(archive)
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 417, in update_archiv
e
    self.__set_parameters(archive)
  File "/home/mailman/Mailman/Archiver/pipermail.py",
line 332, in __set_paramet
ers
    firstdate = self.database.firstdate(archive)
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
207, in firstdate
    self.__openIndices(archive)
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
238, in __openInd
ices
    self.__closeIndices()
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
259, in __closeIn
dices
    index.close()
  File
"/home/mailman/Mailman/Archiver/HyperDatabase.py", line
186, in close
    fp.write(marshal.dumps(self.dict))
MemoryError

I am running mailman 2.0.8 with patches:

mailman-2.0.8-plaintext-0.16.patch
indexing-2.0.6.patch
htdig-2.0.8-0.1.patch

...on NetBSD 1.5.1, python 2.0. Machine is a P3/550
wiuth 512MB.

Let me know if you need the test.mbox file, or if I can
get you any more
information.

Morgan

[http://sourceforge.net/tracker/index.php?func=detail&aid=519708&group_id=103&atid=100103]

Tags: pipermail
Revision history for this message
Barry Warsaw (barry) wrote :

This is a known problem w/ Pipermail. While this bug
specifically won't be fixed, the bin/arch script will grow
--start and --end options which will let you rebuild your
large archives in manageable chunks (although Mailman can't
figure out those chunks for you).

Revision history for this message
Barry Warsaw (barry) wrote :

I'm closing this as (somewhat) fixed for MM2.1. While the
pipermail code makes it impossible to avoid MemoryErrors,
you can now use bin/arch to "chunkify" your archive. Thus
you should be able to run arch over messages in your .mbox
in 10000 msg chunks (or whatever is appropriate for your
system).

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.