bogofilter: last two versions caused db errors

[Debian Bug Importer]

Automatically imported from Debian bug report #293207 http://bugs.debian.org/293207

[Debian Bug Importer (debzilla)]

Automatically imported from Debian bug report #293207 http://bugs.debian.org/293207

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Tue, 01 Feb 2005 13:13:32 -0600
From: Karl Schmidt <email address hidden>
To: Debian Bug Tracking System <email address hidden>
Subject: bogofilter: last two versions caused db errors

Package: bogofilter
Version: 0.93.1-1
Severity: serious
Justification: unkown

-- System Information:
Debian Release: 3.1
  APT prefers testing
  APT policy: (600, 'testing')
Architecture: i386 (i686)
Kernel: Linux 2.6.8-2-k7-smp
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C)

Versions of packages bogofilter depends on:
ii libc6 2.3.2.ds1-20 GNU C Library: Shared libraries an
ii libdb4.2 4.2.52-17 Berkeley v4.2 Database Libraries [
ii libgsl0 1.6-1 The GNU Scientific Library (GSL) -

-- no debconf information

log provides:

datastore_db.c:1056: DB_ENV->open, err: DB_RUNRECOVERY: Fatal error, run database recovery

This applies to both:

 bogofilter_0.93.3.1-1_i386.deb
 bogofilter_0.93.5-1_i386.deb

 I don't think either one should have been in sarge before more testing.

 Version:

 bogofilter_0.93.1-1_i386.deb

 works but also has major bugs filed against it.

 IMHO I think this should be rolled back ASAP due to the very soon release of Sarge.

 I'm not sure of the exact rules, but I don't see the normal time in unstable before these end up in Sarge.

[Clint Adams (clint)]

> Package: bogofilter
> Version: 0.93.1-1
> Severity: serious
> Justification: unkown

Not much of a justification, is it?

> datastore_db.c:1056: DB_ENV->open, err: DB_RUNRECOVERY: Fatal error, run database recovery
>
> bogofilter_0.93.3.1-1_i386.deb
> bogofilter_0.93.5-1_i386.deb
>
> I don't think either one should have been in sarge before more testing.
>
> Version:
>
> bogofilter_0.93.1-1_i386.deb
>
> works but also has major bugs filed against it.
>
> IMHO I think this should be rolled back ASAP due to the very soon release of Sarge.
>
> I'm not sure of the exact rules, but I don't see the normal time in unstable before these end up in Sarge.

Rolled back to what? Did you read NEWS.Debian?

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 1 Feb 2005 15:44:33 -0500
From: Clint Adams <email address hidden>
To: Karl Schmidt <email address hidden>, <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

> Package: bogofilter
> Version: 0.93.1-1
> Severity: serious
> Justification: unkown

Not much of a justification, is it?

> datastore_db.c:1056: DB_ENV->open, err: DB_RUNRECOVERY: Fatal error, run database recovery
>
> bogofilter_0.93.3.1-1_i386.deb
> bogofilter_0.93.5-1_i386.deb
>
> I don't think either one should have been in sarge before more testing.
>
> Version:
>
> bogofilter_0.93.1-1_i386.deb
>
> works but also has major bugs filed against it.
>
> IMHO I think this should be rolled back ASAP due to the very soon release of Sarge.
>
> I'm not sure of the exact rules, but I don't see the normal time in unstable before these end up in Sarge.

Rolled back to what? Did you read NEWS.Debian?

[Karl Schmidt (karl-xtronics)]

Clint Adams wrote:
>>Package: bogofilter
>>Version: 0.93.1-1

Note - the above shoud have read 0.93.5-1

>>Severity: serious
>>Justification: unkown
>
>
> Not much of a justification, is it?

First, I appriciate your efforts.

There really is a policy about spending time in unstable. See:

http://www.debian.org/doc/manuals/developers-reference/ch-pkgs.en.html#s-t-p-u

And

From:

http://release.debian.org/sarge_rc_policy.txt

" makes unrelated software on the system (or the whole system)
   break"

When this breaks it stops Exim in my setup. Thus there are actually two
policy violations.

>>
>> I'm not sure of the exact rules, but I don't see the normal time in unstable before these end up in Sarge.
>
>
> Rolled back to what?

A stable version with a new version number.

At least bogofilter_0.93.1-1_i386.deb or the one before works without
crashing the db file. You have to realize that sarge is about to go
stable. Lots of people are starting to run sarge on production machines.
This didn't spend any time in unstable after the upstream release. I
think 30 days to see what bugs come back would be a starting point.

   Did you read NEWS.Debian?
Yes - even been in the news.

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 FAX [phone]

Merchandise offered without price,
is sure to cost more than it is worth. -kps
----------------------------------------------------------------

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 01 Feb 2005 16:54:14 -0600
From: Karl Schmidt <email address hidden>
To: <email address hidden>
CC: Clint Adams <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

Clint Adams wrote:
>>Package: bogofilter
>>Version: 0.93.1-1

Note - the above shoud have read 0.93.5-1

>>Severity: serious
>>Justification: unkown
>
>
> Not much of a justification, is it?

First, I appriciate your efforts.

There really is a policy about spending time in unstable. See:

http://www.debian.org/doc/manuals/developers-reference/ch-pkgs.en.html#s-t-p-u

And

From:

http://release.debian.org/sarge_rc_policy.txt

" makes unrelated software on the system (or the whole system)
   break"

When this breaks it stops Exim in my setup. Thus there are actually two
policy violations.

>>
>> I'm not sure of the exact rules, but I don't see the normal time in unstable before these end up in Sarge.
>
>
> Rolled back to what?

A stable version with a new version number.

At least bogofilter_0.93.1-1_i386.deb or the one before works without
crashing the db file. You have to realize that sarge is about to go
stable. Lots of people are starting to run sarge on production machines.
This didn't spend any time in unstable after the upstream release. I
think 30 days to see what bugs come back would be a starting point.

   Did you read NEWS.Debian?
Yes - even been in the news.

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 FAX [phone]

Merchandise offered without price,
is sure to cost more than it is worth. -kps
----------------------------------------------------------------

[Clint Adams (clint)]

> There really is a policy about spending time in unstable. See:
>
> http://www.debian.org/doc/manuals/developers-reference/ch-pkgs.en.html#s-t-p-u

bogofilter isn't frozen yet. testing-proposed-updates doesn't apply.

> http://release.debian.org/sarge_rc_policy.txt
>
> " makes unrelated software on the system (or the whole system)
> break"
>
> When this breaks it stops Exim in my setup. Thus there are actually two
> policy violations.

bogofilter isn't breaking exim. Apparently bogofilter is breaking, and
exim is failing to handle that.

> At least bogofilter_0.93.1-1_i386.deb or the one before works without
> crashing the db file. You have to realize that sarge is about to go
> stable. Lots of people are starting to run sarge on production machines.
> This didn't spend any time in unstable after the upstream release. I
> think 30 days to see what bugs come back would be a starting point.

You seem to be the only person to experience such a problem for at least
the past month.

> >For the record, this user is describing switching from a db4.2-linked
> >bogofilter to a db4.3-linked bogofilter.
>
> Not true --- I rebuilt the databases

You may have rebuilt the databases, but that doesn't change the fact
that you're claiming that the db4.2-version is fine, and the
db4.3-version is experiencing corruption.

> It worked for most of 48 hours.
>
> Did the same for the last debian release - with the same problem.

Are you using libdb4.3 4.3.27-1 with the problematic bogofilter versions?

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 1 Feb 2005 18:31:25 -0500
From: Clint Adams <email address hidden>
To: Karl Schmidt <email address hidden>
Cc: <email address hidden>, <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

> There really is a policy about spending time in unstable. See:
>
> http://www.debian.org/doc/manuals/developers-reference/ch-pkgs.en.html#s-t-p-u

bogofilter isn't frozen yet. testing-proposed-updates doesn't apply.

> http://release.debian.org/sarge_rc_policy.txt
>
> " makes unrelated software on the system (or the whole system)
> break"
>
> When this breaks it stops Exim in my setup. Thus there are actually two
> policy violations.

bogofilter isn't breaking exim. Apparently bogofilter is breaking, and
exim is failing to handle that.

> At least bogofilter_0.93.1-1_i386.deb or the one before works without
> crashing the db file. You have to realize that sarge is about to go
> stable. Lots of people are starting to run sarge on production machines.
> This didn't spend any time in unstable after the upstream release. I
> think 30 days to see what bugs come back would be a starting point.

You seem to be the only person to experience such a problem for at least
the past month.

> >For the record, this user is describing switching from a db4.2-linked
> >bogofilter to a db4.3-linked bogofilter.
>
> Not true --- I rebuilt the databases

You may have rebuilt the databases, but that doesn't change the fact
that you're claiming that the db4.2-version is fine, and the
db4.3-version is experiencing corruption.

> It worked for most of 48 hours.
>
> Did the same for the last debian release - with the same problem.

Are you using libdb4.3 4.3.27-1 with the problematic bogofilter versions?

[Karl Schmidt (karl-xtronics)]

Clint Adams wrote:

>
> Are you using libdb4.3 4.3.27-1 with the problematic bogofilter versions?

I have:

libdb4.3 4.3.27-1

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 FAX [phone]

Merchandise offered without price,
is sure to cost more than it is worth. -kps
----------------------------------------------------------------

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 01 Feb 2005 21:03:01 -0600
From: Karl Schmidt <email address hidden>
To: Clint Adams <email address hidden>
CC: <email address hidden>, <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

Clint Adams wrote:

>
> Are you using libdb4.3 4.3.27-1 with the problematic bogofilter versions?

I have:

libdb4.3 4.3.27-1

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 FAX [phone]

Merchandise offered without price,
is sure to cost more than it is worth. -kps
----------------------------------------------------------------

[Karl Schmidt (karl-xtronics)]

Matthias Andree wrote:

>>
>>I have:
>>
>>libdb4.3 4.3.27-1
>
>
> Please run "bogofilter -V" to check the bogofilter and Berkeley DB
> versions, the first two lines are sufficient. Do this with either
> bogofilter version. Remember that if you're inadvertently going forth
> and back between Berkeley DB versions, your database environment may
> break like this. Going backwards isn't supported (so bogoutil -d before
> the upgrade, remove the database, downgrade, bogoutil -l), going
> forwards requires you to remove the environment _BEFORE_ the update.
>
> I have rewritten parts of README.db after the 0.93.5 release, hence I'm
> adding the rewritten version below, perhaps it can help.
>
>
Installing bogofilter on a Debian testing box gives us:

ii bogofilter 0.93.5-1 a fast Bayesian spam filter

$ bogofilter -V
bogofilter version 0.93.5
     Database: Sleepycat Software: Berkeley DB 4.3.27: (December 22, 2004

I delete all the files in the db directory and run the following script
(as I've had to rebuild a few times now<g>):

#!/bin/bash
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2003
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/s-archived-spam
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2004
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/archived
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/list-servers/EXIM
bogofilter -M -n -d /etc/bogofilter -I
chown Debian-exim.Debian-exim /etc/bogofilter/*

Everything works (not sure if it is tagging quite as much spam) then it
ends ups stopping after about 48 hours.

This is on a Tyan MB with ECC memory, antec powersupply - I think a
quite stable system running bind, dhcp,hylasfax, samba, nfs, imap all
flawlessly. I would suspect falky hardware at this point except going
back to the older version fixes things.

Only other thing I can suspect is that exim is threaded - could there be
a locking problem I'm seeing running two requests at a time? I can
imagine that 48 hours would be long enough to be filtering two messages
at the same time. That would explain why most people running in a single
thread POP service manner would not see this bug.

The basic fact is I am sure I recreated the databases and didn't upgrade
and try to run the old data base (which if I remember would have failed
at once.) Going back to the old version and once again reproducing the
databases fixes the problem.

I can think that it would be easy to test by running two or three
instances of bogofilter at the same time on some mail files. One can
write a script that will fork and you might want to add it to your
testing procedure. Hope this helps.

I hope I didn't sound off base here and hope I haven't ruffled any
feathers, but I really do think that these should spend some time in
unstable.

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 ...

[Clint Adams (clint)]

> ii bogofilter 0.93.5-1 a fast Bayesian spam filter
>
> $ bogofilter -V
> bogofilter version 0.93.5
> Database: Sleepycat Software: Berkeley DB 4.3.27: (December 22, 2004

Are you using libc6-i686?

> Only other thing I can suspect is that exim is threaded - could there be
> a locking problem I'm seeing running two requests at a time? I can

Which exim package are you using?

> I hope I didn't sound off base here and hope I haven't ruffled any
> feathers, but I really do think that these should spend some time in
> unstable.

I don't see how that would have helped in this case.

[David Relson (relson-osagesoftware)]

On Wed, 02 Feb 2005 10:40:39 -0600
Karl Schmidt wrote:

> Matthias Andree wrote:

...[snip]...

> Installing bogofilter on a Debian testing box gives us:
>
> ii bogofilter 0.93.5-1 a fast Bayesian spam filter
>
> $ bogofilter -V
> bogofilter version 0.93.5
> Database: Sleepycat Software: Berkeley DB 4.3.27: (December 22, 2004
>
> I delete all the files in the db directory and run the following script
> (as I've had to rebuild a few times now<g>):
>
> #!/bin/bash
> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2003
> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/s-archived-spam
> bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2004
> bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/archived
> bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/list-servers/EXIM
> bogofilter -M -n -d /etc/bogofilter -I
> chown Debian-exim.Debian-exim /etc/bogofilter/*

Your script looks reasonable. I, too, make a practice of to preserving
sequences of useful commands in scripts.

As a tip, using "-B" instead of "-I" allows listing of multiple message
sources (mailboxes, maildirs, etc) in 1 command, e.g.

bogofilter -M -s -d /etc/bogofilter -B /home/karl/mail/*s-archived-spam*

> Everything works (not sure if it is tagging quite as much spam) then it
> ends ups stopping after about 48 hours.
>
> This is on a Tyan MB with ECC memory, antec powersupply - I think a
> quite stable system running bind, dhcp,hylasfax, samba, nfs, imap all
> flawlessly. I would suspect falky hardware at this point except going
> back to the older version fixes things.
>
> Only other thing I can suspect is that exim is threaded - could there be
> a locking problem I'm seeing running two requests at a time? I can
> imagine that 48 hours would be long enough to be filtering two messages
> at the same time. That would explain why most people running in a single
> thread POP service manner would not see this bug.
>
> The basic fact is I am sure I recreated the databases and didn't upgrade
> and try to run the old data base (which if I remember would have failed
> at once.) Going back to the old version and once again reproducing the
> databases fixes the problem.
>
> I can think that it would be easy to test by running two or three
> instances of bogofilter at the same time on some mail files. One can
> write a script that will fork and you might want to add it to your
> testing procedure. Hope this helps.

Bogofilter can be built with "make check" which runs a series of 40 or
so regression tests. The 2 locking tests run multiple copies of
bogofilter to test operating system, database, and bogofilter. If you
have source code, it's probably worth running -- just for grins.

> I hope I didn't sound off base here and hope I haven't ruffled any
> feathers, but I really do think that these should spend some time in
> unstable.

Nope, no ruffled feathers here. Guess you'll have to try harder :-)

Cheers!

David

[Micah Anderson (micah-debian)]

According to README.db, bogofilter is not known to work with anything
greater than 4.3.21:

These versions are supported (with all SleepyCat-posted patches
applied):

[snip]
  Sleepycat Software: Berkeley DB 4.3.21: (November 8, 2004)

Other 3.x or 4.x versions of Berkeley DB may or may not work.

The original submitter reports using libdb4.3 4.3.27-1 which is not
prohibited by the bogofilter control:

Depends: libc6 (>= 2.3.2.ds1-4), libdb4.3, libgsl0 (>= 1.4)

the most recent version of libdb4.3 available in testing is 4.3.27-1
which according to the upstream README.db is not known to be supported.

I too experienced odd corruption when bogofilter and libdb4.3 were
upgraded in a recent dist-upgrade. It caused massive spam hemmoraghing
until I realized that the database was corrupted and I went through a
number of recovery solution paths to get it working again (I ended up
having to dump the wordlist to a text file, clear out my DB
environment and then import my wordlist).

I have had no problem since then (I do not have the 48 hour syndrome
the original submitter indicates).

I suggest either making a versioned depends on libdb4.3 so that people
don't upgrade to 4.3.27, until upstream indicates that they support
that version, or maybe there is a conversion process that will work
that can be scripted?

Micah

[Micah Anderson (micah-debian)]

# Automatically generated email from bts, devscripts version 2.8.10
tags 293207 + moreinfo

[Micah Anderson (micah-debian)]

The original bug report said that you were running:

Version: 0.93.1-1
ii libdb4.2 4.2.52-17 Berkeley v4.2 Database Libraries

and your problem occurs with both:
bogofilter_0.93.3.1-1_i386.deb
bogofilter_0.93.5-1_i386.deb

You say that bogofilter_0.93.1-1_i386.deb works and then go on to say:

"At least bogofilter_0.93.1-1_i386.deb or the one before works without
crashing the db file."

This is because your log format needs to be updated to the new format,
as the NEWS.Debian file indicates. Version 0.93.1 depended on libdb
4.2.25 (as you correctly have in the original report), moving to
0.93.0-2 or anything newer requires you to upgrade your libdb to 4.3
and perform the NEWS.Debian log format change procedure.

I had this same problem because I didn't think I needed to perform the
log format upgrade, but I was wrong. As soon as I did it, things
worked fine again.

I suggest that you either go through the more painful proceedure of
dumping your wordlists to .txt files, destroying your DB environments
and then recreating them by reloading your wordlist.txt files, or
follow the steps outlined in the NEWS.Debian to upgrade your wordlist
databases.

I am fairly confident that you will then find that your problem no
longer exists, as this is exactly what I had to do to make the problem
go away.

[Karl Schmidt (karl-xtronics)]

Micah Anderson wrote:
> The original bug report said that you were running:
>
> Version: 0.93.1-1
> ii libdb4.2 4.2.52-17 Berkeley v4.2 Database Libraries
>
> and your problem occurs with both:
> bogofilter_0.93.3.1-1_i386.deb
> bogofilter_0.93.5-1_i386.deb
>
> You say that bogofilter_0.93.1-1_i386.deb works and then go on to say:
>
> "At least bogofilter_0.93.1-1_i386.deb or the one before works without
> crashing the db file."
>

The version number was wrong because I had already rolled it back to
93.1-1 The bug refers to 93.5-1 (and 93.3.1-1)

> This is because your log format needs to be updated to the new format,
> as the NEWS.Debian file indicates.

I deleted and recreated the db files before I ran with the upgrade.

> Version 0.93.1 depended on libdb
> 4.2.25 (as you correctly have in the original report), moving to
> 0.93.0-2 or anything newer requires you to upgrade your libdb to 4.3
> and perform the NEWS.Debian log format change procedure.

I had and still have installed:
ii libdb1-compat 2.1.3-7 The Berkeley database routines [glibc
2.0/2.
ii libdb3 3.2.9-20 Berkeley v3 Database Libraries [runtime]
ii libdb3-dev 3.2.9-20 Berkeley v3 Database Libraries
[development]
ii libdb4.2 4.2.52-17 Berkeley v4.2 Database Libraries [runtime]
ii libdb4.3 4.3.27-1 Berkeley v4.3 Database Libraries [runtime]

My understanding is that these do not conflict (if so this is a
dependency issue).
  > I suggest that you either go through the more painful proceedure of
> dumping your wordlists to .txt files, destroying your DB environments
> and then recreating them by reloading your wordlist.txt files, or
> follow the steps outlined in the NEWS.Debian to upgrade your wordlist
> databases.

I deleted and recreated the db files.

>
> I am fairly confident that you will then find that your problem no
> longer exists, as this is exactly what I had to do to make the problem
> go away.

The problem exists after recreating all new db and log files via
bogofilter. (I archive all ham AND spam). It only appeared after about
48 hours of operation (the wrong version fails right away).

I don't see how this problem can be related to a change in db files when
they were created after the upgrade.

If bogofilter is using the wrong version of libdb that is not because
old db files because I deleted and recreated them.

For the record -The only lines from a grep db /etc/bogofilter.cf are
commented out just as in the default.

###wordlist i,ignore,~/ignorelist.db,1
###wordlist r,wordlist,~/wordlist.db,2
##db_cachesize=0
###db_cachesize=16
## If you run out of locks, see file README.db,
##db_lk_max_locks=16384
##db_lk_max_objects=16384

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 FAX [phone]

"Even if you are on the right track,
you'll get run over if you just sit there" - Will Rogers
------------------------------------------...

[Micah Anderson (micah-debian)]

On Sat, 05 Feb 2005, Karl Schmidt wrote:

> Micah Anderson wrote:
> >The original bug report said that you were running:
> >
> >Version: 0.93.1-1
> >ii libdb4.2 4.2.52-17 Berkeley v4.2 Database
> >Libraries
> >
> >and your problem occurs with both:
> >bogofilter_0.93.3.1-1_i386.deb
> >bogofilter_0.93.5-1_i386.deb
> >
> >You say that bogofilter_0.93.1-1_i386.deb works and then go on to say:
> >
> >"At least bogofilter_0.93.1-1_i386.deb or the one before works without
> >crashing the db file."
> >
>
> The version number was wrong because I had already rolled it back to
> 93.1-1 The bug refers to 93.5-1 (and 93.3.1-1)

Your report is very confusing because of this.

Let me see if I understand:

1. You upgraded bogofilter to 0.93.5-1 from 0.93.0-1
2. you followed the NEWS.Debian instructions to convert your wordlist
db log format to the new version using db4.2_recover and db4.2_archive
(NOTE! 4.2, not 4.3, or db_recover, db_archive or db3_recover,
db3_archive)
3. About 48 hours later you get the error "datastore_db.c:1056:
DB_ENV->open, err: DB_RUNRECOVERY: Fatal error, run database recovery"
somewhere in an exim logfile and bogofilter ceases to work properly.
4. You remove 0.93.5-1
5. You install 0.93.0-1 and recreate your DB files with a db4.2
environment
6. Things work again.
7. Same problem with 0.93.1-1 following same procedures

> I had and still have installed:
> ii libdb1-compat 2.1.3-7 The Berkeley database routines [glibc
> 2.0/2.
> ii libdb3 3.2.9-20 Berkeley v3 Database Libraries [runtime]
> ii libdb3-dev 3.2.9-20 Berkeley v3 Database Libraries
> [development]
> ii libdb4.2 4.2.52-17 Berkeley v4.2 Database Libraries [runtime]
> ii libdb4.3 4.3.27-1 Berkeley v4.3 Database Libraries [runtime]
>
> My understanding is that these do not conflict (if so this is a
> dependency issue).

This is fine, I've got all these installed and I don't have the
problem you are having.

> The problem exists after recreating all new db and log files via
> bogofilter. (I archive all ham AND spam). It only appeared after about
> 48 hours of operation (the wrong version fails right away).

"It only appeared" means the error:
"datastore_db.c:1056: DB_ENV->open, err: DB_RUNRECOVERY: Fatal error, run database recovery"
appears somewhere in a logfile?

"the wrong version fails right away" -- what is the wrong version? and
how do you determine that it has failed right away?

> I don't see how this problem can be related to a change in db files when
> they were created after the upgrade.

It shouldn't...

Micah

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Wed, 02 Feb 2005 10:40:39 -0600
From: Karl Schmidt <email address hidden>
To: <email address hidden>
CC: <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

Matthias Andree wrote:

>>
>>I have:
>>
>>libdb4.3 4.3.27-1
>
>
> Please run "bogofilter -V" to check the bogofilter and Berkeley DB
> versions, the first two lines are sufficient. Do this with either
> bogofilter version. Remember that if you're inadvertently going forth
> and back between Berkeley DB versions, your database environment may
> break like this. Going backwards isn't supported (so bogoutil -d before
> the upgrade, remove the database, downgrade, bogoutil -l), going
> forwards requires you to remove the environment _BEFORE_ the update.
>
> I have rewritten parts of README.db after the 0.93.5 release, hence I'm
> adding the rewritten version below, perhaps it can help.
>
>
Installing bogofilter on a Debian testing box gives us:

ii bogofilter 0.93.5-1 a fast Bayesian spam filter

$ bogofilter -V
bogofilter version 0.93.5
     Database: Sleepycat Software: Berkeley DB 4.3.27: (December 22, 2004

I delete all the files in the db directory and run the following script
(as I've had to rebuild a few times now<g>):

#!/bin/bash
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2003
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/s-archived-spam
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2004
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/archived
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/list-servers/EXIM
bogofilter -M -n -d /etc/bogofilter -I
chown Debian-exim.Debian-exim /etc/bogofilter/*

Everything works (not sure if it is tagging quite as much spam) then it
ends ups stopping after about 48 hours.

This is on a Tyan MB with ECC memory, antec powersupply - I think a
quite stable system running bind, dhcp,hylasfax, samba, nfs, imap all
flawlessly. I would suspect falky hardware at this point except going
back to the older version fixes things.

Only other thing I can suspect is that exim is threaded - could there be
a locking problem I'm seeing running two requests at a time? I can
imagine that 48 hours would be long enough to be filtering two messages
at the same time. That would explain why most people running in a single
thread POP service manner would not see this bug.

The basic fact is I am sure I recreated the databases and didn't upgrade
and try to run the old data base (which if I remember would have failed
at once.) Going back to the old version and once again reproducing the
databases fixes the problem.

I can think that it would be easy to test by running two or three
instances of bogofilter at the same time on some mail files. One can
write a script that will fork and you might want to add it to your
testing procedure. Hope this helps.

I hope I didn't sound off base here and hope I haven't ruffled any
feathers, but I really do think that these should spend some time in
unstable.

-----------...

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Wed, 2 Feb 2005 17:24:04 -0500
From: Clint Adams <email address hidden>
To: Karl Schmidt <email address hidden>
Cc: <email address hidden>, <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

> ii bogofilter 0.93.5-1 a fast Bayesian spam filter
>
> $ bogofilter -V
> bogofilter version 0.93.5
> Database: Sleepycat Software: Berkeley DB 4.3.27: (December 22, 2004

Are you using libc6-i686?

> Only other thing I can suspect is that exim is threaded - could there be
> a locking problem I'm seeing running two requests at a time? I can

Which exim package are you using?

> I hope I didn't sound off base here and hope I haven't ruffled any
> feathers, but I really do think that these should spend some time in
> unstable.

I don't see how that would have helped in this case.

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Wed, 2 Feb 2005 19:33:44 -0500
From: David Relson <email address hidden>
To: Karl Schmidt <email address hidden>
Cc: <email address hidden>, <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

On Wed, 02 Feb 2005 10:40:39 -0600
Karl Schmidt wrote:

> Matthias Andree wrote:

...[snip]...

> Installing bogofilter on a Debian testing box gives us:
>
> ii bogofilter 0.93.5-1 a fast Bayesian spam filter
>
> $ bogofilter -V
> bogofilter version 0.93.5
> Database: Sleepycat Software: Berkeley DB 4.3.27: (December 22, 2004
>
> I delete all the files in the db directory and run the following script
> (as I've had to rebuild a few times now<g>):
>
> #!/bin/bash
> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2003
> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/s-archived-spam
> bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2004
> bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/archived
> bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/list-servers/EXIM
> bogofilter -M -n -d /etc/bogofilter -I
> chown Debian-exim.Debian-exim /etc/bogofilter/*

Your script looks reasonable. I, too, make a practice of to preserving
sequences of useful commands in scripts.

As a tip, using "-B" instead of "-I" allows listing of multiple message
sources (mailboxes, maildirs, etc) in 1 command, e.g.

bogofilter -M -s -d /etc/bogofilter -B /home/karl/mail/*s-archived-spam*

> Everything works (not sure if it is tagging quite as much spam) then it
> ends ups stopping after about 48 hours.
>
> This is on a Tyan MB with ECC memory, antec powersupply - I think a
> quite stable system running bind, dhcp,hylasfax, samba, nfs, imap all
> flawlessly. I would suspect falky hardware at this point except going
> back to the older version fixes things.
>
> Only other thing I can suspect is that exim is threaded - could there be
> a locking problem I'm seeing running two requests at a time? I can
> imagine that 48 hours would be long enough to be filtering two messages
> at the same time. That would explain why most people running in a single
> thread POP service manner would not see this bug.
>
> The basic fact is I am sure I recreated the databases and didn't upgrade
> and try to run the old data base (which if I remember would have failed
> at once.) Going back to the old version and once again reproducing the
> databases fixes the problem.
>
> I can think that it would be easy to test by running two or three
> instances of bogofilter at the same time on some mail files. One can
> write a script that will fork and you might want to add it to your
> testing procedure. Hope this helps.

Bogofilter can be built with "make check" which runs a series of 40 or
so regression tests. The 2 locking tests run multiple copies of
bogofilter to test operating system, database, and bogofilter. If you
have source code, it's probably worth running -- just for grins.

> I hope I didn't sound off base h...

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sat, 5 Feb 2005 10:48:19 -0600
From: Micah Anderson <email address hidden>
To: <email address hidden>
Subject: Add versioned depends on libdb4.3

--Qz2CZ664xQdCRdPu
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

According to README.db, bogofilter is not known to work with anything
greater than 4.3.21:

These versions are supported (with all SleepyCat-posted patches
applied):

[snip]
  Sleepycat Software: Berkeley DB 4.3.21: (November 8, 2004)

Other 3.x or 4.x versions of Berkeley DB may or may not work.

The original submitter reports using libdb4.3 4.3.27-1 which is not
prohibited by the bogofilter control:

Depends: libc6 (>=3D 2.3.2.ds1-4), libdb4.3, libgsl0 (>=3D 1.4)

the most recent version of libdb4.3 available in testing is 4.3.27-1
which according to the upstream README.db is not known to be supported.=20

I too experienced odd corruption when bogofilter and libdb4.3 were
upgraded in a recent dist-upgrade. It caused massive spam hemmoraghing
until I realized that the database was corrupted and I went through a
number of recovery solution paths to get it working again (I ended up
having to dump the wordlist to a text file, clear out my DB
environment and then import my wordlist).

I have had no problem since then (I do not have the 48 hour syndrome
the original submitter indicates).

I suggest either making a versioned depends on libdb4.3 so that people
don't upgrade to 4.3.27, until upstream indicates that they support
that version, or maybe there is a conversion process that will work
that can be scripted?

Micah

--Qz2CZ664xQdCRdPu
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCBPjT9n4qXRzy1ioRAuc/AJ9mmfP77Fs8Jvckosopoxc8uwyqowCfYhZQ
yj4xQOkxijygFTyUQA1rsLk=
=96+N
-----END PGP SIGNATURE-----

--Qz2CZ664xQdCRdPu--

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sat, 5 Feb 2005 12:03:50 -0600
From: Micah Anderson <email address hidden>
To: <email address hidden>
Cc: <email address hidden>
Subject: Re: bogofilter: last two versions caused db errors

--0rSojgWGcpz+ezC3
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

The original bug report said that you were running:

Version: 0.93.1-1
ii libdb4.2 4.2.52-17 Berkeley v4.2 Database Librari=
es

and your problem occurs with both:
bogofilter_0.93.3.1-1_i386.deb
bogofilter_0.93.5-1_i386.deb

You say that bogofilter_0.93.1-1_i386.deb works and then go on to say:

"At least bogofilter_0.93.1-1_i386.deb or the one before works without=20
crashing the db file."

This is because your log format needs to be updated to the new format,
as the NEWS.Debian file indicates. Version 0.93.1 depended on libdb
4.2.25 (as you correctly have in the original report), moving to
0.93.0-2 or anything newer requires you to upgrade your libdb to 4.3
and perform the NEWS.Debian log format change procedure.=20

I had this same problem because I didn't think I needed to perform the
log format upgrade, but I was wrong. As soon as I did it, things
worked fine again.

I suggest that you either go through the more painful proceedure of
dumping your wordlists to .txt files, destroying your DB environments
and then recreating them by reloading your wordlist.txt files, or
follow the steps outlined in the NEWS.Debian to upgrade your wordlist
databases.=20

I am fairly confident that you will then find that your problem no
longer exists, as this is exactly what I had to do to make the problem
go away.

--0rSojgWGcpz+ezC3
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCBQqG9n4qXRzy1ioRArlrAKCUASWkyXs8IIemOihS4zPKTdu8jgCfWTH2
Vxfje/fU3kPrC/OgJn3T/pY=
=ohtF
-----END PGP SIGNATURE-----

--0rSojgWGcpz+ezC3--

[Debian Bug Importer (debzilla)]

Message-Id: <20050205180346.C2DC63ABB0@pond>
Date: Sat, 5 Feb 2005 12:03:46 -0600
From: Micah Anderson <email address hidden>
To: <email address hidden>
Subject: tagging 293207

# Automatically generated email from bts, devscripts version 2.8.10
tags 293207 + moreinfo

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sat, 05 Feb 2005 14:35:36 -0600
From: Karl Schmidt <email address hidden>
To: Micah Anderson <email address hidden>
CC: <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

Micah Anderson wrote:
> The original bug report said that you were running:
>
> Version: 0.93.1-1
> ii libdb4.2 4.2.52-17 Berkeley v4.2 Database Libraries
>
> and your problem occurs with both:
> bogofilter_0.93.3.1-1_i386.deb
> bogofilter_0.93.5-1_i386.deb
>
> You say that bogofilter_0.93.1-1_i386.deb works and then go on to say:
>
> "At least bogofilter_0.93.1-1_i386.deb or the one before works without
> crashing the db file."
>

The version number was wrong because I had already rolled it back to
93.1-1 The bug refers to 93.5-1 (and 93.3.1-1)

> This is because your log format needs to be updated to the new format,
> as the NEWS.Debian file indicates.

I deleted and recreated the db files before I ran with the upgrade.

> Version 0.93.1 depended on libdb
> 4.2.25 (as you correctly have in the original report), moving to
> 0.93.0-2 or anything newer requires you to upgrade your libdb to 4.3
> and perform the NEWS.Debian log format change procedure.

I had and still have installed:
ii libdb1-compat 2.1.3-7 The Berkeley database routines [glibc
2.0/2.
ii libdb3 3.2.9-20 Berkeley v3 Database Libraries [runtime]
ii libdb3-dev 3.2.9-20 Berkeley v3 Database Libraries
[development]
ii libdb4.2 4.2.52-17 Berkeley v4.2 Database Libraries [runtime]
ii libdb4.3 4.3.27-1 Berkeley v4.3 Database Libraries [runtime]

My understanding is that these do not conflict (if so this is a
dependency issue).
  > I suggest that you either go through the more painful proceedure of
> dumping your wordlists to .txt files, destroying your DB environments
> and then recreating them by reloading your wordlist.txt files, or
> follow the steps outlined in the NEWS.Debian to upgrade your wordlist
> databases.

I deleted and recreated the db files.

>
> I am fairly confident that you will then find that your problem no
> longer exists, as this is exactly what I had to do to make the problem
> go away.

The problem exists after recreating all new db and log files via
bogofilter. (I archive all ham AND spam). It only appeared after about
48 hours of operation (the wrong version fails right away).

I don't see how this problem can be related to a change in db files when
they were created after the upgrade.

If bogofilter is using the wrong version of libdb that is not because
old db files because I deleted and recreated them.

For the record -The only lines from a grep db /etc/bogofilter.cf are
commented out just as in the default.

###wordlist i,ignore,~/ignorelist.db,1
###wordlist r,wordlist,~/wordlist.db,2
##db_cachesize=0
###db_cachesize=16
## If you run out of locks, see file README.db,
##db_lk_max_locks=16384
##db_lk_max_objects=16384

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.c...

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sat, 5 Feb 2005 15:42:04 -0600
From: Micah Anderson <email address hidden>
To: Karl Schmidt <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

On Sat, 05 Feb 2005, Karl Schmidt wrote:

> Micah Anderson wrote:
> >The original bug report said that you were running:
> >
> >Version: 0.93.1-1
> >ii libdb4.2 4.2.52-17 Berkeley v4.2 Database
> >Libraries
> >
> >and your problem occurs with both:
> >bogofilter_0.93.3.1-1_i386.deb
> >bogofilter_0.93.5-1_i386.deb
> >
> >You say that bogofilter_0.93.1-1_i386.deb works and then go on to say:
> >
> >"At least bogofilter_0.93.1-1_i386.deb or the one before works without
> >crashing the db file."
> >
>
> The version number was wrong because I had already rolled it back to
> 93.1-1 The bug refers to 93.5-1 (and 93.3.1-1)

Your report is very confusing because of this.

Let me see if I understand:

1. You upgraded bogofilter to 0.93.5-1 from 0.93.0-1
2. you followed the NEWS.Debian instructions to convert your wordlist
db log format to the new version using db4.2_recover and db4.2_archive
(NOTE! 4.2, not 4.3, or db_recover, db_archive or db3_recover,
db3_archive)
3. About 48 hours later you get the error "datastore_db.c:1056:
DB_ENV->open, err: DB_RUNRECOVERY: Fatal error, run database recovery"
somewhere in an exim logfile and bogofilter ceases to work properly.
4. You remove 0.93.5-1
5. You install 0.93.0-1 and recreate your DB files with a db4.2
environment
6. Things work again.
7. Same problem with 0.93.1-1 following same procedures

> I had and still have installed:
> ii libdb1-compat 2.1.3-7 The Berkeley database routines [glibc
> 2.0/2.
> ii libdb3 3.2.9-20 Berkeley v3 Database Libraries [runtime]
> ii libdb3-dev 3.2.9-20 Berkeley v3 Database Libraries
> [development]
> ii libdb4.2 4.2.52-17 Berkeley v4.2 Database Libraries [runtime]
> ii libdb4.3 4.3.27-1 Berkeley v4.3 Database Libraries [runtime]
>
> My understanding is that these do not conflict (if so this is a
> dependency issue).

This is fine, I've got all these installed and I don't have the
problem you are having.

> The problem exists after recreating all new db and log files via
> bogofilter. (I archive all ham AND spam). It only appeared after about
> 48 hours of operation (the wrong version fails right away).

"It only appeared" means the error:
"datastore_db.c:1056: DB_ENV->open, err: DB_RUNRECOVERY: Fatal error, run database recovery"
appears somewhere in a logfile?

"the wrong version fails right away" -- what is the wrong version? and
how do you determine that it has failed right away?

> I don't see how this problem can be related to a change in db files when
> they were created after the upgrade.

It shouldn't...

Micah

[Karl Schmidt (karl-xtronics)]

Micah Anderson wrote:

>
>
> Can you provide the configuration you use with exim to plug bogofilter
> in? I am particularly interested because version 4.34-10 of exim
> depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> to know if exim uses libdb to interface with bogofilter in any way.
>
> micah

I can use libdb - but I'm not using any db lists.

Here are the bogofilter config areas:

begin routers

#-----------------------------------------------------------

# Bogofilter will add X-Bogosity header to all incoming mail.
# This usually goes right after the dns_lookup router and
# before any local deliver routers. Location is important!
bogo_router:
     domains = +local_domains
     no_verify
     condition = ${if !eq {$received_protocol}{bogodone} {1}{0}}
     driver = accept
     transport = bogo_transport

<snip>
begin transports
#------------------------------------------------
# Bogofilter will add X-Bogosity header to all incoming mail.
# This can go anywhere in the transport section, usually at
# the very end after address_reply
bogo_transport:
   driver = pipe
   command = /usr/sbin/exim4 -oMr bogodone -bS
   use_bsmtp = true
   headers_add = X-Bogofilterd: true
  transport_filter = /usr/bin/bogofilter -d /etc/bogofilter -l -p -e -u
   return_fail_output = true
   group = Debian-exim
   user = Debian-exim
   home_directory = "/tmp"
   current_directory = "/tmp"
   log_output = true
   return_path_add = false

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 FAX [phone]

If it is worth doing, it is worth doing for money.
----------------------------------------------------------------

[Micah Anderson (micah-debian)]

Karl,

Can you send your /etc/bogofilter.cf?

I am trying to replicate your setup to see if I can get the same
problem.

Micah

On Tue, 08 Feb 2005, Karl Schmidt wrote:

> Micah Anderson wrote:
>
> >
> >
> >Can you provide the configuration you use with exim to plug bogofilter
> >in? I am particularly interested because version 4.34-10 of exim
> >depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> >to know if exim uses libdb to interface with bogofilter in any way.
> >
> >micah
>
> I can use libdb - but I'm not using any db lists.
>
> Here are the bogofilter config areas:
>
> begin routers
>
> #-----------------------------------------------------------
>
> # Bogofilter will add X-Bogosity header to all incoming mail.
> # This usually goes right after the dns_lookup router and
> # before any local deliver routers. Location is important!
> bogo_router:
> domains = +local_domains
> no_verify
> condition = ${if !eq {$received_protocol}{bogodone} {1}{0}}
> driver = accept
> transport = bogo_transport
>
>
>
> <snip>
> begin transports
> #------------------------------------------------
> # Bogofilter will add X-Bogosity header to all incoming mail.
> # This can go anywhere in the transport section, usually at
> # the very end after address_reply
> bogo_transport:
> driver = pipe
> command = /usr/sbin/exim4 -oMr bogodone -bS
> use_bsmtp = true
> headers_add = X-Bogofilterd: true
> transport_filter = /usr/bin/bogofilter -d /etc/bogofilter -l -p -e -u
> return_fail_output = true
> group = Debian-exim
> user = Debian-exim
> home_directory = "/tmp"
> current_directory = "/tmp"
> log_output = true
> return_path_add = false
>
>
>
>
>
>
> ----------------------------------------------------------------
> Karl Schmidt EMail <email address hidden>
> Transtronics, Inc. WEB http://xtronics.com
> 3209 West 9th Street Ph [phone]
> Lawrence, KS 66049 FAX [phone]
>
>
> If it is worth doing, it is worth doing for money.
> ----------------------------------------------------------------

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 08 Feb 2005 21:18:42 -0600
From: Karl Schmidt <email address hidden>
To: Micah Anderson <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

Micah Anderson wrote:

>
>
> Can you provide the configuration you use with exim to plug bogofilter
> in? I am particularly interested because version 4.34-10 of exim
> depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> to know if exim uses libdb to interface with bogofilter in any way.
>
> micah

I can use libdb - but I'm not using any db lists.

Here are the bogofilter config areas:

begin routers

#-----------------------------------------------------------

# Bogofilter will add X-Bogosity header to all incoming mail.
# This usually goes right after the dns_lookup router and
# before any local deliver routers. Location is important!
bogo_router:
     domains = +local_domains
     no_verify
     condition = ${if !eq {$received_protocol}{bogodone} {1}{0}}
     driver = accept
     transport = bogo_transport

<snip>
begin transports
#------------------------------------------------
# Bogofilter will add X-Bogosity header to all incoming mail.
# This can go anywhere in the transport section, usually at
# the very end after address_reply
bogo_transport:
   driver = pipe
   command = /usr/sbin/exim4 -oMr bogodone -bS
   use_bsmtp = true
   headers_add = X-Bogofilterd: true
  transport_filter = /usr/bin/bogofilter -d /etc/bogofilter -l -p -e -u
   return_fail_output = true
   group = Debian-exim
   user = Debian-exim
   home_directory = "/tmp"
   current_directory = "/tmp"
   log_output = true
   return_path_add = false

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 FAX [phone]

If it is worth doing, it is worth doing for money.
----------------------------------------------------------------

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Wed, 9 Feb 2005 09:53:56 -0600
From: Micah Anderson <email address hidden>
To: Karl Schmidt <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

Karl,

Can you send your /etc/bogofilter.cf?

I am trying to replicate your setup to see if I can get the same
problem.

Micah

On Tue, 08 Feb 2005, Karl Schmidt wrote:

> Micah Anderson wrote:
>
> >
> >
> >Can you provide the configuration you use with exim to plug bogofilter
> >in? I am particularly interested because version 4.34-10 of exim
> >depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> >to know if exim uses libdb to interface with bogofilter in any way.
> >
> >micah
>
> I can use libdb - but I'm not using any db lists.
>
> Here are the bogofilter config areas:
>
> begin routers
>
> #-----------------------------------------------------------
>
> # Bogofilter will add X-Bogosity header to all incoming mail.
> # This usually goes right after the dns_lookup router and
> # before any local deliver routers. Location is important!
> bogo_router:
> domains = +local_domains
> no_verify
> condition = ${if !eq {$received_protocol}{bogodone} {1}{0}}
> driver = accept
> transport = bogo_transport
>
>
>
> <snip>
> begin transports
> #------------------------------------------------
> # Bogofilter will add X-Bogosity header to all incoming mail.
> # This can go anywhere in the transport section, usually at
> # the very end after address_reply
> bogo_transport:
> driver = pipe
> command = /usr/sbin/exim4 -oMr bogodone -bS
> use_bsmtp = true
> headers_add = X-Bogofilterd: true
> transport_filter = /usr/bin/bogofilter -d /etc/bogofilter -l -p -e -u
> return_fail_output = true
> group = Debian-exim
> user = Debian-exim
> home_directory = "/tmp"
> current_directory = "/tmp"
> log_output = true
> return_path_add = false
>
>
>
>
>
>
> ----------------------------------------------------------------
> Karl Schmidt EMail <email address hidden>
> Transtronics, Inc. WEB http://xtronics.com
> 3209 West 9th Street Ph [phone]
> Lawrence, KS 66049 FAX [phone]
>
>
> If it is worth doing, it is worth doing for money.
> ----------------------------------------------------------------

[Clint Adams (clint)]

> >Can you provide the configuration you use with exim to plug bogofilter
> >in? I am particularly interested because version 4.34-10 of exim
> >depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> >to know if exim uses libdb to interface with bogofilter in any way.
> >
> >micah
>
> I can use libdb - but I'm not using any db lists.

That's an interesting question; maybe you should try the exim in
project/experimental, which is linked against libdb4.2.

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Thu, 10 Feb 2005 10:53:17 -0500
From: Clint Adams <email address hidden>
To: Karl Schmidt <email address hidden>, <email address hidden>
Cc: Micah Anderson <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

> >Can you provide the configuration you use with exim to plug bogofilter
> >in? I am particularly interested because version 4.34-10 of exim
> >depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> >to know if exim uses libdb to interface with bogofilter in any way.
> >
> >micah
>
> I can use libdb - but I'm not using any db lists.

That's an interesting question; maybe you should try the exim in
project/experimental, which is linked against libdb4.2.

[Micah Anderson (micah-debian)]

A report on my findings thus far:

I set up a pristine test environment to try and replicate the problem
that Karl has been having. I have been able to setup a system running
exim with the same configuration that Karl has, plugged into
delivering mail through bogofilter, using the same bogofilter
configuration as Karl. To setup the bogofilter databases, I fed
approximately 2,000 ham messages and approximately 6,000 spam messages
into bogofilter, creating database files in /etc/bogofilter with the
same permissions as Exim.

I then proceeded to send test mails. In the beginning bogofilter
failed after about 20 messages due to database corruption. This was
because the /etc/bogofilter directory was not set to have the correct
ownership and permissions as was needed. After confirming with Karl
how he had his setup, I changed it in the test scenario and restarted
the tests. I ran tests that simulated mail delivery of approximately
1200 messages an hour (approximately 20/minute) for over 24 hours with
no failures, I delivered approximately 35,000 messages with no
problems at all.

This is with exim 4.34-10 and bogofilter 0.93.5-1. It certainly seems
that with a fresh installation of these versions everything works as
expected.

I am now going to try the same suite of tests with version 0.93.3 and
then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
down an older version of the .deb.

micah

On Thu, 10 Feb 2005, Clint Adams wrote:

> > >Can you provide the configuration you use with exim to plug bogofilter
> > >in? I am particularly interested because version 4.34-10 of exim
> > >depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> > >to know if exim uses libdb to interface with bogofilter in any way.
> > >
> > >micah
> >
> > I can use libdb - but I'm not using any db lists.
>
> That's an interesting question; maybe you should try the exim in
> project/experimental, which is linked against libdb4.2.

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Thu, 10 Feb 2005 23:55:49 -0600
From: Micah Anderson <email address hidden>
To: Clint Adams <email address hidden>
Cc: Karl Schmidt <email address hidden>, <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

A report on my findings thus far:

I set up a pristine test environment to try and replicate the problem
that Karl has been having. I have been able to setup a system running
exim with the same configuration that Karl has, plugged into
delivering mail through bogofilter, using the same bogofilter
configuration as Karl. To setup the bogofilter databases, I fed
approximately 2,000 ham messages and approximately 6,000 spam messages
into bogofilter, creating database files in /etc/bogofilter with the
same permissions as Exim.

I then proceeded to send test mails. In the beginning bogofilter
failed after about 20 messages due to database corruption. This was
because the /etc/bogofilter directory was not set to have the correct
ownership and permissions as was needed. After confirming with Karl
how he had his setup, I changed it in the test scenario and restarted
the tests. I ran tests that simulated mail delivery of approximately
1200 messages an hour (approximately 20/minute) for over 24 hours with
no failures, I delivered approximately 35,000 messages with no
problems at all.

This is with exim 4.34-10 and bogofilter 0.93.5-1. It certainly seems
that with a fresh installation of these versions everything works as
expected.

I am now going to try the same suite of tests with version 0.93.3 and
then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
down an older version of the .deb.

micah

On Thu, 10 Feb 2005, Clint Adams wrote:

> > >Can you provide the configuration you use with exim to plug bogofilter
> > >in? I am particularly interested because version 4.34-10 of exim
> > >depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> > >to know if exim uses libdb to interface with bogofilter in any way.
> > >
> > >micah
> >
> > I can use libdb - but I'm not using any db lists.
>
> That's an interesting question; maybe you should try the exim in
> project/experimental, which is linked against libdb4.2.

[Justin Pryzby (justinpryzby-users)]

On Thu, Feb 10, 2005 at 11:55:49PM -0600, Micah Anderson wrote:
> A report on my findings thus far:
>
> I set up a pristine test environment to try and replicate the problem
> that Karl has been having. I have been able to setup a system running
> exim with the same configuration that Karl has, plugged into
> delivering mail through bogofilter, using the same bogofilter
> configuration as Karl. To setup the bogofilter databases, I fed
> approximately 2,000 ham messages and approximately 6,000 spam messages
> into bogofilter, creating database files in /etc/bogofilter with the
> same permissions as Exim.
>
> I then proceeded to send test mails. In the beginning bogofilter
> failed after about 20 messages due to database corruption. This was
> because the /etc/bogofilter directory was not set to have the correct
> ownership and permissions as was needed. After confirming with Karl
> how he had his setup, I changed it in the test scenario and restarted
> the tests. I ran tests that simulated mail delivery of approximately
> 1200 messages an hour (approximately 20/minute) for over 24 hours with
> no failures, I delivered approximately 35,000 messages with no
> problems at all.
>
> This is with exim 4.34-10 and bogofilter 0.93.5-1. It certainly seems
> that with a fresh installation of these versions everything works as
> expected.
>
> I am now going to try the same suite of tests with version 0.93.3 and
> then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
> have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
> down an older version of the .deb.
Well, normally there is
http://www.backports.org/debian/dists/woody/bogofilter/binary-i386/
but, it doesn't have any other versions.

I also tried Google:
http://www.google.com/search?hl=en&lr=&q=inurl%3Abogofilter_0.93.3&btnG=Search
but without success.

:-/

Justin

> On Thu, 10 Feb 2005, Clint Adams wrote:
>
> > > >Can you provide the configuration you use with exim to plug bogofilter
> > > >in? I am particularly interested because version 4.34-10 of exim
> > > >depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> > > >to know if exim uses libdb to interface with bogofilter in any way.
> > > >
> > > >micah
> > >
> > > I can use libdb - but I'm not using any db lists.
> >
> > That's an interesting question; maybe you should try the exim in
> > project/experimental, which is linked against libdb4.2.
>
>
> --
> To UNSUBSCRIBE, email to <email address hidden>
> with a subject of "unsubscribe". Trouble? Contact <email address hidden>
>

--
Justin Pryzby
"Now seeking qualified employers"

References

[0]

[Debian Bug Importer (debzilla)]

Message-ID: <20050211134040.GA13435@andromeda>
Date: Fri, 11 Feb 2005 08:40:40 -0500
From: Justin Pryzby <email address hidden>
To: Micah Anderson <email address hidden>, <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

On Thu, Feb 10, 2005 at 11:55:49PM -0600, Micah Anderson wrote:
> A report on my findings thus far:
>
> I set up a pristine test environment to try and replicate the problem
> that Karl has been having. I have been able to setup a system running
> exim with the same configuration that Karl has, plugged into
> delivering mail through bogofilter, using the same bogofilter
> configuration as Karl. To setup the bogofilter databases, I fed
> approximately 2,000 ham messages and approximately 6,000 spam messages
> into bogofilter, creating database files in /etc/bogofilter with the
> same permissions as Exim.
>
> I then proceeded to send test mails. In the beginning bogofilter
> failed after about 20 messages due to database corruption. This was
> because the /etc/bogofilter directory was not set to have the correct
> ownership and permissions as was needed. After confirming with Karl
> how he had his setup, I changed it in the test scenario and restarted
> the tests. I ran tests that simulated mail delivery of approximately
> 1200 messages an hour (approximately 20/minute) for over 24 hours with
> no failures, I delivered approximately 35,000 messages with no
> problems at all.
>
> This is with exim 4.34-10 and bogofilter 0.93.5-1. It certainly seems
> that with a fresh installation of these versions everything works as
> expected.
>
> I am now going to try the same suite of tests with version 0.93.3 and
> then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
> have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
> down an older version of the .deb.
Well, normally there is
http://www.backports.org/debian/dists/woody/bogofilter/binary-i386/
but, it doesn't have any other versions.

I also tried Google:
http://www.google.com/search?hl=en&lr=&q=inurl%3Abogofilter_0.93.3&btnG=Search
but without success.

:-/

Justin

> On Thu, 10 Feb 2005, Clint Adams wrote:
>
> > > >Can you provide the configuration you use with exim to plug bogofilter
> > > >in? I am particularly interested because version 4.34-10 of exim
> > > >depends on libdb3 and exim 4.44-2 depends on libdb4.2 so I am curious
> > > >to know if exim uses libdb to interface with bogofilter in any way.
> > > >
> > > >micah
> > >
> > > I can use libdb - but I'm not using any db lists.
> >
> > That's an interesting question; maybe you should try the exim in
> > project/experimental, which is linked against libdb4.2.
>
>
> --
> To UNSUBSCRIBE, email to <email address hidden>
> with a subject of "unsubscribe". Trouble? Contact <email address hidden>
>

--
Justin Pryzby
"Now seeking qualified employers"

References

[0]

[Micah Anderson (micah-debian)]

On Fri, 11 Feb 2005, Justin Pryzby wrote:

> On Thu, Feb 10, 2005 at 11:55:49PM -0600, Micah Anderson wrote:

> > I am now going to try the same suite of tests with version 0.93.3 and
> > then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
> > have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
> > down an older version of the .deb.
> Well, normally there is
> http://www.backports.org/debian/dists/woody/bogofilter/binary-i386/
> but, it doesn't have any other versions.
>
> I also tried Google:
> http://www.google.com/search?hl=en&lr=&q=inurl%3Abogofilter_0.93.3&btnG=Search
> but without success.
>
I found snapshot.debian.org which has old versions of packages and the
tests are on their way.

micah

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Fri, 11 Feb 2005 08:53:21 -0600
From: Micah Anderson <email address hidden>
To: Justin Pryzby <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

On Fri, 11 Feb 2005, Justin Pryzby wrote:

> On Thu, Feb 10, 2005 at 11:55:49PM -0600, Micah Anderson wrote:

> > I am now going to try the same suite of tests with version 0.93.3 and
> > then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
> > have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
> > down an older version of the .deb.
> Well, normally there is
> http://www.backports.org/debian/dists/woody/bogofilter/binary-i386/
> but, it doesn't have any other versions.
>
> I also tried Google:
> http://www.google.com/search?hl=en&lr=&q=inurl%3Abogofilter_0.93.3&btnG=Search
> but without success.
>
I found snapshot.debian.org which has old versions of packages and the
tests are on their way.

micah

[Clint Adams (clint)]

> Well, normally there is
> http://www.backports.org/debian/dists/woody/bogofilter/binary-i386/
> but, it doesn't have any other versions.

Based on past reports, I would say that using any bogofilter debs from
backports.org is a surefire way to introduce corruption.

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sat, 12 Feb 2005 10:21:49 -0500
From: Clint Adams <email address hidden>
To: Justin Pryzby <email address hidden>,
 <email address hidden>
Cc: Micah Anderson <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

> Well, normally there is
> http://www.backports.org/debian/dists/woody/bogofilter/binary-i386/
> but, it doesn't have any other versions.

Based on past reports, I would say that using any bogofilter debs from
backports.org is a surefire way to introduce corruption.

[Micah Anderson (micah-debian)]

> >This is with exim 4.34-10 and bogofilter 0.93.5-1. It certainly seems
> >that with a fresh installation of these versions everything works as
> >expected.
> >
> >I am now going to try the same suite of tests with version 0.93.3 and
> >then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
> >have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
> >down an older version of the .deb.

I ran the same suite of tests with 0.93.0-1, ran some 46,000 emails
through the thing with no problems. I then upgraded to
bogofilter_0.93.3.1-1_i386.deb (the submitter sent me his .deb, so I
used that one), deleted and recreated the DB files before running
anything (as the bug submitter says he did). I ran another 30,000
emails through this version, doing some 3,000 per hour. No problems.

I will now move from 0.93.3 to 0.93.5 in the same manner.

However, it is increasingly clear that the chances for reproducing the
original submitter's problems are approaching impossible. My reason is
thus: 0.93.5 ran fine on a fresh install, with the same configuration
as the submitter, it did not encounter database corruption. If I am
destroying and rebuilding the database environment between 0.93.3 and
0.93.5 then I am doing nothing different than if I installed 0.93.5
fresh. The only difference is that I had the old version installed
before. It seems highly likely that a permissions problem was the
result of the corrupt DB. Especially in light of the fact that there
has been no other person who has reported the same problem.

micah

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sun, 13 Feb 2005 18:29:53 -0600
From: Micah Anderson <email address hidden>
To: <email address hidden>
Cc: Karl Schmidt <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

--mP3DRpeJDSE+ciuQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

> >This is with exim 4.34-10 and bogofilter 0.93.5-1. It certainly seems
> >that with a fresh installation of these versions everything works as
> >expected.
> >
> >I am now going to try the same suite of tests with version 0.93.3 and
> >then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
> >have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
> >down an older version of the .deb.

I ran the same suite of tests with 0.93.0-1, ran some 46,000 emails
through the thing with no problems. I then upgraded to
bogofilter_0.93.3.1-1_i386.deb (the submitter sent me his .deb, so I
used that one), deleted and recreated the DB files before running
anything (as the bug submitter says he did). I ran another 30,000
emails through this version, doing some 3,000 per hour. No problems.

I will now move from 0.93.3 to 0.93.5 in the same manner.

However, it is increasingly clear that the chances for reproducing the
original submitter's problems are approaching impossible. My reason is
thus: 0.93.5 ran fine on a fresh install, with the same configuration
as the submitter, it did not encounter database corruption. If I am
destroying and rebuilding the database environment between 0.93.3 and
0.93.5 then I am doing nothing different than if I installed 0.93.5
fresh. The only difference is that I had the old version installed
before. It seems highly likely that a permissions problem was the
result of the corrupt DB. Especially in light of the fact that there
has been no other person who has reported the same problem.

micah

--mP3DRpeJDSE+ciuQ
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFCD/EB9n4qXRzy1ioRAtmkAKCJctXB2SmjdmTSNhRruNIc6faG0ACffyF6
d7ghciGIPNiZV9kK8wvfqe4=
=Y8hU
-----END PGP SIGNATURE-----

--mP3DRpeJDSE+ciuQ--

[Karl Schmidt (karl-xtronics)]

>
> The only thing I can think of is that perhaps when you upgrade your
> exim is still delivering files and things get in a confused state. If
> you were actually removing and recreating your /etc/bogofilter
> directory contents, then you would of course need to stop exim in this
> process to keep this from happening.
>
> Micah

I'm pretty sure I turned exim off by hand before I ran the script - (I will add
it to my scrip when I test the next release).

Even if I failed to turn it off shouldn't the locking prevent such a problem? If
not - that would be a bug in my mind. Not a bad idea to test - I could see it
getting corrupted but working only to fail after some use.

I'm no expert on libdb, but most data bases support concurrency - I could see a
problem when creating the data base - but shouldn't one be able to lock it to
one instance for the creation phase?

More I think about this - I don't think exim could do that as the script I run
as root creates root owned db files that the Debian-exim user couldn't talk to.

But, the other way around would be possible - Where exim creates the db and then
the script adds to the db and possibly creates some log files with the root
ownership that the Debian-exim user can't see.

Checking some ownerships...<time passes> ...

The directory has drwxrwx--- 3 root Debian-exim

all the db files are -rw-r--r-- 1 Debian-exim Debian-exim

World readable -- owner only writable --

Let me test this right now. .. <more time passes> ..

With bogofilter 0.93.1-1
Ok deleting the db results in exims call of bogofilter creation of a db with:
-rw------- 1 Debian-exim Debian-exim

running:

bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004

Creates some log files with -rw-r--r-- 1 root root

Note the different owner AND permissions -- ok it is using the users create
mask. -- Hmm should that really be the case? Could this scramble the db??

now with bogfilter 0.93.5-1

results in the same thing

Ok, I've changed my db recreation script to:

#!/bin/bash
wajig stop exim4
# yes my db directory should really be in var some place - but it is in /etc
rm /etc/bogofilter/* -f
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2003
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/s-archived-spam
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2003
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2004
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/archived
chown Debian-exim.Debian-exim /etc/bogofilter/*
chmod go-r /etc/bogofilter/*
wajig start exim4

If this is the cause of the problem, it still leaves a question of how
bogofilter should handle running as a different user than the db files?

I wonder if the problem was exim running

"bogofilter -d /etc/bogofilter -l -p -e -u"

as db creation command-- this all assumes I forgot to stop exim two different
time (don't think so)?

As of now I'm running bogfilter 0.93.5-1 again and will let you know what
happens in a few days time.

-----------------------------------------------------...

[Micah Anderson (micah-debian)]

On Sun, 13 Feb 2005, Micah Anderson wrote:

>
> > >This is with exim 4.34-10 and bogofilter 0.93.5-1. It certainly seems
> > >that with a fresh installation of these versions everything works as
> > >expected.
> > >
> > >I am now going to try the same suite of tests with version 0.93.3 and
> > >then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
> > >have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
> > >down an older version of the .deb.
>
> I ran the same suite of tests with 0.93.0-1, ran some 46,000 emails
> through the thing with no problems. I then upgraded to
> bogofilter_0.93.3.1-1_i386.deb (the submitter sent me his .deb, so I
> used that one), deleted and recreated the DB files before running
> anything (as the bug submitter says he did). I ran another 30,000
> emails through this version, doing some 3,000 per hour. No problems.
>
> I will now move from 0.93.3 to 0.93.5 in the same manner.

This has completed, delivering ~38,000 emails, no problem at all.

I am going to tag this bug unreproducable, and we'll see what results
Karl has after his new tests.

Micah

[Micah Anderson (micah-debian)]

# Automatically generated email from bts, devscripts version 2.8.10
tags 293207 + unreproducible

[Micah Anderson (micah-debian)]

On Sun, 13 Feb 2005, Karl Schmidt wrote:

>
> >
> >The only thing I can think of is that perhaps when you upgrade your
> >exim is still delivering files and things get in a confused state. If
> >you were actually removing and recreating your /etc/bogofilter
> >directory contents, then you would of course need to stop exim in this
> >process to keep this from happening.
> >
> >Micah
>
> I'm pretty sure I turned exim off by hand before I ran the script - (I will
> add it to my scrip when I test the next release).
>
> Even if I failed to turn it off shouldn't the locking prevent such a
> problem? If not - that would be a bug in my mind. Not a bad idea to test -
> I could see it getting corrupted but working only to fail after some use.

No, because of permission problems. This is common with DB files,
especially with DB .log files. If the Debian-exim user suddenly cannot
read/write/execute one of the files in the database environment
because it is owned by the root user, then database corruption ensues.
This is not a problem with bogofilter at all, but with permissions on
your system. You can argue until you are blue in the face that it is a
problem with berkeley DB, but it is simply a permissions problem. If a
user who is not root does transactions to a database that this user
has access to, and then (due to the BDB configuration) a new
transaction log has to be created and the existing logfiles rotated,
there will be trouble if one of those .log files is owned by another
user other than the one doing the transaction. This is one of the most
common problems with Subversion using a BDB backend, setting up
permissions is paramount to keeping the DB from being corrupt.

> More I think about this - I don't think exim could do that as the script I
> run as root creates root owned db files that the Debian-exim user couldn't
> talk to.
> But, the other way around would be possible - Where exim creates the db and
> then the script adds to the db and possibly creates some log files with the
> root ownership that the Debian-exim user can't see.

This is exactly the problem. If your /etc/bogofilter directory has
files that are owned by root, and are not properly permissioned and
exim attempts to do transactions and cannot operate in the database
environment freely, you will get corruption. This is easily solved by
doing a db_recover, or simply making the permissions correct (a common
solution is to set the group sticky bit on the directory, or you could
run your bogofilter seeding as the Debian-exim user).

> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
>
> Creates some log files with -rw-r--r-- 1 root root
>
> Note the different owner AND permissions -- ok it is using the users create
> mask. -- Hmm should that really be the case? Could this scramble the db??

Absolutely... this is *the most common* cause of BDB database
corruption. If exim continues to run, it will soon want to do
operations on the database. It can do so many operations on the
database before a new transaction log is created, it'll be able to
read your root owned -rw-r--r-- files, but if it needs to write,
rename, move or anything to t...

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sun, 13 Feb 2005 21:26:01 -0600
From: Karl Schmidt <email address hidden>
To: Micah Anderson <email address hidden>, <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

>
> The only thing I can think of is that perhaps when you upgrade your
> exim is still delivering files and things get in a confused state. If
> you were actually removing and recreating your /etc/bogofilter
> directory contents, then you would of course need to stop exim in this
> process to keep this from happening.
>
> Micah

I'm pretty sure I turned exim off by hand before I ran the script - (I will add
it to my scrip when I test the next release).

Even if I failed to turn it off shouldn't the locking prevent such a problem? If
not - that would be a bug in my mind. Not a bad idea to test - I could see it
getting corrupted but working only to fail after some use.

I'm no expert on libdb, but most data bases support concurrency - I could see a
problem when creating the data base - but shouldn't one be able to lock it to
one instance for the creation phase?

More I think about this - I don't think exim could do that as the script I run
as root creates root owned db files that the Debian-exim user couldn't talk to.

But, the other way around would be possible - Where exim creates the db and then
the script adds to the db and possibly creates some log files with the root
ownership that the Debian-exim user can't see.

Checking some ownerships...<time passes> ...

The directory has drwxrwx--- 3 root Debian-exim

all the db files are -rw-r--r-- 1 Debian-exim Debian-exim

World readable -- owner only writable --

Let me test this right now. .. <more time passes> ..

With bogofilter 0.93.1-1
Ok deleting the db results in exims call of bogofilter creation of a db with:
-rw------- 1 Debian-exim Debian-exim

running:

bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004

Creates some log files with -rw-r--r-- 1 root root

Note the different owner AND permissions -- ok it is using the users create
mask. -- Hmm should that really be the case? Could this scramble the db??

now with bogfilter 0.93.5-1

results in the same thing

Ok, I've changed my db recreation script to:

#!/bin/bash
wajig stop exim4
# yes my db directory should really be in var some place - but it is in /etc
rm /etc/bogofilter/* -f
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2003
bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/s-archived-spam
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2003
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/z-archived2004
bogofilter -M -n -d /etc/bogofilter -I /home/karl/mail/archived
chown Debian-exim.Debian-exim /etc/bogofilter/*
chmod go-r /etc/bogofilter/*
wajig start exim4

If this is the cause of the problem, it still leaves a question of how
bogofilter should handle running as a different user than the db files?

I wonder if the problem was exim running

"bogofilter -d /etc/bogofilter -l -p -e -u"

as db...

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Mon, 14 Feb 2005 10:49:21 -0600
From: Micah Anderson <email address hidden>
To: Micah Anderson <email address hidden>
Cc: <email address hidden>, Karl Schmidt <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

On Sun, 13 Feb 2005, Micah Anderson wrote:

>
> > >This is with exim 4.34-10 and bogofilter 0.93.5-1. It certainly seems
> > >that with a fresh installation of these versions everything works as
> > >expected.
> > >
> > >I am now going to try the same suite of tests with version 0.93.3 and
> > >then attempt to upgrade to 0.93.5-1 as this is where Karl seemed to
> > >have a problem, moving from 0.93.3 to 0.93.5. I need to somehow track
> > >down an older version of the .deb.
>
> I ran the same suite of tests with 0.93.0-1, ran some 46,000 emails
> through the thing with no problems. I then upgraded to
> bogofilter_0.93.3.1-1_i386.deb (the submitter sent me his .deb, so I
> used that one), deleted and recreated the DB files before running
> anything (as the bug submitter says he did). I ran another 30,000
> emails through this version, doing some 3,000 per hour. No problems.
>
> I will now move from 0.93.3 to 0.93.5 in the same manner.

This has completed, delivering ~38,000 emails, no problem at all.

I am going to tag this bug unreproducable, and we'll see what results
Karl has after his new tests.

Micah

[Debian Bug Importer (debzilla)]

Message-Id: <20050214164940.E443B3AAEE@pond>
Date: Mon, 14 Feb 2005 10:49:40 -0600
From: Micah Anderson <email address hidden>
To: <email address hidden>
Subject: tagging 293207

# Automatically generated email from bts, devscripts version 2.8.10
tags 293207 + unreproducible

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Mon, 14 Feb 2005 11:10:16 -0600
From: Micah Anderson <email address hidden>
To: Karl Schmidt <email address hidden>
Cc: <email address hidden>, <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

On Sun, 13 Feb 2005, Karl Schmidt wrote:

>
> >
> >The only thing I can think of is that perhaps when you upgrade your
> >exim is still delivering files and things get in a confused state. If
> >you were actually removing and recreating your /etc/bogofilter
> >directory contents, then you would of course need to stop exim in this
> >process to keep this from happening.
> >
> >Micah
>
> I'm pretty sure I turned exim off by hand before I ran the script - (I will
> add it to my scrip when I test the next release).
>
> Even if I failed to turn it off shouldn't the locking prevent such a
> problem? If not - that would be a bug in my mind. Not a bad idea to test -
> I could see it getting corrupted but working only to fail after some use.

No, because of permission problems. This is common with DB files,
especially with DB .log files. If the Debian-exim user suddenly cannot
read/write/execute one of the files in the database environment
because it is owned by the root user, then database corruption ensues.
This is not a problem with bogofilter at all, but with permissions on
your system. You can argue until you are blue in the face that it is a
problem with berkeley DB, but it is simply a permissions problem. If a
user who is not root does transactions to a database that this user
has access to, and then (due to the BDB configuration) a new
transaction log has to be created and the existing logfiles rotated,
there will be trouble if one of those .log files is owned by another
user other than the one doing the transaction. This is one of the most
common problems with Subversion using a BDB backend, setting up
permissions is paramount to keeping the DB from being corrupt.

> More I think about this - I don't think exim could do that as the script I
> run as root creates root owned db files that the Debian-exim user couldn't
> talk to.
> But, the other way around would be possible - Where exim creates the db and
> then the script adds to the db and possibly creates some log files with the
> root ownership that the Debian-exim user can't see.

This is exactly the problem. If your /etc/bogofilter directory has
files that are owned by root, and are not properly permissioned and
exim attempts to do transactions and cannot operate in the database
environment freely, you will get corruption. This is easily solved by
doing a db_recover, or simply making the permissions correct (a common
solution is to set the group sticky bit on the directory, or you could
run your bogofilter seeding as the Debian-exim user).

> bogofilter -M -s -d /etc/bogofilter -I /home/karl/mail/zs-archived-spam2004
>
> Creates some log files with -rw-r--r-- 1 root root
>
> Note the different owner AND permissions -- ok it is using the users create
> mask. -- Hmm should that really be the case? Could this scramble the db??

Absolutely... this is *the most common* cause of BDB databa...

[Micah Anderson (micah-debian)]

# Automatically generated email from bts, devscripts version 2.8.10
severity 293207 important

[Micah Anderson (micah-debian)]

The severity for bugs that might be RC but aren't confirmed to exist
is 'important' -- this should not be grave any longer.

Micah

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Wed, 16 Feb 2005 14:07:36 -0600
From: Micah Anderson <email address hidden>
To: <email address hidden>
Subject: Adjusting severity

--AHVSF3we4xtO5oi5
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

The severity for bugs that might be RC but aren't confirmed to exist
is 'important' -- this should not be grave any longer.

Micah

--AHVSF3we4xtO5oi5
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFCE6gI9n4qXRzy1ioRAkr9AKCu2i2OCLgE2OuvOk+XuDlPuCkqGACgnPOI
0c5BNzQIE+eJPWW2UwootcI=
=dz8r
-----END PGP SIGNATURE-----

--AHVSF3we4xtO5oi5--

[Debian Bug Importer (debzilla)]

Message-Id: <20050216200621.7ABCF3AB28@pond>
Date: Wed, 16 Feb 2005 14:06:21 -0600
From: Micah Anderson <email address hidden>
To: <email address hidden>
Subject: severity of 293207 is important

# Automatically generated email from bts, devscripts version 2.8.10
severity 293207 important

[Micah Anderson (micah-debian)]

It has been several weeks now without any further information about
this bug report, have you been able to produce any more information?

Thanks!
micah

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Mon, 28 Feb 2005 11:44:26 -0600
From: Micah Anderson <email address hidden>
To: <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

It has been several weeks now without any further information about
this bug report, have you been able to produce any more information?

Thanks!
micah

[Karl Schmidt (karl-xtronics)]

Micah Anderson wrote:
> It has been several weeks now without any further information about
> this bug report, have you been able to produce any more information?
>
> Thanks!
> micah

It is working right now - my current belief is it was a permissions issue on the
log files that caused the corruption -- something that you don't consider a bug.

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 FAX [phone]

Age is an issue of mind over matter.
If you don't mind, it doesn't matter.
-- Mark Twain
----------------------------------------------------------------

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Mon, 28 Feb 2005 16:07:13 -0600
From: Karl Schmidt <email address hidden>
To: Micah Anderson <email address hidden>
Subject: Re: Bug#293207: bogofilter: last two versions caused db errors

Micah Anderson wrote:
> It has been several weeks now without any further information about
> this bug report, have you been able to produce any more information?
>
> Thanks!
> micah

It is working right now - my current belief is it was a permissions issue on the
log files that caused the corruption -- something that you don't consider a bug.

----------------------------------------------------------------
Karl Schmidt EMail <email address hidden>
Transtronics, Inc. WEB http://xtronics.com
3209 West 9th Street Ph [phone]
Lawrence, KS 66049 FAX [phone]

Age is an issue of mind over matter.
If you don't mind, it doesn't matter.
-- Mark Twain
----------------------------------------------------------------

[Adam Conrad (adconrad)]

This bug was user error.