libdb4.4 in gutsy breaks postgrey and subversion

[Matt LaPlante]

Binary package hint: libdb4.4

libdb4.4 (4.4.20-8.1ubuntu3) in gutsy breaks postgrey and subversion severely.

The main postgrey bug for this is bug 135038.

TEST CASE:
1. Install postfix, postgrey
2. Enable postgrey, e.g. by using "smtpd_recipient_restrictions = check_policy_service inet:127.0.0.1:60000"
3. Hack /usr/sbin/postgrey, so that it always does maintenance, with every connection:
  166,167c166,167
  < if($hour > 1 and $hour < 7 and
  < $now - $self->{postgrey}{last_maint_keys} >= 82800)
  ---
  > # if($hour > 1 and $hour < 7 and
  > # $now - $self->{postgrey}{last_maint_keys} >= 82800)
  331a332
  > $self->do_maintenance($now);
4. Restart postfix and postgrey
5. Send a mail to your postfix or use "telnet localhost 25" to do a manual session
6. You'll get the following in the /var/log/mail.info:
Nov 5 21:54:39 localhost postgrey[28451]: fatal: Can't call method "txn_commit" on an undefined value at /usr/sbin/postgrey line 223.
(and postgrey will exit/crash)

The main subversion bug is bug 154555.

TEST CASE:
 In Feisty create a repository:
 svnadmin create --fs-type bdb test-svn
 svn co file:///$PWD/test-svn foo-svn
 cd foo-svn/
 touch foo
 svn add foo
 svn ci -m 'bar'
Copy the repository to a Gutsy environment then and do:
 $ LANG=C svn co file:///$PWD/test-svn foo-svn
 svn: Unable to open an ra_local session to URL
 svn: Unable to open repository 'file:///home/daniel/test-svn'
 svn: Berkeley DB error for filesystem '/home/daniel/test-svn/db' while opening environment:
svn: Cannot allocate memory
 svn: bdb: unable to allocate memory for mutex; resize mutex region

Debian had a similar issue (in version 8.1 to 9), and determined that this was a problem with libdb4.4, as documented here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=441152

[Daniel Hahler (blueyed)]

I'm making this the metabug for this.
It seems to also affect subversion (bug 154555 and bug 139315).

The Debian bug is not really the same, because there it broke from -8.1 to -9, while in Ubuntu this happened with 8.1ubuntu2.

[Daniel Hahler (blueyed)]

I've tested the build on Gutsy and verified that it fixes the regressions for subversion and postgrey.

[Daniel Hahler (blueyed)]

Test builds are available in my PPA: https://edge.launchpad.net/~blueyed/+archive/+index?name_filter=db4.4

[Leonel Nunez (leonelnunez)]

Builded and installed fine the debdiff for gutsy

Testing on a production server no problem sofar

I'll give 24 hours test and let you know 9 hours running now ..

[Leonel Nunez (leonelnunez)]

more than 24 hours and no problems

[Matt LaPlante (cybrmatt)]

I can also confirm; tested Daniel's PPA version and it's run 48 hours without a crash. First time since the upgrade to gutsy that it's made it over 24 hrs.

[Daniel Hahler (blueyed)]

Regarding the SRU for Gutsy and its regression potential:
There are quite a few other packages, which depend on libdb4.4 (see "apt-cache rdepends libdb4.4").

Therefor the regression potential of this change seems quite high! But because I don't know what _really_ got changed/fixed here, it might instead also fix similar problems with those other packages.

[loafer (launchpad-net-iloaf)]

<apache>
+1
</apache>

Since this is fatal to a dependency would it qualify as a security update?

[Heitzso (heitzso)]

Just want to be emailed when this gets fixed.
I got bit by the postgrey variation when I upgraded my email server to gutsy this weekend.
I'm also surprised that this bug, that breaks postfix (and svn), was reported back in August
and is still around.

[Matt LaPlante (cybrmatt)]

Well folks, I've been posting to ubuntu-motu and more recently ubuntu-devel-discuss trying to get some attention for this issue. Clearly it hasn't helped much (except for Daniel building the PPA package for us). I'm really dumbfounded that such a normally progressive distro like Ubuntu is turning a blind eye to high priority breakage. In most cases I wouldn't even take reporting problems out of the bug tracker to mailing lists, but I'm out of ideas. Maybe it's time to consider moving my mail systems back to Debian...?

[Martin Pitt (pitti)]

For Hardy I'd just like to sync from Debian. This will drop the NTPL change and the libdb4.4-java-gcj package, which is not used by anything anyway. I wait for doko's confirmation until I do that.

[Martin Pitt (pitti)]

Package upload sponsored and accepted into gutsy-proposed. Please test the packages there and give feedback here. Thank you!

[Martin Pitt (pitti)]

Matthias confirmed, I synced db4.4 from unstable to Hardy.

[Martijn Heemels (yggdrasil)]

Thanks Martin,

To test the new package I just updated libdb4.4 (only that package) from the gutsy-proposed main repository. I got version 4.4.20-8.1ubuntu3.1.

The Postfix/Postgrey testcase as stated in the Bug Description no longer gives an error. I'll let you know tomorrow if Postgrey's nightly maintenance also goes as it should.

I don't use Subversion so would appreciate it if someone else could test that program. See testcase in Bug Description. Verify if testcase works before upgrading, then upgrade libdb4.4 and test again.

Also, if some of the other subscribers to this bug would like to test it, please do.

[Martijn Heemels (yggdrasil)]

As I said yesterday, I would let you know if the proposed libdb4.4 package fixes the postfix/postgrey problems for Gutsy. Well, since I've had no more problems with postgrey last night I would say the new package works fine for me. No more errors in my logs, and no more crashing postgrey process.

[Jeff Balderson (jbalders)]

I was experiencing the same problem, so I installed Daniel Haler's PPA version and about two weeks ago. It cured my immediate ills and had been running ever since without problems.

Last night, I installed the version released into gutsy-proposed. My initial impression is that it also seems to fix the problem and not cause any problems with any other packages I use, although my testing is minimal at this point. I'll report again if I find the postgrey problem return or other issues.

[Matt LaPlante (cybrmatt)]

Martin, thanks very much for pushing forward with this fix.

My comments are pretty much identical to Jeff Balderson; upgraded from Daniel Haler's release to the gutsy-proposed yesterday and have witnessed no new or recurring problems since.

[loafer (launchpad-net-iloaf)]

Confirmed - this proposed release fixes the issue.

I would like to point out that this was a critical failure in a somewhat popular (and a core addon package for postfix) ( Bug #135038, first reported on 2007-08-27 ) and should have been addressed with a little more urgency, and I've just noticed this appears to have been duplicated in the wrong direction, rather than having subversion added to the original.

That said, Thank you for addressing it swiftly Martin.

[Ronald Hummelink (ronald-hummelink)]

Another positive tets report....

I've been running on a version from debian/lenny for a while and 2 days ago 'downgraded' to the version in ubuntu/gutsy-proposed.

Both seem to do their job just fine.

[Matt LaPlante (cybrmatt)]

Update... One of my two machines running 4.4.20-8.1ubuntu3.1 from gutsy-proposed had another postgrey crash last night. This is the first crash since installing Daniel's db4.4. It would seem that either the proposed package does not fix the issue as well as hoped, or that I've somehow made an error or omission when installing this version.

Dec 1 02:21:31 mail3 postgrey[5458]: cleaning up old logs...
Dec 1 02:21:31 mail3 postgrey[5458]: cleaning up old entries...
Dec 1 02:21:32 mail3 postgrey[5458]: cleaning main database finished. before: 1992, after: 1195
Dec 1 02:21:32 mail3 postgrey[5458]: fatal: Can't call method "txn_commit" on an undefined value at /usr/sbin/postgrey line 223.

My other server, also running this release, has not experienced the same problems, but it's also under much less load so perhaps it didn't trigger the same postgrey routines.

[Matt LaPlante (cybrmatt)]

Ok, sorry for the noise. I think my previous entry may have been a false alarm. It's my theory that, although at the time of the crash the fixed libdb was installed, postgrey may not have been restarted following the upgrade, and may therefore have still had the faulty library loaded. Since the crash and subsequent restart, things have gone smoothly, including a successful cleaning run last night:

Dec 2 02:01:12 mail3 postgrey[24320]: cleaning up old logs...
Dec 2 02:01:13 mail3 postgrey[24320]: cleaning up old entries...
Dec 2 02:01:45 mail3 postgrey[24320]: cleaning main database finished. before: 2273, after: 1316
Dec 2 02:01:47 mail3 postgrey[24320]: cleaning clients database finished. before: 966, after: 725

It's my recommendation to proceed with releasing the fixed version. If anything new should arise, I'll of course report it.

[KM (ubuntubug-acrasis)]

Looks good here. I started postgrey about 60 hours ago, after installing libdb4.4.20-8.1ubuntu3.1. No trouble so far.

[Paul Bailey (paul-m-bailey)]

So where can we this new deb? Or when is it going to be released in the repositories? I could use the fix.

[KM (ubuntubug-acrasis)]

Add this to your sources.list:

   deb http://us.archive.ubuntu.com/ubuntu gutsy-proposed main

and apt-get update && apt-get install libdb4.4. At least, that worked for me.

[Matt LaPlante (cybrmatt)]

If you want to avoid installing everything out of gutsy-proposed when doing an apt-upgrade, create the file /etc/apt/preferences with something like the following:

Package: libdb4.4
Pin: release a=gutsy-proposed
Pin-Priority: 600

Package: *
Pin: release a=gutsy-proposed
Pin-Priority: 400

[Paul Bailey (paul-m-bailey)]

Thanks for the Repo tip. I've installed the new package, now it's time to hurry up and wait.

[Martin Pitt (pitti)]

Thanks to everybody for testing postgrey, this seems to be a rock solid verification.

Can anyone do the test of subversion? If this goes well, too, I'll move the package to -updates at Thursday (when the 7 day aging period is over).

[Brian J. Murrell (brian-interlinx)]

On Thu, 2007-11-29 at 08:24 +0000, Martin Pitt wrote:
> Package upload sponsored and accepted into gutsy-proposed. Please test
> the packages there and give feedback here. Thank you!
>
> ** Changed in: db4.4 (Ubuntu Gutsy)
> Status: Triaged => In Progress
>
> ** Changed in: db4.4 (Ubuntu Gutsy)
> Status: In Progress => Fix Committed

Is this not supposed to migrate to gutsy proper by now? I thought
packages in proposed move to main after 7 days. Maybe I am wrong.

b.

[Brian J. Murrell (brian-interlinx)]

On Tue, 2007-12-11 at 12:57 +0000, Brian J. Murrell wrote:
> On Thu, 2007-11-29 at 08:24 +0000, Martin Pitt wrote:
> > Package upload sponsored and accepted into gutsy-proposed. Please test
> > the packages there and give feedback here. Thank you!
> >
> > ** Changed in: db4.4 (Ubuntu Gutsy)
> > Status: Triaged => In Progress
> >
> > ** Changed in: db4.4 (Ubuntu Gutsy)
> > Status: In Progress => Fix Committed
>
> Is this not supposed to migrate to gutsy proper by now? I thought
> packages in proposed move to main after 7 days. Maybe I am wrong.

And still, 3 days later, no response to my query and it's *still* not in
gutsy main. What has to happen to get this update moved from proposed
to main?

b.

[Martin Eisenhardt (martin-eisenhardt)]

Brian: It might just be that Martin Pitt waits for people giving the requested feedback for this patch.

Well, I might as well go in first: Installed the package from gutsy-proposed, seems to work fine for me with postgrey though I cannot comment on subversion (who uses bdb with subversion, anyway? ;-) )

[Brian J. Murrell (brian-interlinx)]

On Fri, 2007-12-14 at 12:36 +0000, Martin Eisenhardt wrote:
> Brian: It might just be that Martin Pitt waits for people giving the
> requested feedback for this patch.

There has been lots of positive feedback over the last few weeks, and
apparently the policy is that if a package sits in proposed for 7 days
with no complaints it gets pushed to main/updates.

b.

[Christian Holtje (docwhat)]

Here is some feedback on libdb4.4_4.4.20-8.1ubuntu3_i386.deb from proposed:
  It has been running inside postgrey, successfully, since Dec 6th (8 days). Postgrey hasn't crashed and hasn't stopped responding. I haven't had any reports of any processes breaking because of this upgrade. Since the longest my automated tasks run is one week, I feel confident saying that this works.

Ciao!

[Mouse (a-kachalov)]

Not work for me:
$ svnadmin create --fs-type bdb $PWD/test
$ svnadmin dump $PWD/test
Segmentation fault (core dumped)

$ strace -fF svnadmin dump $PWD/test
open("/tmp/test/db/lock-tokens", O_RDWR|O_LARGEFILE) = 15
fcntl64(15, F_SETFD, FD_CLOEXEC) = 0
read(15, "\0\0\0\0\1\0\0\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0\0"..., 512) = 512
close(15) = 0
stat64("/tmp/test/db/lock-tokens", {st_mode=S_IFREG|0644, st_size=8192, ...}) = 0
open("/tmp/test/db/lock-tokens", O_RDWR|O_LARGEFILE) = 15
fcntl64(15, F_SETFD, FD_CLOEXEC) = 0
fstat64(15, {st_mode=S_IFREG|0644, st_size=8192, ...}) = 0
time(NULL) = 1197650286
open("/tmp/test/db/format", O_RDONLY|O_LARGEFILE) = 16
read(16, "2\n", 80) = 2
close(16) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV (core dumped) +++

GDB says:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1216657744 (LWP 28205)]
0x00000000 in ?? ()

Pkgs:
subversion = 1.4.4dfsg1-1ubuntu3
libdb4.4 = 4.4.20-8.1ubuntu3.1

[Mouse (a-kachalov)]

...
I install package from http://ru.archive.ubuntu.com/ubuntu/dists/gutsy-proposed/main/binary-i386/
There is 4.4.20-8.1ubuntu3.1 version of libdb4.4

[Daniel Hahler (blueyed)]

Mouse, this appears to be another bug. While I cannot reproduce it in Feisty, I get a segfault in Gutsy, too. Please report a separate bug about it.

I'm going to attach a test-svn repository created on a feisty system. Please verify, if the libdb4.4 from -proposed fixes the issue for subversion.

[Mouse (a-kachalov)]

Dumping is ok:
$ svnadmin dump test-svn|wc -l
* Dumped revision 0.
* Dumped revision 1.
43

But, listing were failed:
$ svn ls svn://localhost/foo
foo
$ svn ls svn://localhost/foo/foo
svn: Connection closed unexpectedly
At this point, server says:
[pid 28510] read(4, "( stat ( 0: ( 1 ) ) ) ", 4096) = 22
[pid 28510] time(NULL) = 1197720554
[pid 28510] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 28510 detached
<... accept resumed> 0x80672e0, [16]) = ? ERESTARTSYS (To be restarted)
--- SIGCHLD (Child exited) @ 0 (0) ---
sigreturn() = ? (mask now [])
waitpid(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV && WCOREDUMP(s)}], WNOHANG|WSTOPPED) = 28510
waitpid(-1, 0xbfde4bf8, WNOHANG|WSTOPPED) = -1 ECHILD (No child processes)
accept(3,

There should be:
$ svn ls svn://localhost/foo/foo
svn: Failure opening '/foo/foo'
svn: '/foo' is not a directory in filesystem '/home/mouse/tmp/test--svn/db'

[Mouse (a-kachalov)]

"Should be" within SVN of type FSFS instead of BDB.
I mean, that test-svn is BDB and test--svn is FSFS.

[Ian McCall (ian-eruvia)]

Would it be possible to split these into two separate bugs? If the postgrey issue is fixed I am -very- keen to see a release, my server is currently processing literally thousands more spam mails per hour as a result of having to disable postgrey.

I'm very keen to see this progressed - if it's imperfect due to the svn issue that's fine, as svn is no worse off but postgrey is much better off. Overall improvement.

Cheers,
Ian

[Spamfree (ubuntu-netryx)]

Bump.

This bug remains despite being months old now and despite being fixed in upstream Debian for months. The present status of "Fix Committed" is now over two weeks old and it has still not been released.

Is anyone working on this?

[jbowen (john-bowenweb)]

I'm just piling on here. I followed KM's tip above (https://bugs.launchpad.net/ubuntu/+source/db4.4/+bug/153996/comments/26) and it seems to have resolved the issue - going on 4 days now without the mail server (Postgrey) crashing.

[Martin Pitt (pitti)]

Thanks to all for testing, there seems to be sufficient feedback now. I copied it to -updates.

[Martin Eisenhardt (martin-eisenhardt)]

Thanks for the fix, but the problem with Subversion and BDB repositories is still around, at least on my systems. Could you have a look at that use of libdb4.4, too? Thanks!

[Martin Eisenhardt (martin-eisenhardt)]

To clarify things a bit:

The problem with subversion and libdb4.4 still persists, as you can see from the following:

<cut>
$ svnadmin create --fs-type bdb /home/mneisen/tmp/test-repo
$ svn checkout file:///home/mneisen/tmp/test-repo/ working-copy
Segmentation fault (core dumped)

$ sudo dpkg -l | grep ^ii | grep libdb4.4
ii libdb4.4 4.4.20-8.1ubuntu3.1 Berkeley v4.4 Database Libraries [runtime]

$ sudo dpkg -l | grep ^ii | grep subversion
ii subversion 1.4.4dfsg1-1ubuntu3 Advanced version control system
ii subversion-helper-scripts 0.5-2 various tools to help with the use and maint
ii subversion-tools 1.4.4dfsg1-1ubuntu3 Assorted tools related to Subversion
</cut>

So, it would be nice if libdb4.4 would be fixed so that subversion can be used with BDB repositories. Thanks!

[Ronald Hummelink (ronald-hummelink)]

Postgrey has worked flawlessly for me since installing libdb4.4 from debian/lenny at the time, and the version from gusty-proposed (now in gutsy-updates) when it became available.

Confirming Martin Eisenhardt's problem:

$ sudo apt-get install libdb4.4/gutsy-updates
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version 4.4.20-8.1ubuntu3.1 (Ubuntu:7.10/gutsy-updates) for libdb4.4
libdb4.4 is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

$ svnadmin create --fs-type bdb ~/tmp/test-repo
$ svn checkout file:///home/ronald/tmp/test-repo/ working-copy
Segmentation fault

[Daniel Hahler (blueyed)]

I've just reported bug 182974 about the issue which the new subversion issue.
Please refer to this bug report for the subversion segfault.
Unfortunately, I've not found a solution - so subversion with bdb still regresses in Gutsy - only in a different way now.
If it was me, I would switch to fs-type "fsfs" anyway.

[Ian Justman (ianj)]

I appear to be completely up to date on Postgrey and its dependencies as well (running libdb4.4 4.4.20-8.1ubuntu3.1), and as of right now, it is STILL broken as evidenced by the following:

Mar 16 02:22:36 haruhi postgrey[4042]: cleaning up old logs...
Mar 16 02:22:36 haruhi postgrey[4042]: cleaning up old entries...
Mar 16 02:22:37 haruhi postgrey[4042]: cleaning main database finished. before: 6394, after: 1424
Mar 16 02:22:37 haruhi postgrey[4042]: fatal: Can't call method "txn_commit" on an undefined value at /usr/sbin/postgrey line 223.

I just reenabled Postgrey on this machine after not using it for a while (the process had been running the whole time, but I just started using it again on Saturday, March 15, 2008). My users are complaining about mounting spam. About the only way I can fix this is a cronned restart of postgrey every night after its scheduled cleanup time, which is not an acceptable solution.

Thank you for your assistance with this matter.

--Ian.

[Rick Mach (rmach)]

After the fix, postgrey worked reliability for me. Here is my versions:

postgrey 1.27-4
libdb4.4 4.4.20-8.1ubuntu3.1

[Daniel Hahler (blueyed)]

Ian, you may want to make sure that the service gets restarted (stopped and started), not reloaded.
I can imagine that it still has the old/broken libraries loaded. See the comment from Matt above (https://bugs.launchpad.net/ubuntu/+source/db4.4/+bug/153996/comments/23)

[Ian Justman (ianj)]

Ordinarily, I would do that anyway (restart, not reload), but since Postgrey sit there not being used for a while, I missed that little detail.

In any event, I had migrated to the Debian version temporarily (which worked pretty well), but upon receiving this bit of advice, I went back to 4.4..20-8.1ubuntu3.1 and did the restart. That did the trick.

I feel minorly like a fool, but eh, I guess it happens to all of us. :)

Thanks!

--Ian.