slapd: Index and Data corruption with openldap2 2.1.30

[Debian Bug Importer]

Automatically imported from Debian bug report #291408 http://bugs.debian.org/291408

[Debian Bug Importer (debzilla)]

Automatically imported from Debian bug report #291408 http://bugs.debian.org/291408

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Thu, 20 Jan 2005 17:16:48 +0100
From: Matthijs Mohlmann <email address hidden>
To: Debian Bug Tracking System <email address hidden>
Subject: slapd: Index and Data corruption with openldap2 2.1.30

Package: slapd
Version: 2.1.30-3
Severity: critical
Justification: causes serious data loss

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

We are running debian sarge on some of our servers now and are getting
data and index corruption (every few weeks). On different servers
(configured the same) we experience those problems. I think it is
openldap that causes this problem because openldap 2.2 doesn't have this
problems with the same bdb backend. (db4.2)

We only use this option in DB_CONFIG but i think it doesn't have
anything to do with the problem:
set_cachesize 0 52428800 0

I've searched the logs for any inconsistency but i couldn't find anything
in the logs.

Another little problem i experienced is the bad performance with 2.1.30
i've benched 2.1.30 and 2.2.20 a little bit and i got this:
On the server (openldap 2.1.30):
IBM eServer series: 1 Ghz with 1.5 Gb RAM (Kernel: 2.6.8)
www0:~# time id user_from_ldap
uid=1003(user_from_ldap) gid=1003(user_from_ldap) groups=1003(user_from_ldap)

real 0m1.588s
user 0m0.059s
sys 0m0.017s

the real is between the 1.4 to 1.8 seconds.

Desktop system (openldap 2.2.20)
Pentium 2 350Mhz with 192Mb RAM (Kernel: 2.6.8)
desktop:~# time id user_from_ldap

real 0m0.388s
user 0m0.059s
sys 0m0.017s

The real on this one is between 0.2 and 0.4 seconds.

On both machines ldap is running on localhost.

I've now a few weeks openldap 2.2 running on my desktop system and i
didn't get any data or index corruption and it's fast as hell.

I've seen that the main blocker is the gnutls patch. Well i ported the
patch almost to 2.2.20. There are a few things i have to look for. And
then i hope it will work. I don't know how far you are with that patch
but i hope, really hope that 2.2.20 can make it into Sarge.

If you need more information please mail me.

Regards,

Matthijs Mohlmann

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFB79lw2n1ROIkXqbARAnC9AJ9t2O8g+r4l6NAxLYXAnSKgTTy8OQCgsdPd
e4hE3GCHUOI+QWxD6aU8FGc=
=FRFn
-----END PGP SIGNATURE-----

[Stephen Frost (sfrost)]

* Matthijs Mohlmann (<email address hidden>) wrote:
> We are running debian sarge on some of our servers now and are getting
> data and index corruption (every few weeks). On different servers
> (configured the same) we experience those problems. I think it is
> openldap that causes this problem because openldap 2.2 doesn't have this
> problems with the same bdb backend. (db4.2)

Can you try setting "LD_ASSUME_KERNEL=2.4.1" in the startup scripts and
see if that helps?

 Thanks,

  Stephen

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Thu, 20 Jan 2005 18:36:23 -0500
From: Stephen Frost <email address hidden>
To: Matthijs Mohlmann <email address hidden>, <email address hidden>
Subject: Re: [debian-openldap] Bug#291408: slapd: Index and Data corruption with openldap2 2.1.30

--rNhM/seQCZduifTx
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

* Matthijs Mohlmann (<email address hidden>) wrote:
> We are running debian sarge on some of our servers now and are getting
> data and index corruption (every few weeks). On different servers
> (configured the same) we experience those problems. I think it is
> openldap that causes this problem because openldap 2.2 doesn't have this
> problems with the same bdb backend. (db4.2)

Can you try setting "LD_ASSUME_KERNEL=2.4.1" in the startup scripts and
see if that helps?

 Thanks,

  Stephen

--rNhM/seQCZduifTx
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFB8EB3rzgMPqB3kigRAp7YAJ9/D+ASOqcYiY1aFReOLPWEVZl6zQCfaR1e
D0Bh/42jwCbKgL71QWHqGT4=
=NgDz
-----END PGP SIGNATURE-----

--rNhM/seQCZduifTx--

[Matthijs Mohlmann (matthijs)]

Stephen Frost wrote:
> * Matthijs Mohlmann (<email address hidden>) wrote:
>
>>We are running debian sarge on some of our servers now and are getting
>>data and index corruption (every few weeks). On different servers
>>(configured the same) we experience those problems. I think it is
>>openldap that causes this problem because openldap 2.2 doesn't have this
>>problems with the same bdb backend. (db4.2)
>
>
> Can you try setting "LD_ASSUME_KERNEL=2.4.1" in the startup scripts and
> see if that helps?
>
> Thanks,
>
> Stephen

This is a workaround i think, but i'll check if it works. (The
workaround for data corruption is: db4.2_recover) And for index
corruption i need to replace a backup.

Matthijs Mohlmann

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Fri, 21 Jan 2005 08:58:42 +0100
From: Matthijs Mohlmann <email address hidden>
To: Stephen Frost <email address hidden>
Cc: <email address hidden>
Subject: Re: [debian-openldap] Bug#291408: slapd: Index and Data corruption
 with openldap2 2.1.30

Stephen Frost wrote:
> * Matthijs Mohlmann (<email address hidden>) wrote:
>
>>We are running debian sarge on some of our servers now and are getting
>>data and index corruption (every few weeks). On different servers
>>(configured the same) we experience those problems. I think it is
>>openldap that causes this problem because openldap 2.2 doesn't have this
>>problems with the same bdb backend. (db4.2)
>
>
> Can you try setting "LD_ASSUME_KERNEL=2.4.1" in the startup scripts and
> see if that helps?
>
> Thanks,
>
> Stephen

This is a workaround i think, but i'll check if it works. (The
workaround for data corruption is: db4.2_recover) And for index
corruption i need to replace a backup.

Matthijs Mohlmann

[Matthijs Mohlmann (matthijs)]

Stephen Frost wrote:
> * Matthijs Mohlmann (<email address hidden>) wrote:
>
>>We are running debian sarge on some of our servers now and are getting
>>data and index corruption (every few weeks). On different servers
>>(configured the same) we experience those problems. I think it is
>>openldap that causes this problem because openldap 2.2 doesn't have this
>>problems with the same bdb backend. (db4.2)
>
>
> Can you try setting "LD_ASSUME_KERNEL=2.4.1" in the startup scripts and
> see if that helps?
>
> Thanks,
>
> Stephen

Ok,

This doesn't help, we could also reproduce it with a backport of 2.1.30
on woody systems, reproduced on sarge with the same version. And got the
same results, index and data corruption.

Look also at this bug: #278329
(http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=278329)

Matthijs Mohlmann

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Fri, 21 Jan 2005 12:41:00 +0100
From: Matthijs Mohlmann <email address hidden>
To: Stephen Frost <email address hidden>
Cc: <email address hidden>
Subject: Re: [debian-openldap] Bug#291408: slapd: Index and Data corruption
 with openldap2 2.1.30

Stephen Frost wrote:
> * Matthijs Mohlmann (<email address hidden>) wrote:
>
>>We are running debian sarge on some of our servers now and are getting
>>data and index corruption (every few weeks). On different servers
>>(configured the same) we experience those problems. I think it is
>>openldap that causes this problem because openldap 2.2 doesn't have this
>>problems with the same bdb backend. (db4.2)
>
>
> Can you try setting "LD_ASSUME_KERNEL=2.4.1" in the startup scripts and
> see if that helps?
>
> Thanks,
>
> Stephen

Ok,

This doesn't help, we could also reproduce it with a backport of 2.1.30
on woody systems, reproduced on sarge with the same version. And got the
same results, index and data corruption.

Look also at this bug: #278329
(http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=278329)

Matthijs Mohlmann

[Torsten Landschoff (torsten)]

Hi there,

Please help testing the new OpenLDAP 2.2 packages at

 http://pkg-openldap.alioth.debian.org/

I'd like to know if they fix the index and data corruption problems.

Thanks

 Torsten

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Fri, 25 Feb 2005 16:22:47 +0100
From: Torsten Landschoff <email address hidden>
To: <email address hidden>
Subject: OpenLDAP 2.2 packages

--dDRMvlgZJXvWKvBx
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi there,=20

Please help testing the new OpenLDAP 2.2 packages at

 http://pkg-openldap.alioth.debian.org/

I'd like to know if they fix the index and data corruption problems.

Thanks

 Torsten

--dDRMvlgZJXvWKvBx
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCH0LHdQgHtVUb5EcRAgh2AJ4xVbqsjaX5FjqPz0xACvO+prtokgCfZI/4
jENElOveN430HqcSntgibAE=
=lVUn
-----END PGP SIGNATURE-----

--dDRMvlgZJXvWKvBx--

[Matthijs Mohlmann (matthijs)]

> Hi there,
>
> Please help testing the new OpenLDAP 2.2 packages at
>
> http://pkg-openldap.alioth.debian.org/
>
> I'd like to know if they fix the index and data corruption problems.
>
> Thanks
>
> Torsten
>
I will directly test it after i'm back from fosdem.

Thank you for response.

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sat, 26 Feb 2005 21:09:29 +0100 (CET)
From: <email address hidden>
To: "Torsten Landschoff" <email address hidden>,
 <email address hidden>
Subject: Re: Bug#291408: OpenLDAP 2.2 packages

> Hi there,
>
> Please help testing the new OpenLDAP 2.2 packages at
>
> http://pkg-openldap.alioth.debian.org/
>
> I'd like to know if they fix the index and data corruption problems.
>
> Thanks
>
> Torsten
>
I will directly test it after i'm back from fosdem.

Thank you for response.

[Torsten Landschoff (torsten)]

Hi Matthijs,

On Tue, Mar 01, 2005 at 10:51:17AM +0100, Matthijs Mohlmann wrote:
> Ok i'm testing now. I'll wait for a week or something because the
> data/index corruption appears after a couple of days. And when it went
> all ok i'll test it on another more heavy server (more load on the LDAP
> database) i am still a little bit careful.

Very good!

> I found a bug in the existing openldap2.2 in experimental. It doesn't
> have a build dependency on libperl-dev (2.2.23-0.pre3) (I think
> experimental isn't updated yet)

I uploaded pre4 which should fix that for real. Somehow I messed up in
pre3 which was supposed to be fixed already.

> There still isn't a upgrade path for the config file slapd.conf. (ACL
> stuff is somewhat changed) Probably you are already working on this.

Thanks for the reminder. I still have to do a woody->sid upgrade with
the new packages.

> In two weeks you get the final results.

Great.

Thank you very much

 Torsten

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 1 Mar 2005 12:11:59 +0100
From: Torsten Landschoff <email address hidden>
To: Matthijs Mohlmann <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#291408: OpenLDAP 2.2 packages

--9amGYk9869ThD9tj
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi Matthijs,=20

On Tue, Mar 01, 2005 at 10:51:17AM +0100, Matthijs Mohlmann wrote:
> Ok i'm testing now. I'll wait for a week or something because the
> data/index corruption appears after a couple of days. And when it went
> all ok i'll test it on another more heavy server (more load on the LDAP
> database) i am still a little bit careful.

Very good!

> I found a bug in the existing openldap2.2 in experimental. It doesn't
> have a build dependency on libperl-dev (2.2.23-0.pre3) (I think
> experimental isn't updated yet)

I uploaded pre4 which should fix that for real. Somehow I messed up in
pre3 which was supposed to be fixed already.

> There still isn't a upgrade path for the config file slapd.conf. (ACL
> stuff is somewhat changed) Probably you are already working on this.

Thanks for the reminder. I still have to do a woody->sid upgrade with
the new packages.

> In two weeks you get the final results.

Great.=20

Thank you very much

 Torsten

--9amGYk9869ThD9tj
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCJE3/dQgHtVUb5EcRAqCKAJ9dOGCRBrQRK1SLWCmbKBiy7XPjKACdFkr7
//jaCSRHH+k5lFmXfOsmkhM=
=VScU
-----END PGP SIGNATURE-----

--9amGYk9869ThD9tj--

[Torsten Landschoff (t-landschoff)]

retitle 291408 [fixed in 2.2.23] database corruption

[Torsten Landschoff (t-landschoff)]

tags 291408 fixed-in-experimental

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Wed, 16 Mar 2005 13:23:19 +0100 (CET)
From: <email address hidden> (Torsten Landschoff)
To: <email address hidden>
Subject: retitle 291408 to [fixed in 2.2.23] database corruption

retitle 291408 [fixed in 2.2.23] database corruption

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Wed, 16 Mar 2005 13:23:30 +0100 (CET)
From: <email address hidden> (Torsten Landschoff)
To: <email address hidden>
Subject: tagging 291408

tags 291408 fixed-in-experimental

[Gerald Hochegger (gerald-hochegger)]

Hi Torsten,

thanks for your 2.2 packages.

I've already upgraded our production servers because I was
hoping to get rid of the database corruption problems.

Here are my results so far:

* ldapi:/// is not working after the upgrade
 Use
 --localstatedir=/var (instead of /var/run)
 in debian/configure.options

 slapd tries to create the socket in /var/run/run/ldapi - which fails.
 (We are using heimdal with openldap backend and need ldapi:///)

 Furthermore the socket should be 600 - but I don't know how to set this.

* DB_CONFIG gets lost on upgrades
 /var/lib/ldap/DB_CONFIG gets lost on upgrades, which makes
 reimporting the Ldap database with slapadd very slow with big ldif files.
 If possible save the DB_CONFIG on upgrades.

* slapd with db4.3 has indeed problems.
 My database corruption problems were still present with pre4 - I upgraded
 to pre6 on 2005-04-02. Let's hope db4.2 fixes the database corruption.

* If possible upgrade to openldap 2.2.24
 I've applied your patches to 2.2.24 - no problems so far.

Greetings,
Gerald

--
---------------------------------------------------------------------------
Gerald Hochegger Internet: <email address hidden>
Computer Center Voice : (++43 463) 2700/9613
University of Klagenfurt Fax : (++43 463) 2700/9699
Universitaetsstrasse 65-67
A-9020 Klagenfurt, Austria, Europe
---------------------------------------------------------------------------

[Torsten Landschoff (torsten)]

Hi Gerald,

On Sun, Apr 03, 2005 at 05:36:59PM +0200, Gerald Hochegger wrote:
> I've already upgraded our production servers because I was
> hoping to get rid of the database corruption problems.
>
> Here are my results so far:
>
> * ldapi:/// is not working after the upgrade
> Use
> --localstatedir=/var (instead of /var/run)
> in debian/configure.options

Checked in for 2.2.23-2. I really wonder why upstream has changed the
meaning of this option.

> slapd tries to create the socket in /var/run/run/ldapi - which fails.
> (We are using heimdal with openldap backend and need ldapi:///)

Interesting. Did not know it was really /needed/ for anything.

> Furthermore the socket should be 600 - but I don't know how to set this.

That should work by using
 slapd -h 'ldapi:///???x-mod=600'

> * DB_CONFIG gets lost on upgrades
> /var/lib/ldap/DB_CONFIG gets lost on upgrades, which makes
> reimporting the Ldap database with slapadd very slow with big ldif files.
> If possible save the DB_CONFIG on upgrades.

Known issue, it's on the TODO list.

> * slapd with db4.3 has indeed problems.
> My database corruption problems were still present with pre4 - I upgraded
> to pre6 on 2005-04-02. Let's hope db4.2 fixes the database corruption.

Interesting. I really hope db4.2 fixes those problems.

> * If possible upgrade to openldap 2.2.24
> I've applied your patches to 2.2.24 - no problems so far.

Upgrade will be done after the packaging is stable.

Greetings

 Torsten

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sun, 3 Apr 2005 17:36:59 +0200
From: Gerald Hochegger <email address hidden>
To: Torsten Landschoff <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#291408: OpenLDAP 2.2 packages

Hi Torsten,

thanks for your 2.2 packages.

I've already upgraded our production servers because I was
hoping to get rid of the database corruption problems.

Here are my results so far:

* ldapi:/// is not working after the upgrade
 Use
 --localstatedir=/var (instead of /var/run)
 in debian/configure.options

 slapd tries to create the socket in /var/run/run/ldapi - which fails.
 (We are using heimdal with openldap backend and need ldapi:///)

 Furthermore the socket should be 600 - but I don't know how to set this.

* DB_CONFIG gets lost on upgrades
 /var/lib/ldap/DB_CONFIG gets lost on upgrades, which makes
 reimporting the Ldap database with slapadd very slow with big ldif files.
 If possible save the DB_CONFIG on upgrades.

* slapd with db4.3 has indeed problems.
 My database corruption problems were still present with pre4 - I upgraded
 to pre6 on 2005-04-02. Let's hope db4.2 fixes the database corruption.

* If possible upgrade to openldap 2.2.24
 I've applied your patches to 2.2.24 - no problems so far.

Greetings,
Gerald

--
---------------------------------------------------------------------------
Gerald Hochegger Internet: <email address hidden>
Computer Center Voice : (++43 463) 2700/9613
University of Klagenfurt Fax : (++43 463) 2700/9699
Universitaetsstrasse 65-67
A-9020 Klagenfurt, Austria, Europe
---------------------------------------------------------------------------

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sun, 3 Apr 2005 20:03:42 +0200
From: Torsten Landschoff <email address hidden>
To: Gerald Hochegger <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#291408: OpenLDAP 2.2 packages

--1yeeQ81UyVL57Vl7
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi Gerald,=20

On Sun, Apr 03, 2005 at 05:36:59PM +0200, Gerald Hochegger wrote:
> I've already upgraded our production servers because I was
> hoping to get rid of the database corruption problems.
>=20
> Here are my results so far:
>=20
> * ldapi:/// is not working after the upgrade
> Use
> --localstatedir=3D/var (instead of /var/run)
> in debian/configure.options

Checked in for 2.2.23-2. I really wonder why upstream has changed the
meaning of this option.

> slapd tries to create the socket in /var/run/run/ldapi - which fails.
> (We are using heimdal with openldap backend and need ldapi:///)=20

Interesting. Did not know it was really /needed/ for anything.

> Furthermore the socket should be 600 - but I don't know how to set this.

That should work by using
 slapd -h 'ldapi:///???x-mod=3D600'

> * DB_CONFIG gets lost on upgrades
> /var/lib/ldap/DB_CONFIG gets lost on upgrades, which makes
> reimporting the Ldap database with slapadd very slow with big ldif files.
> If possible save the DB_CONFIG on upgrades.

Known issue, it's on the TODO list.

> * slapd with db4.3 has indeed problems.
> My database corruption problems were still present with pre4 - I upgraded
> to pre6 on 2005-04-02. Let's hope db4.2 fixes the database corruption.

Interesting. I really hope db4.2 fixes those problems.

> * If possible upgrade to openldap 2.2.24
> I've applied your patches to 2.2.24 - no problems so far.

Upgrade will be done after the packaging is stable.

Greetings

 Torsten

--1yeeQ81UyVL57Vl7
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFCUC/+dQgHtVUb5EcRAtSVAJ4l/iEyvVoxLtLjCjpgZHHAU1ztPACfUxaC
CoWF5R4CMLYxmlP6Q9G0p1g=
=yqBS
-----END PGP SIGNATURE-----

--1yeeQ81UyVL57Vl7--

[Steve Langasek (vorlon)]

# Automatically generated email from bts, devscripts version 2.8.14
tags 291408 sarge

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Wed, 6 Apr 2005 19:09:07 -0700
From: Steve Langasek <email address hidden>
To: <email address hidden>
Subject: tagging 291408

# Automatically generated email from bts, devscripts version 2.8.14
tags 291408 sarge

[Torsten Landschoff (t-landschoff)]

tags 291408 - fixed-in-experimental

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Thu, 7 Apr 2005 10:51:22 +0200 (CEST)
From: <email address hidden> (Torsten Landschoff)
To: <email address hidden>
Subject: tagging 291408

tags 291408 - fixed-in-experimental

[Steve Langasek (vorlon)]

As slapd 2.2.23 has now slipped into testing (which was eventually our
intention anyway), this particular bug is presumably fixed; there are other
new ones that need to be dealt with, but none that any longer match this
description.

Cheers,
--
Steve Langasek
postmodern programmer

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Sat, 16 Apr 2005 03:00:09 -0700
From: Steve Langasek <email address hidden>
To: <email address hidden>
Subject: Re: [fixed in 2.2.23] database corruption

--ILuaRSyQpoVaJ1HG
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

As slapd 2.2.23 has now slipped into testing (which was eventually our
intention anyway), this particular bug is presumably fixed; there are other
new ones that need to be dealt with, but none that any longer match this
description.

Cheers,
--=20
Steve Langasek
postmodern programmer

--ILuaRSyQpoVaJ1HG
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFCYOIoKN6ufymYLloRAs4lAJwKUM+w1q9cYEfsLdYacbDip3n6KwCgrSdi
QqC8oS8OdI+C8qe8Wplc7no=
=9Q9A
-----END PGP SIGNATURE-----

--ILuaRSyQpoVaJ1HG--

[Adam Conrad (adconrad)]

This bug is fixed, now that we ship slapd 2.2