slapd: checkpoint directive missed from bdb backend

[Debian Bug Importer]

Automatically imported from Debian bug report #272984 http://bugs.debian.org/272984

[Debian Bug Importer (debzilla)]

Automatically imported from Debian bug report #272984 http://bugs.debian.org/272984

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Thu, 23 Sep 2004 09:36:17 +0100
From: Doug Winter <email address hidden>
To: Debian Bug Tracking System <email address hidden>
Subject: slapd: checkpoint directive missed from bdb backend

Package: slapd
Version: 2.1.30-3
Severity: critical
Justification: causes serious data loss

When the slapd package is installed, it writes an /etc/ldap/slapd.conf
file automatically, based on user input.

By default this uses the bdb backend. Without a "checkpoint" directive
in the configuration file, the bdb database never checkpoints, which
means that if the system fails without shutting slapd down cleanly all
changes made are lost.

This happened to me several times because of power issues in our
facility, which was immensely frustrating :)

If the bdb backend is used, there should always be a checkpoint entry.

Cheers,

Doug.

-- System Information:
Debian Release: testing/unstable
  APT prefers testing
  APT policy: (900, 'testing'), (300, 'unstable')
Architecture: i386 (i686)
Kernel: Linux 2.6.7-1-k7-smp
Locale: LANG=en_GB, LC_CTYPE=en_GB

Versions of packages slapd depends on:
ii coreutils [fileutils] 5.0.91-2 The GNU core utilities
ii debconf 1.4.29 Debian configuration management sy
ii fileutils 5.0.91-2 The GNU file management utilities
ii libc6 2.3.2.ds1-13 GNU C Library: Shared libraries an
ii libdb4.2 4.2.52-16 Berkeley v4.2 Database Libraries [
ii libgcrypt11 1.2.0-4 LGPL Crypto library - runtime libr
ii libgnutls11 1.0.16-3 GNU TLS library - runtime library
ii libgpg-error0 0.7-1 library for common error values an
ii libiodbc2 3.51.2-2 iODBC Driver Manager
ii libldap2 2.1.30-3 OpenLDAP libraries
ii libltdl3 1.5.6-1 A system independent dlopen wrappe
ii libsasl2 2.1.18-4.1 Authentication abstraction library
ii libslp1 1.0.11-7 OpenSLP libraries
ii libwrap0 7.6.dbs-4 Wietse Venema's TCP wrappers libra
ii perl [libmime-base64-perl] 5.8.3-3 Larry Wall's Practical Extraction
ii psmisc 21.4-1 Utilities that use the proc filesy
ii zlib1g 1:1.2.1-5 compression library - runtime

-- debconf information:
  slapd/password_mismatch:
  slapd/fix_directory: true
  slapd/invalid_config: true
* shared/organization: ICP Europe PLC
  slapd/upgrade_slapcat_failure:
  slapd/upgrade_slapadd_failure:
  slapd/backend: BDB
* slapd/allow_ldap_v2: false
  slapd/no_configuration: false
  slapd/move_old_database: true
  slapd/suffix_change: false
  slapd/slave_databases_require_updateref:
  slapd/autoconf_modules: true
  slapd/purge_database: false
  slapd/admin:
* slapd/domain: icpeurope.net

[Fabio Massimo Di Nitto (fabbione)]

Fixed with openldap2_2.1.30-2ubuntu3 upload.

[Stephen Frost (sfrost)]

Greetings,

  The bdb backend has 'dbnosync' off by default which means that changes
  should be sync'd to disk immediately. I think you may have a problem
  elsewhere. Did you try to dbrecover? When dbnosync is off, what does
  checkpoint do, exactly? My best guess is that it writes a checkpoint,
  which may mean you don't have to dbrecover but at the same time,
  everything between that checkpoint and the time the machine crashed
  would be lost.

  This bug isn't 'critical'. It might be normal, maybe.

   Stephen

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Thu, 21 Oct 2004 08:28:09 -0400
From: Stephen Frost <email address hidden>
To: <email address hidden>
Subject: Re: slapd: checkpoint directive missed from bdb backend

--uX2tiToO0oGq+LKk
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Greetings,

  The bdb backend has 'dbnosync' off by default which means that changes
  should be sync'd to disk immediately. I think you may have a problem
  elsewhere. Did you try to dbrecover? When dbnosync is off, what does
  checkpoint do, exactly? My best guess is that it writes a checkpoint,
  which may mean you don't have to dbrecover but at the same time,
  everything between that checkpoint and the time the machine crashed
  would be lost.

  This bug isn't 'critical'. It might be normal, maybe.

   Stephen

--uX2tiToO0oGq+LKk
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBd6tZrzgMPqB3kigRAg8DAKCEVTD99vEW2hMYp0lBzk6YdvgQSwCfYs0P
BvJqfIumWPaVrmh+S3Jx/9s=
=GTgL
-----END PGP SIGNATURE-----

--uX2tiToO0oGq+LKk--

[Torsten Landschoff (torsten)]

Hi,

On Thu, Oct 21, 2004 at 08:28:09AM -0400, Stephen Frost wrote:
> The bdb backend has 'dbnosync' off by default which means that changes
> should be sync'd to disk immediately. I think you may have a problem
> elsewhere. Did you try to dbrecover? When dbnosync is off, what does
> checkpoint do, exactly? My best guess is that it writes a checkpoint,
> which may mean you don't have to dbrecover but at the same time,
> everything between that checkpoint and the time the machine crashed
> would be lost.
>
> This bug isn't 'critical'. It might be normal, maybe.

I've read up on Berkeley DB transactions and it seems to me that it
would make sense to write a checkpoint from time to time. No data should
be lost but in the default configuration slapd only checkpoints the DB
during shutdown which means that reopening the database after a crash
(which looks for me like the only reason why slapd would go down short
of a system reboot) will take a lot of time wading through old log
files.

I don't want to mess with the slapd defaults for this small problem
though and I wonder what upstream has to say. AFAICT the checkpoint
directive leads to a massive breakdown of write performance for not that
much gain.

Greetings

 Torsten

[Torsten Landschoff (t-landschoff)]

tags 272984 upstream

[Torsten Landschoff (t-landschoff)]

severity 272984 minor

[Torsten Landschoff (torsten)]

Hi all,

A Debian user reported about a missing checkpoint directive in
slapd.conf leading to data loss (see http://bugs.debian.org/272984). I
can't imagine how that would happen but I wonder why checkpointing is
only done during shutdown of slapd by default (I'd expect that to happen
really seldom).

I'd rather have the default changed if it increases data security
or slapd behaviour in case of a system crash. But I guess that's
upstreams call.

Comments?

Greetings

 Torsten

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Tue, 9 Nov 2004 01:38:12 +0100 (CET)
From: <email address hidden> (Torsten Landschoff)
To: <email address hidden>
Subject: tagging 272984

tags 272984 upstream

[Debian Bug Importer (debzilla)]

Message-Id: <email address hidden>
Date: Tue, 9 Nov 2004 01:38:22 +0100 (CET)
From: <email address hidden> (Torsten Landschoff)
To: <email address hidden>
Subject: severity of 272984 is minor

severity 272984 minor

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 9 Nov 2004 01:37:50 +0100
From: Torsten Landschoff <email address hidden>
To: <email address hidden>
Subject: Re: Bug#272984: slapd: checkpoint directive missed from bdb backend

--vtzGhvizbBRQ85DL
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi,=20

On Thu, Oct 21, 2004 at 08:28:09AM -0400, Stephen Frost wrote:
> The bdb backend has 'dbnosync' off by default which means that changes
> should be sync'd to disk immediately. I think you may have a problem
> elsewhere. Did you try to dbrecover? When dbnosync is off, what does
> checkpoint do, exactly? My best guess is that it writes a checkpoint,
> which may mean you don't have to dbrecover but at the same time,
> everything between that checkpoint and the time the machine crashed
> would be lost.
>=20
> This bug isn't 'critical'. It might be normal, maybe.

I've read up on Berkeley DB transactions and it seems to me that it
would make sense to write a checkpoint from time to time. No data should
be lost but in the default configuration slapd only checkpoints the DB
during shutdown which means that reopening the database after a crash=20
(which looks for me like the only reason why slapd would go down short
of a system reboot) will take a lot of time wading through old log
files.

I don't want to mess with the slapd defaults for this small problem
though and I wonder what upstream has to say. AFAICT the checkpoint
directive leads to a massive breakdown of write performance for not that=20
much gain.=20

Greetings

 Torsten

--vtzGhvizbBRQ85DL
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBkBFedQgHtVUb5EcRAmK3AJ9ElGWLD9e0GV5AG3uW/o2Keh+K9ACcCojG
sPPCt7fnH5qeBsCW2lAuHUs=
=uWH+
-----END PGP SIGNATURE-----

--vtzGhvizbBRQ85DL--

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 9 Nov 2004 01:49:47 +0100
From: Torsten Landschoff <email address hidden>
To: <email address hidden>
Cc: <email address hidden>
Subject: Checkpointing in slapd bdb backend?

--EuxKj2iCbKjpUGkD
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi all,=20

A Debian user reported about a missing checkpoint directive in
slapd.conf leading to data loss (see http://bugs.debian.org/272984). I
can't imagine how that would happen but I wonder why checkpointing is
only done during shutdown of slapd by default (I'd expect that to happen
really seldom).

I'd rather have the default changed if it increases data security=20
or slapd behaviour in case of a system crash. But I guess that's=20
upstreams call.=20

Comments?

Greetings

 Torsten

--EuxKj2iCbKjpUGkD
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBkBQrdQgHtVUb5EcRAhCfAJ9hEs4pvowCQs3EMBGnXt12+/hYOwCfc9An
HUBjmCxZDlP0VHPxKtN6eMg=
=oKAy
-----END PGP SIGNATURE-----

--EuxKj2iCbKjpUGkD--

[Doug Winter (winjer)]

Torsten Landschoff wrote:
> I've read up on Berkeley DB transactions and it seems to me that it
> would make sense to write a checkpoint from time to time. No data should
> be lost but in the default configuration slapd only checkpoints the DB
> during shutdown which means that reopening the database after a crash
> (which looks for me like the only reason why slapd would go down short
> of a system reboot) will take a lot of time wading through old log
> files.
>
> I don't want to mess with the slapd defaults for this small problem
> though and I wonder what upstream has to say. AFAICT the checkpoint
> directive leads to a massive breakdown of write performance for not that
> much gain.

I have lost significant data because this directive wasn't in the
logfile, and I didn't know how to recover a berkeley db. The third
time this happened I spent several hours reading up on it and managed to
get my data back. If slapd or the system crashes (in this case it was
because of a power outage), then the database needs to be manually
recovered.

It doesn't happen automatically, so can lead to data loss if the
database is not recovered.

doug.

--
6973E2CF: 2C95 66AD 1596 37D2 41FC 609F 76C0 A4EC 6973 E2CF
http://adju.st/

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 09 Nov 2004 07:41:42 +0000
From: Doug Winter <email address hidden>
To: Torsten Landschoff <email address hidden>,
 <email address hidden>
Subject: Re: Bug#272984: slapd: checkpoint directive missed from bdb backend

Torsten Landschoff wrote:
> I've read up on Berkeley DB transactions and it seems to me that it
> would make sense to write a checkpoint from time to time. No data should
> be lost but in the default configuration slapd only checkpoints the DB
> during shutdown which means that reopening the database after a crash
> (which looks for me like the only reason why slapd would go down short
> of a system reboot) will take a lot of time wading through old log
> files.
>
> I don't want to mess with the slapd defaults for this small problem
> though and I wonder what upstream has to say. AFAICT the checkpoint
> directive leads to a massive breakdown of write performance for not that
> much gain.

I have lost significant data because this directive wasn't in the
logfile, and I didn't know how to recover a berkeley db. The third
time this happened I spent several hours reading up on it and managed to
get my data back. If slapd or the system crashes (in this case it was
because of a power outage), then the database needs to be manually
recovered.

It doesn't happen automatically, so can lead to data loss if the
database is not recovered.

doug.

--
6973E2CF: 2C95 66AD 1596 37D2 41FC 609F 76C0 A4EC 6973 E2CF
http://adju.st/

[Stephen Frost (sfrost)]

* Doug Winter (<email address hidden>) wrote:
> Torsten Landschoff wrote:
> >I've read up on Berkeley DB transactions and it seems to me that it
> >would make sense to write a checkpoint from time to time. No data should
> >be lost but in the default configuration slapd only checkpoints the DB
> >during shutdown which means that reopening the database after a crash
> >(which looks for me like the only reason why slapd would go down short
> >of a system reboot) will take a lot of time wading through old log
> >files.
> >
> >I don't want to mess with the slapd defaults for this small problem
> >though and I wonder what upstream has to say. AFAICT the checkpoint
> >directive leads to a massive breakdown of write performance for not that
> >much gain.
>
> I have lost significant data because this directive wasn't in the
> logfile, and I didn't know how to recover a berkeley db. The third
> time this happened I spent several hours reading up on it and managed to
> get my data back. If slapd or the system crashes (in this case it was
> because of a power outage), then the database needs to be manually
> recovered.
>
> It doesn't happen automatically, so can lead to data loss if the
> database is not recovered.

The point is that the database *can* be recovered unless something worse
has happened (such as file system corruption). Just because you can't
open it immediately upon reboot doesn't mean the data has been lost.
Torsten, perhaps we could add something to the README about how to use a
Berkley db, with some URLs to more documentation.

 Stephen

[Torsten Landschoff (torsten)]

Hi Stephen,

On Tue, Nov 09, 2004 at 07:29:08AM -0500, Stephen Frost wrote:
> > It doesn't happen automatically, so can lead to data loss if the
> > database is not recovered.
>
> The point is that the database *can* be recovered unless something worse
> has happened (such as file system corruption). Just because you can't
> open it immediately upon reboot doesn't mean the data has been lost.
> Torsten, perhaps we could add something to the README about how to use a
> Berkley db, with some URLs to more documentation.

After skipping through the DB docs I am sure you could write a book
about that... Therefore I'll just add some little notes.

Speaking about db_recover: Originally that was done automatically in
slapd. It was dropped exactly because data loss could result. Recovering
the database is risky in the sense that doing it twice at the same time
/will/ kill your data. AFAIK power loss during database recovery could
also result in complete data loss.

So basically you should backup your db before running recover. Which is
another good reason to not do it automatically.

Greetings

 Torsten

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 9 Nov 2004 07:29:08 -0500
From: Stephen Frost <email address hidden>
To: Doug Winter <email address hidden>, <email address hidden>
Cc: Torsten Landschoff <email address hidden>
Subject: Re: [debian-openldap] Bug#272984: slapd: checkpoint directive missed from bdb backend

--QwcgtzFXO4v1D05N
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

* Doug Winter (<email address hidden>) wrote:
> Torsten Landschoff wrote:
> >I've read up on Berkeley DB transactions and it seems to me that it
> >would make sense to write a checkpoint from time to time. No data should
> >be lost but in the default configuration slapd only checkpoints the DB
> >during shutdown which means that reopening the database after a crash=20
> >(which looks for me like the only reason why slapd would go down short
> >of a system reboot) will take a lot of time wading through old log
> >files.
> >
> >I don't want to mess with the slapd defaults for this small problem
> >though and I wonder what upstream has to say. AFAICT the checkpoint
> >directive leads to a massive breakdown of write performance for not that=
=20
> >much gain.=20
>=20
> I have lost significant data because this directive wasn't in the=20
> logfile, and I didn't know how to recover a berkeley db. The third=20
> time this happened I spent several hours reading up on it and managed to=
=20
> get my data back. If slapd or the system crashes (in this case it was=20
> because of a power outage), then the database needs to be manually=20
> recovered.
>=20
> It doesn't happen automatically, so can lead to data loss if the=20
> database is not recovered.

The point is that the database *can* be recovered unless something worse
has happened (such as file system corruption). Just because you can't
open it immediately upon reboot doesn't mean the data has been lost.
Torsten, perhaps we could add something to the README about how to use a
Berkley db, with some URLs to more documentation.

 Stephen

--QwcgtzFXO4v1D05N
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFBkLgUrzgMPqB3kigRAkEAAJ9+vBoQmMOgiBSYHsOY9NrPP5p19QCdGw0u
RoQ+a2DUvKxYaCelQKf+g3o=
=DGiL
-----END PGP SIGNATURE-----

--QwcgtzFXO4v1D05N--

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 9 Nov 2004 13:47:47 +0100
From: Torsten Landschoff <email address hidden>
To: Stephen Frost <email address hidden>
Cc: Doug Winter <email address hidden>, <email address hidden>
Subject: Re: [debian-openldap] Bug#272984: slapd: checkpoint directive missed from bdb backend

--UlVJffcvxoiEqYs2
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi Stephen,=20

On Tue, Nov 09, 2004 at 07:29:08AM -0500, Stephen Frost wrote:
> > It doesn't happen automatically, so can lead to data loss if the=20
> > database is not recovered.
>=20
> The point is that the database *can* be recovered unless something worse
> has happened (such as file system corruption). Just because you can't
> open it immediately upon reboot doesn't mean the data has been lost.
> Torsten, perhaps we could add something to the README about how to use a
> Berkley db, with some URLs to more documentation.

After skipping through the DB docs I am sure you could write a book
about that... Therefore I'll just add some little notes.=20

Speaking about db_recover: Originally that was done automatically in
slapd. It was dropped exactly because data loss could result. Recovering
the database is risky in the sense that doing it twice at the same time
/will/ kill your data. AFAIK power loss during database recovery could
also result in complete data loss.=20

So basically you should backup your db before running recover. Which is
another good reason to not do it automatically.=20

Greetings

 Torsten

--UlVJffcvxoiEqYs2
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBkLxydQgHtVUb5EcRAiC5AJ9oD9KAfUjjNYP2r3WFPvcmBQOIxwCbBhYR
TjVjiGWvpNZxjT8tI6MTjgg=
=ECvg
-----END PGP SIGNATURE-----

--UlVJffcvxoiEqYs2--

[Doug Winter (winjer)]

Stephen Frost wrote:
> The point is that the database *can* be recovered unless something worse
> has happened (such as file system corruption). Just because you can't
> open it immediately upon reboot doesn't mean the data has been lost.
> Torsten, perhaps we could add something to the README about how to use a
> Berkley db, with some URLs to more documentation.

Something in the README saying either put in a checkpoint, or this is
how to recover after a crash would save a lot of people some grief I
think. For a lot of us, performance is unimportant compared to data
integrity ;)

doug.

--
6973E2CF: 2C95 66AD 1596 37D2 41FC 609F 76C0 A4EC 6973 E2CF
http://adju.st/

[Debian Bug Importer (debzilla)]

Message-ID: <email address hidden>
Date: Tue, 09 Nov 2004 20:26:23 +0000
From: Doug Winter <email address hidden>
To: <email address hidden>
Subject: Re: [debian-openldap] Bug#272984: slapd: checkpoint directive missed
 from bdb backend

Stephen Frost wrote:
> The point is that the database *can* be recovered unless something worse
> has happened (such as file system corruption). Just because you can't
> open it immediately upon reboot doesn't mean the data has been lost.
> Torsten, perhaps we could add something to the README about how to use a
> Berkley db, with some URLs to more documentation.

Something in the README saying either put in a checkpoint, or this is
how to recover after a crash would save a lot of people some grief I
think. For a lot of us, performance is unimportant compared to data
integrity ;)

doug.

--
6973E2CF: 2C95 66AD 1596 37D2 41FC 609F 76C0 A4EC 6973 E2CF
http://adju.st/

[Matthijs Mohlmann (matthijs)]

Hi,

There is documentation in README.DB_CONFIG which explains how to tune the BDB/HDB backend. There is also referred to the checkpoint directive. Closing bugreport.

Regards,

Matthijs Mohlmann