slapd package configuration aborts with "Program version 4.7 doesn't match environment version 0.44" error during Hardy -> Lucid upgrade

Bug #536958 reported by Nathan Stratton Treadway on 2010-03-10
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openldap (Ubuntu)
High
Thierry Carrez
Lucid
High
Thierry Carrez

Bug Description

I am attempting to upgrade my Ubuntu server running Hardy to Lucid, using "do-release-upgrade -d" from the command line.

After the downloaded "lucid" upgrade script churned away for a while, I noticed that the slapd package was aborting during the package configuration.

Here are the applicable lines from the upgrade session:

Preparing to replace ldap-utils 2.4.9-0ubuntu0.8.04.2 (using .../ldap-utils_2.4.21-0ubuntu1_amd64.deb) ...
Unpacking replacement ldap-utils ...
Preparing to replace libldap-2.4-2 2.4.9-0ubuntu0.8.04.2 (using .../libldap-2.4-2_2.4.21-0ubuntu1_amd64.deb) ...
Unpacking replacement libldap-2.4-2 ...
[...]
Preparing to replace slapd 2.4.9-0ubuntu0.8.04.2 (using .../slapd_2.4.21-0ubuntu1_amd64.deb) ...
Stopping OpenLDAP: slapd.
  Dumping to /var/backups/slapd-2.4.9-0ubuntu0.8.04.2:
  - directory dc=ontko,dc=com... slapcat: Symbol `ldap_int_global_options' has different size in shared object, consider re-linking
done.
Unpacking replacement slapd ...
[...]
Setting up libldap-2.4-2 (2.4.21-0ubuntu1) ...
[...]
Setting up libldap-2.4-2 (2.4.21-0ubuntu1) ...
[...]
Setting up ldap-utils (2.4.21-0ubuntu1) ...
[...]
Setting up slapd (2.4.21-0ubuntu1) ...
Installing new version of config file /etc/ldap/schema/README ...
Installing new version of config file /etc/ldap/schema/cosine.ldif ...
Installing new version of config file /etc/ldap/schema/inetorgperson.ldif ...
Installing new version of config file /etc/ldap/schema/nis.ldif ...
Installing new version of config file /etc/ldap/schema/openldap.ldif ...
Installing new version of config file /etc/ldap/schema/duaconf.schema ...
Installing new version of config file /etc/ldap/schema/dyngroup.schema ...
Installing new version of config file /etc/ldap/schema/inetorgperson.schema ...
Installing new version of config file /etc/ldap/schema/misc.schema ...
Installing new version of config file /etc/ldap/schema/nis.schema ...
Installing new version of config file /etc/ldap/schema/openldap.schema ...

Configuration file `/etc/ldap/schema/ldapns.schema'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ? Your options are:
    Y or I : install the package maintainer's version
    N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : background this process to examine the situation
 The default action is to keep your current version.
*** ldapns.schema (Y/I/N/O/D/Z) [default=N] ? y
Installing new version of config file /etc/ldap/schema/ldapns.schema ...
Installing new version of config file /etc/apparmor.d/usr.sbin.slapd ...

Configuration file `/etc/default/slapd'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:
    Y or I : install the package maintainer's version
    N or O : keep your currently-installed version
      D : show the differences between the versions
      Z : background this process to examine the situation
 The default action is to keep your current version.
*** slapd (Y/I/N/O/D/Z) [default=N] ? n
Installing new version of config file /etc/init.d/slapd ...
  Backing up /etc/ldap/slapd.conf in /var/backups/slapd-2.4.9-0ubuntu0.8.04.2... done.
failed.
Migrating slapd.conf file (/etc/ldap/slapd.conf) to slapd.d failed with the following error while running slaptest:
    bdb(dc=ontko,dc=com): Program version 4.7 doesn't match environment version 0.44
    hdb_db_open: database "dc=ontko,dc=com" cannot be opened, err -30971. Restore from backup!
    backend_startup_one (type=hdb, suffix="dc=ontko,dc=com"): bi_db_open failed! (-30971)
    slap_startup failed (test would succeed using the -u switch)
dpkg: error processing slapd (--configure):
 subprocess installed post-installation script returned error exit status 1
[...]
Errors were encountered while processing:
 slapd
[do-release-upgrade run terminated...]

The do-release-upgrade process continued configuring other packages, but after it completed the "slapd" package is listed as unconfigured; when I tried to configure it manually I got similar errors:

~# dpkg --configure --pending
Setting up slapd (2.4.21-0ubuntu1) ...
  Backing up /etc/ldap/slapd.conf in /var/backups/slapd-2.4.9-0ubuntu0.8.04.2... done.
failed.

Migrating slapd.conf file (/etc/ldap/slapd.conf) to slapd.d failed with the following error while running slaptest:
    bdb(dc=ontko,dc=com): Program version 4.7 doesn't match environment version 0.44
    hdb_db_open: database "dc=ontko,dc=com" cannot be opened, err -30971. Restore from backup!
    backend_startup_one (type=hdb, suffix="dc=ontko,dc=com"): bi_db_open failed! (-30971)
    slap_startup failed (test would succeed using the -u switch)
dpkg: error processing slapd (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 slapd

As it turns out, my do-release-upgrade session was already "planning" to abort due to an non-zero exit status from another package, but I assume this error from the slapd configuration script would also prevent the Hardy -> Lucid upgrade from running to completion.

Related branches

This issue seems to be caused by the fact that the openldap packages switched from using v4.2 to v4.7 of the Berkeley Database libraries, as mentioned in the changelog.Debian.gz file:
   openldap (2.4.14-0ubuntu1) jaunty; urgency=low
   [...]
     * Build against db4.7 instead of db4.2 at last! Closes: #421946.
   [...]
    -- Mathias Gug <email address hidden> Wed, 18 Feb 2009 18:44:00 -0500

(Since this happened during the Jaunty timeframe, it would make sense that an upgrade from Hardy to Lucid would be affected.)

The best discussion about changing to a new BDB version that I found was an old README.db4.4 that was part of the Debian "subversion" package back in 2006. The document has since been removed from that package, once Subversion started handling the DB conversion automatically, but the text is still available from the source repository for the subversion package:
  http://svn.debian.org/wsvn/pkg-subversion/src/tags/1.4.0-1/debian/README.db4.4

The OpenLDAP changelog files and other discussions out on the web imply that slapd normally does automatic "recovery" of the BDB data files, which I understand should transparently take care of situations where the library version has changed.

However, in the case of the Ubuntu package upgrade, the "slaptest" command is getting executed before the new version of "slapd" is run, and (given the error that is occuring) it would appear that "slaptest" does not performat that same automatic recovery operation....

More specifically, the slapd.postinst script is attempting to use the "slaptest" command to migrate the slapd.conf config file to new slapd.d format, but in the case where old-BDB-version database files exist under /var/lib/ldap, that command errors out, thus causing the postinst script to abort.

In my case, I was able to get past this situation by using the commands described in the above-mentioned README.db4.4 file, as follows:
   (first, install the db4.2-util package)
   # cd /var/lib/ldap
   # db4.2_checkpoint -1
   # db4.2_recover
   # db4.2_archive -d

After that I was able to run a plain "slaptest" command manually without getting the "Program version" error, and when I did another
    # dpkg --pending --configure
, the slapd.postinst script's attempt to convert the configuration info was finally successful.

As far as I know there was nothing unusual about my slapd instance and associated data files before the upgrade -- but I don't have any other installation to compare them to, so it's possible that not all Hardy -> Lucid upgrades will trigger this issue. Still, I'm wondering if there is any way that the slapd.postinst script could detected and avoid this situation without requiring manual intervention. (And if not, whether this issue should be mentioned on the Lucid release notes?)

summary: - slapd package configuration aborts during Hardy -> Lucid upgrade
+ slapd package configuration aborts with "Program version 4.7 doesn't
+ match environment version 0.44" error during Hardy -> Lucid upgrade

I did some more investigation into this issue, and it looks like this bug will affect all Hardy -> Lucid upgrades.

To test this I went through the following steps:

On a machine running Hardy (and which had never had "slapd" installed before):
1a) installed the "slapd" package, allowed the postinst script to configure the installation in the normal manner, and then ran "/etc/init.d/slapd stop" to cleanly shut down the daemon process.

1b) ran "slaptest -f /etc/ldap/slapd.conf" and confirmed that the BDB v4.2 database files did not cause any errors on that machine.

Back on the server recently upgraded to Lucid, I made a simple test directory:
2a) Created a test directory "/root/ldap_hardy_test", and under that an empty subdirectory "libldap".

2b) copied /etc/ldap/slapd.conf.old to ldap_hardy_test/slapd.conf, and edited the "directory" line to point to the /root/ldap_hardy_test/libldap directory.

2c) copied all files in /var/lib/ldap/ into /root/ldap_hardy_test/libldap/

2d) ran "slaptest -f slapd.conf -d 1" to confirm that the slaptest run completed without errors, and that it opened the files in ldap_hardy_test/libldap instead of /var/lib/ldap .

Then I tested the BDB files from the Hardy install:
3a) deleted all files in /root/ldap_hardy_test/libldap, and then copied all the files from the Hardy machine's /var/lib/ldap/ directory into the Lucid machine's "libldap" directory.

3b) ran "slaptest -f slapd.conf" again... and got the " Program version 4.7 doesn't match environment version" error message.

So, it appears that the BDB database files from even the simplest Hardy slapd install with trigger this error....

Steve Langasek (vorlon) wrote :

Mathias, can you take a look at this for beta 2?

Changed in openldap (Ubuntu Lucid):
assignee: nobody → Mathias Gug (mathiaz)
importance: Undecided → High
milestone: none → ubuntu-10.04-beta-2
status: New → Triaged

(As a side note, I noticed that the actual version number printed out in the "doesn't match environment version 0.XXX" message was not consistent -- when I did the original upgrade it was "0.44", but in my later testing it started out at 0.143 and then kept getting larger.

It turns out that the format of the Berkeley DB "environment" file changed between BDB 4.3 and 4.4, and in particular the location of the bytes that encode the database version number changed. In the the case of lucid's OpenLDAP tools (which use DBD v4.7) looking at the environment file left over from hardy (v4.2), the bytes that are interpreted as the "minor" version number actually contains the "Locks granted without waiting" count.

So, for example, if I run
  db4.2_stat -e -N | head | grep "without waiting"
from within the BDB directory, the count shown will match the XXX printed in the "environment version 0.XXX" message from slaptest.

This means that running some other command that changes the locks-granted count [e.g. "db4.2_stat -e" , without the "-N" ] will actually cause the XXX number to change between different attempts to run slaptest .

It seems like the header format of the environment file was stabilized as of db4.4, so db4.7 tools shouldn't have the same problem with files generated by 4.4 and later. )

I took a closer look at the slapd.postinst script, and I believe I see what is causing this issue.

In the "postinst_upgrade_configuration" function, the script first checks to see if the configuration info needs to be converted from "slapd.conf" to "slapd.d" format, and if so it runs the "slaptest" command to perform that conversion.

The script then checks to see if the previous version of the of package used a different Berkeley DB version, and if so it actually moves the old BDB files out of the way and creates new database files using the "slapadd" program (based on the export of the directory data that was saved using the prior version of "slapcat" in the slapd.preinst script).

This arrangement worked fine for the Hardy -> Intrepid -> Jaunty upgrade path, because the two conversions did not happen during the same run of the postinst script. (The configuration was converted when going to Intrepid, and the BDB version was changed during the upgrade to Jaunty.)

However, the direct Hardy -> Lucid upgrade does need to perform both conversions in the same postinst run -- but in the existing arrangement, the script tries to run "slapadd" before the BDB files have been regenerated, and thus triggering the "Program version 4.7 doesn't match environment version" error....

So, it appears that the solution to this problem is to perform the conversions in the opposite order.

Currently, the "load_databases" function assumes that the configuration information is in the directory format, so it can't be used until after that conversion has happened. However, it seems like it should be fairly easy to tweak it to work with either configuration format (e.g. using the same logic as is currently found in the "dump_databases" function). The two conversion steps in "postinst_upgrade_configuration" could then be switched, which would (hopefully) allow both to be performed successfully during the same upgrade....

Thierry Carrez (ttx) on 2010-03-29
Changed in openldap (Ubuntu Lucid):
assignee: Mathias Gug (mathiaz) → Thierry Carrez (ttx)
Thierry Carrez (ttx) on 2010-03-29
Changed in openldap (Ubuntu Lucid):
status: Triaged → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openldap - 2.4.21-0ubuntu3

---------------
openldap (2.4.21-0ubuntu3) lucid; urgency=low

  * debian/slapd.postinst, debian/slapd.scripts-common: Upgrade databases
    before trying to convert to slapd.d, to avoid upgrade failure from hardy
    (LP: #536958)
  * debian/slapd.postinst: Add a {1} numeric index to olcAccess entry in
    olcDatabase={0}config.ldif to avoid upgrade failures (LP: #538516, #526230)
 -- Thierry Carrez <email address hidden> Mon, 29 Mar 2010 13:31:47 +0200

Changed in openldap (Ubuntu Lucid):
status: In Progress → Fix Released

I just did another hardy -> lucid upgrade run (on a test machine running an as-installed-by-the-package slapd configuration), and can confirm that the new version of the slapd.postinst was able to complete without triggering the "Program version 4.7 doesn't match environment version" error.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers