Init script dependency error: krb5-kdc starts before slapd

[Alexander Betaev]

Binary package hint: krb5-kdc

If Kerberos5 configured to use LDAP directory on the same computer, it does not launches at startup due to init script dependency is not configured. The update-rc.d script creates symlinks for krb5-kdc and slapd with the following names: S18krb5-kdc, S19slapd. This makes Kerberos key distribution center launch before LDAP directory which contains data for this service and I get the following in the /var/log/daemon.log:
krb5kdc[1018]: Can't contact LDAP server - while initializing database for realm MYREALM

I think its no problem to make KDC to start after LDAP server and it will definitely solve this issue.

[Alexander Betaev (infestator)]

Forgot to say that this is Ubuntu 10.04.

[Russ Allbery (rra-debian)]

infestator <email address hidden> writes:

> If Kerberos5 configured to use LDAP directory on the same computer, it
> does not launches at startup due to init script dependency is not
> configured. The update-rc.d script creates symlinks for krb5-kdc and
> slapd with the following names: S18krb5-kdc, S19slapd. This makes
> Kerberos key distribution center launch before LDAP directory which
> contains data for this service and I get the following in the
> /var/log/daemon.log:

> krb5kdc[1018]: Can't contact LDAP server - while initializing database for realm MYREALM

> I think its no problem to make KDC to start after LDAP server and it
> will definitely solve this issue.

It's definitely a problem for the KDC to start after the LDAP server if
the LDAP server is using Kerberos for authentication, which is probably
still a more common configuration than putting the KDC data in LDAP.

Unfortunately, both init script orderings break different things for
different people. What really needs to happen is that one or the other
(or preferrably both) services need to be robust against the other service
not yet being initialized.

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

[Alexander Betaev (infestator)]

Russ, you are right.

But in what case does LDAP performs an authentication using Kerberos on local machine? I cannot imagine what for can LDAP use local kerberos authentication.
I am not very skilled in all these and my questions may be a little bit stupid :-[ I just can suppose that Kerberos authentication may be used for authenticating replication servers, but there is not any word about Kerberos in LDAP manual. The SSL/TLS authentication is used instead.

[Russ Allbery (rra-debian)]

infestator <email address hidden> writes:

> Russ, you are right.

> But in what case does LDAP performs an authentication using Kerberos on
> local machine? I cannot imagine what for can LDAP use local kerberos
> authentication.

The case that's most often cited is if you're co-locating infrastructure
on single machines, in which case you may have an LDAP replica and a KDC
on the same host. The LDAP replica then needs to do a GSSAPI
authentication to the master for replication, which requires access to the
KDC.

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

[Clint Byrum (clint-fewbar)]

Since both services may depend on the other in ways that will break, we can only support a default configuration.

The server guide currently does not have kerberos depending on LDAP, nor does it suggest LDAP depend on kerberos.

So, the current configuration is probably sufficient, and dependencies can be adjusted for specific configurations as necessary.

Setting importance to wishlist, as this is ultimately a feature request not a bug.

Marking Opinion, as there is no clear reason to reject or accept this feature request.

[Sam Hartman (hartmans)]

In Debian unstable installing krb5-kdxc-ldap automatically changes the order. This could be backported.

"Clint Byrum" <email address hidden> wrote:

>Since both services may depend on the other in ways that will break, we
>can only support a default configuration.
>
>The server guide currently does not have kerberos depending on LDAP, nor
>does it suggest LDAP depend on kerberos.
>
>So, the current configuration is probably sufficient, and dependencies
>can be adjusted for specific configurations as necessary.
>
>Setting importance to wishlist, as this is ultimately a feature request
>not a bug.
>
>Marking Opinion, as there is no clear reason to reject or accept this
>feature request.
>
>** Changed in: krb5 (Ubuntu)
> Importance: Undecided => Wishlist
>
>** Changed in: krb5 (Ubuntu)
> Status: New => Opinion
>
>--
>Init script dependency error: krb5-kdc starts before slapd
>https://bugs.launchpad.net/bugs/652433
>You received this bug notification because you are subscribed to krb5 in
>ubuntu.
>

[Thomas Schweikle (tps)]

Russ Allbery wrote on 2010-09-30:
> It's definitely a problem for the KDC to start after the LDAP
> server if the LDAP server is using Kerberos for authentication,
> which is probably still a more common configuration than
> putting the KDC data in LDAP.

I am putting Kerberos Data into an LDAP-Server since this is possible. Kerberos depends on LDAP, but it doesn't mater if kerberos isn't up and running --- you can assume having both servers on one and the same system in such cases and ldap configured to use sockets or local interfaces only communication with kdc or kadmin. If not you'll have a biddy and egg problem. But it is absolutely not usefull to have slapd start *AFTER* krb5-kdc: it can't get any neccessary data this way.

> Unfortunately, both init script orderings break different things
> for different people. What really needs to happen is that one
> or the other (or preferrably both) services need to be robust
> against the other service not yet being initialized.

LDAP ist robust against kerberos not running at the moment slapd starts. Kerberos can't be robust about that. No way. If it stores data in LDAP it has to have access to the server.

At the moment this breaks the whole thing.

[Thomas Schweikle (tps)]

Clint Byrum:
This is not an opinion. It is a necessity if you like to have stable running systems. At the moment kdc will not run after a reboot. I suppose this being an error, not an "opinion".
If Ubuntu wants parts of the server market, than change this! A simple reboot should not break a default setup.

Sam Hartman pointed out, Debian changed it --- just because of this reason.

[Russ Allbery (rra-debian)]

Thomas Schweikle <email address hidden> writes:

> LDAP ist robust against kerberos not running at the moment slapd starts.

I'm not sure that this is the case for an LDAP replica that uses GSS-API
to authenticate to the master, since I believe the very first thing that
slapd does is attempt the authentication to the master.

If this is not the case, or if slapd handles this cleanly (by sleeping and
retrying until it can get a connection without any other negative
consequences), then it's indeed robust here and slapd can start first.
But someone should verify that rather than assuming, since I know we've
had trouble with it in the past.

> Kerberos can't be robust about that. No way. If it stores data in LDAP
> it has to have access to the server.

It can. All it has to do is sleep if it can't open an LDAP connection for
a few seconds and then try again.

There's a tradeoff, of course, in that you lose error reporting from the
init script if it currently attempts to open the LDAP connection before
backgrounding itself. I'm not sure if that's the case or not. If it
already doesn't open the LDAP connection until after it's backgrounded,
you lose nothing by adding some pauses and repeated attempts to contact
the LDAP server.

Ideally, they should both be robust against the other not being up yet.

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

[Clint Byrum (clint-fewbar)]

Ok, so now I'm confused. This should have been fixed in Debian, as Sam Hartman shows us, here:

krb5 (1.8.1+dfsg-3) unstable; urgency=high

  * CVE-2010-1321 GSS-API accept sec context null pointer deref, Closes:
    #582261
  * Force use of bash for build, Closes: #581473
  * Start slapd before krb5 when krb5-kdc-ldap installed, Closes:
    #582122

 -- Sam Hartman <email address hidden> Wed, 19 May 2010 16:37:36 -0400

Testing this on natty by installing krb5-kdc-ldap, and then slapd:

# ls -l /etc/rc2.d
total 4
-rw-r--r-- 1 root root 677 Nov 1 09:36 README
lrwxrwxrwx 1 root root 18 Feb 4 07:55 S18krb5-kdc -> ../init.d/krb5-kdc
lrwxrwxrwx 1 root root 15 Feb 4 07:56 S19slapd -> ../init.d/slapd
lrwxrwxrwx 1 root root 18 Nov 2 09:51 S99ondemand -> ../init.d/ondemand
lrwxrwxrwx 1 root root 18 Nov 2 09:51 S99rc.local -> ../init.d/rc.local

The problem is that the override isn't being respected, because it relies on insserv being called. insserv isn't called, because on Ubuntu systems, legacy-bootordering is the norm, so this override will not help unfortunately. If I manually run 'insserv' as root, this does reorder things:

# ls -l /etc/rc2.d
total 4
-rw-r--r-- 1 root root 677 Nov 1 09:36 README
lrwxrwxrwx 1 root root 15 Feb 4 08:04 S01slapd -> ../init.d/slapd
lrwxrwxrwx 1 root root 18 Feb 4 08:04 S02krb5-kdc -> ../init.d/krb5-kdc
lrwxrwxrwx 1 root root 18 Feb 4 08:04 S03ondemand -> ../init.d/ondemand
lrwxrwxrwx 1 root root 18 Feb 4 08:04 S03rc.local -> ../init.d/rc.local

So, this is really caused by Ubuntu's sysv-rc disabling insserv. Since Ubuntu has chosen a different boot, this is just going to be something we have to maintain delta for I think.

In this case I think the right fix for Ubuntu is going to be to add this to krb5-kdc-slapd's postinst:
update-rc.d slapd remove
update-rc.d slapd start 17 2 3 4 5 . stop 19 0 1 6 .

Either way, I have to agree that I was wrong, and this does have a solution and so can be set to Confirmed. I'll also raise the importance to Low, because the default config does not work in what would probably be a very common use case (kdc on the same box as ldap).

The workaround, btw, is to run the two update-rc.d commands above, or 'insserv'.

[Sam Hartman (hartmans)]

I'm not against including a patch in the Debian package to reduce Ubuntu
deltas. I want to make sure that things continue to work if inserv is
used as that's where Debian is going. If we can preserve that, I think
that having a patch mostly intended for Ubuntu is fine.

[Fede (beffa)]

I believe that the proposed solution is not enough.

$ ls -l rc2.d/
total 12
drwxr-xr-x 2 root root 4096 2011-08-17 16:18 ./
drwxr-xr-x 102 root root 4096 2011-08-17 15:56 ../
-rw-r--r-- 1 root root 677 2011-06-09 21:46 README
lrwxrwxrwx 1 root root 15 2011-08-09 18:26 S15bind9 -> ../init.d/bind9*
lrwxrwxrwx 1 root root 15 2011-08-17 16:18 S17slapd -> ../init.d/slapd*
lrwxrwxrwx 1 root root 27 2011-08-14 14:47 S18krb5-admin-server -> ../init.d/krb5-admin-server*
lrwxrwxrwx 1 root root 18 2011-08-14 14:47 S18krb5-kdc -> ../init.d/krb5-kdc*
lrwxrwxrwx 1 root root 21 2011-08-14 17:03 S20libnss-ldap -> ../init.d/libnss-ldap*
lrwxrwxrwx 1 root root 27 2011-08-09 21:42 S20nfs-kernel-server -> ../init.d/nfs-kernel-server*
lrwxrwxrwx 1 root root 17 2011-08-11 20:24 S20postfix -> ../init.d/postfix*
lrwxrwxrwx 1 root root 18 2011-08-11 21:30 S21quotarpc -> ../init.d/quotarpc*
lrwxrwxrwx 1 root root 13 2011-08-09 21:36 S23ntp -> ../init.d/ntp*
lrwxrwxrwx 1 root root 18 2011-08-11 21:28 S50netatalk -> ../init.d/netatalk*
lrwxrwxrwx 1 root root 15 2011-08-09 18:26 S50rsync -> ../init.d/rsync*
lrwxrwxrwx 1 root root 19 2011-08-09 18:26 S70dns-clean -> ../init.d/dns-clean*
lrwxrwxrwx 1 root root 18 2011-08-09 18:26 S70pppd-dns -> ../init.d/pppd-dns*
lrwxrwxrwx 1 root root 21 2011-08-09 18:27 S99grub-common -> ../init.d/grub-common*
lrwxrwxrwx 1 root root 18 2011-08-09 18:16 S99ondemand -> ../init.d/ondemand*
lrwxrwxrwx 1 root root 18 2011-08-09 18:16 S99rc.local -> ../init.d/rc.local*

$ cat /var/log/daemon.log
...
Aug 17 15:56:04 xxx named[944]: running
Aug 17 15:56:05 xxx kadmind[971]: Can't contact LDAP server while initializing, aborting
Aug 17 15:56:05 xxx krb5kdc[974]: Can't contact LDAP server - while initializing database for realm XXX.XXX
...

Could this be related to the fact that when the KDC complains and stops the server has not yet received its (static) IP address from the DHCP server? In any case adding a line in /etc/hosts does not help.

My system runs 10.04

[Clint Byrum (clint-fewbar)]

Excerpts from Fede's message of Wed Aug 17 15:27:24 UTC 2011:
> I believe that the proposed solution is not enough.
>
> $ ls -l rc2.d/
> total 12
> drwxr-xr-x 2 root root 4096 2011-08-17 16:18 ./
> drwxr-xr-x 102 root root 4096 2011-08-17 15:56 ../
> -rw-r--r-- 1 root root 677 2011-06-09 21:46 README
> lrwxrwxrwx 1 root root 15 2011-08-09 18:26 S15bind9 -> ../init.d/bind9*
> lrwxrwxrwx 1 root root 15 2011-08-17 16:18 S17slapd -> ../init.d/slapd*
> lrwxrwxrwx 1 root root 27 2011-08-14 14:47 S18krb5-admin-server -> ../init.d/krb5-admin-server*
> lrwxrwxrwx 1 root root 18 2011-08-14 14:47 S18krb5-kdc -> ../init.d/krb5-kdc*
> lrwxrwxrwx 1 root root 21 2011-08-14 17:03 S20libnss-ldap -> ../init.d/libnss-ldap*
> lrwxrwxrwx 1 root root 27 2011-08-09 21:42 S20nfs-kernel-server -> ../init.d/nfs-kernel-server*
> lrwxrwxrwx 1 root root 17 2011-08-11 20:24 S20postfix -> ../init.d/postfix*
> lrwxrwxrwx 1 root root 18 2011-08-11 21:30 S21quotarpc -> ../init.d/quotarpc*
> lrwxrwxrwx 1 root root 13 2011-08-09 21:36 S23ntp -> ../init.d/ntp*
> lrwxrwxrwx 1 root root 18 2011-08-11 21:28 S50netatalk -> ../init.d/netatalk*
> lrwxrwxrwx 1 root root 15 2011-08-09 18:26 S50rsync -> ../init.d/rsync*
> lrwxrwxrwx 1 root root 19 2011-08-09 18:26 S70dns-clean -> ../init.d/dns-clean*
> lrwxrwxrwx 1 root root 18 2011-08-09 18:26 S70pppd-dns -> ../init.d/pppd-dns*
> lrwxrwxrwx 1 root root 21 2011-08-09 18:27 S99grub-common -> ../init.d/grub-common*
> lrwxrwxrwx 1 root root 18 2011-08-09 18:16 S99ondemand -> ../init.d/ondemand*
> lrwxrwxrwx 1 root root 18 2011-08-09 18:16 S99rc.local -> ../init.d/rc.local*
>
> $ cat /var/log/daemon.log
> ...
> Aug 17 15:56:04 xxx named[944]: running
> Aug 17 15:56:05 xxx kadmind[971]: Can't contact LDAP server while initializing, aborting
> Aug 17 15:56:05 xxx krb5kdc[974]: Can't contact LDAP server - while initializing database for realm XXX.XXX
> ...
>
> Could this be related to the fact that when the KDC complains and stops
> the server has not yet received its (static) IP address from the DHCP
> server? In any case adding a line in /etc/hosts does not help.
>

Its entirely possible, especially if you've specified the hostname of
the server and it is bound to that specific IP.

Oneiric includes a fix that delays runlevel 2 until all interfaces in
/etc/network/interfaces are available. I'm not sure if we'll be able to
push that into 10.04, but its at least worth looking into as the solution
is fairly simple, just adding a few new events and jobs.

See bug #580319 for more info on that.

Anyway, this sounds like that bug.. which affects pretty much all services
that start on runlevel 2 and might be addressed by a specific IP.

[Ryan Tandy (rtandy)]

I have noticed that the slapd init script terminates before slapd is actually ready to accept connections, and I think that is the problem you're having too. In my scripts that stop/start slapd I always have to insert a 'sleep 1' before I can do any LDAP operations. I've also noticed that on a sufficiently fast machine the time between S17slapd and S18krb5-kdc is short enough that the KDC can fail to start. I worked around it by adding 'invoke-rc.d krb5-kdc start' in /etc/rc.local but I'm sure a better solution is possible.

[Clint Byrum (clint-fewbar)]

Excerpts from Ryan Tandy's message of Wed Aug 17 17:29:36 UTC 2011:
> I have noticed that the slapd init script terminates before slapd is
> actually ready to accept connections, and I think that is the problem
> you're having too. In my scripts that stop/start slapd I always have to
> insert a 'sleep 1' before I can do any LDAP operations. I've also
> noticed that on a sufficiently fast machine the time between S17slapd
> and S18krb5-kdc is short enough that the KDC can fail to start. I
> worked around it by adding 'invoke-rc.d krb5-kdc start' in /etc/rc.local
> but I'm sure a better solution is possible.

Looking through slapd's code, it does in fact fork and exit before
activating its listener threads. The detach code needs to actually wait
for some message from the children that the listeners have started,
or the parent should do the listening before forking.

I filed bug #828237 to track this. Thanks for the tip Ryan!

[Clint Byrum (clint-fewbar)]

Excerpts from Clint Byrum's message of Wed Aug 17 10:56:55 -0700 2011:
> Excerpts from Ryan Tandy's message of Wed Aug 17 17:29:36 UTC 2011:
> > I have noticed that the slapd init script terminates before slapd is
> > actually ready to accept connections, and I think that is the problem
> > you're having too. In my scripts that stop/start slapd I always have to
> > insert a 'sleep 1' before I can do any LDAP operations. I've also
> > noticed that on a sufficiently fast machine the time between S17slapd
> > and S18krb5-kdc is short enough that the KDC can fail to start. I
> > worked around it by adding 'invoke-rc.d krb5-kdc start' in /etc/rc.local
> > but I'm sure a better solution is possible.
>
> Looking through slapd's code, it does in fact fork and exit before
> activating its listener threads. The detach code needs to actually wait
> for some message from the children that the listeners have started,
> or the parent should do the listening before forking.
>
> I filed bug #828237 to track this. Thanks for the tip Ryan!

FYI, bug 828237 is actually fixed in Oneiric, I didn't realize that
there was a patch to do just that included.

Not sure if its SRU'able to lucid, but the workaround of sleeping for
1 second after it starts is probably the best workaround at present.

[Ryan Tandy (rtandy)]

Thanks Clint for following up on that. I added the service-operational-before-detach patch from oneiric to my slapd and from initial testing it looks like it works as advertised. With that change (and the init scripts re-ordered) my kdc is now starting properly even on fast machines.

[Fede (beffa)]

Thanks you very much for the help! I've added "sleep 1" at the end of the
slapd init script and now everything starts fine.

[Timo Aaltonen (tjaalton)]

this is fixed at least in 16.04, from /lib/systemd/system/krb5-kdc.service.d/slapd-before-kdc.conf:

After=slapd.service