If many queries come in too quickly, apache2 freezes.

[d1zzyg]

I have set up a Xeon-based server with a number of Subversion repositories, served through libmod-dav_svn on apache2, and using OpenLDAP to provide directory and authentication services.

Now for simple queries, for instance from a Windows desktop using TortoiseSVN, there is no problem with this. In fact it works beautifully. However when doing a more complex query involving multiple rapid accesses it causes all the free apache threads to be consumed, which all appear to be stuck in a loop waiting for a response from slapd, which has already been given. The Subversive plugin for Eclipse seems to cause this nearly every time.

Here's the kicker: when I turned on logging in slapd using loglevel 256 (for example), the problem goes away. Could it be that the request for info is fired off, then slapd responds before mod-dav_svn is ready for the response, whereas if it is slowed down by logging, the Apache module is ready in time? I'd have expected some sort of pipeline in there, but who knows?

Here's slapd.conf:

# This is the main slapd configuration file. See slapd.conf(5) for more
# info on the configuration options.

#######################################################################
# Global Directives:

# Features to permit
#allow bind_v2

# Schema and objectClass definitions
include /etc/ldap/schema/core.schema
include /etc/ldap/schema/cosine.schema
include /etc/ldap/schema/nis.schema
include /etc/ldap/schema/inetorgperson.schema
include /etc/ldap/schema/misc.schema

TLSCACertificateFile /etc/ldap/cacert.pem
TLSCertificateFile /etc/ldap/servercrt.pem
TLSCertificateKeyFile /etc/ldap/serverkey.pem

pidfile /var/run/slapd/slapd.pid
argsfile /var/run/slapd/slapd.args
loglevel 256
modulepath /usr/lib/ldap
moduleload back_hdb
tool-threads 1

sizelimit 500

backend hdb
database hdb
suffix "dc=netdb,dc=cark,dc=com"
rootdn "cn=admin,dc=netdb,dc=cark,dc=com"
rootpw passwerd
directory "/var/lib/ldap"
dbconfig set_cachesize 0 2097152 0
dbconfig set_lk_max_objects 1500
dbconfig set_lk_max_locks 1500
dbconfig set_lk_max_lockers 1500
index cn,sn,uid pres,eq,sub
index objectClass eq
lastmod on
checkpoint 512 30

replogfile /var/lib/ldap/replog

access to attrs=userPassword,shadowLastChange
        by dn="cn=admin,dc=netdb,dc=cark,dc=com" write
        by anonymous auth
        by self write
        by * none

access to *
        by dn="cn=admin,dc=netdb,dc=cark,dc=com" write
        by * read

And here's the setup for the repos:

LDAPTrustedGlobalCert CA_BASE64 /etc/apache2/cacert.pem

NameVirtualHost *:443
<VirtualHost *:443>
  ServerAdmin <email address hidden>
  ServerName netdb.cark.com
  SSLEngine On
  SSLCACertificateFile /etc/apache2/cacert.pem
  SSLCertificateFile /etc/apache2/servercrt.pem
  SSLCertificateKeyFile /etc/apache2/serverkey.pem
  SSLProtocol all
  SSLCipherSuite HIGH:MEDIUM
  LDAPTrustedMode TLS

  <Location /svn>
    Order allow,deny
    Allow from all
    SSLRequireSSL
    DAV svn
    SVNParentPath /home/svn/
    SVNListParentPath on
    AuthType Basic
    AuthName "Subversion Repository"

    AuthBasicProvider ldap
    AuthzLDAPAuthoritative off
    AuthLDAPUrl ldap://netdb.cark.com:389/ou=People,dc=netdb,dc=cark,dc=com?uid?sub TLS
    AuthLDAPGroupAttribute uniqueMember
    AuthLDAPGroupAttributeIsDN on
    Require ldap-group cn=subversion,ou=Group,dc=netdb,dc=cark,dc=com

# Other testing with files
# AuthUserFile /etc/apache2/svn_passwd
# AuthzSVNAccessFile /etc/apache2/svn_access
# Require valid-user
  </Location>
  ErrorLog /var/log/apache2/error.log

  # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
  LogLevel warn
  CustomLog /var/log/apache2/access.log combined
</VirtualHost>

[Chuck Short (zulcss)]

Which versions of apache and Ubuntu are you using?

Thanks
chuck

[d1zzyg (brianjamesgillespie)]

I'm using:

- Ubuntu 2.6.24-16-server SMP
- Apache 2.2.8-1ubuntu0.3 (apache2-mpm-worker)

Thanks
Bri

[d1zzyg (brianjamesgillespie)]

More info:

I had the same setup working fine on a single-core machine.

The problem shows up especially when using the Subversive plugin for Eclipse. Running top you have one, sometimes more than one apache processes running at 100% utilization. Could it be a deadlock? Switching on verbose debugging in slapd seems to reduce, but not eliminate the problem.

The only way I have found to get rid of the problem is to stop apache, kill the remaining apache processes, then restart apache. This obviously isn't much good to us.

Thanks
Bri

[d1zzyg (brianjamesgillespie)]

I have tried taking slapd out of the equation, using access and password files, and it works fine that way. My slapd version is 2.4.9-0ubuntu0.8.04, libapache2-svn is version 1.4.6dfsg1-2ubuntu1

Looking at apache's access logs, Eclipse is sending a slew of OPTION and PROPFINDs. I don't *think* the fault lies with slapd itself because accessing the repos via browser works, but further use of Eclipse/Subversive fails.

[Stefan Fritsch (sf-sfritsch)]

Some ideas to narrow down the problem:
- configure the ldap-status url and see if it gives any useful information while the problem appears
(see http://httpd.apache.org/docs/2.2/mod/mod_ldap.html )
- increase apache loglevel
- use netstat to watch the number of connections from apache to slapd
- possibly increase ldap cache sizes

[Patrick Brueckner (madmuffin)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. You reported this bug a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue for you. Can you try with the latest Ubuntu release? Thanks in advance.

[d1zzyg (brianjamesgillespie)]

Hi Patrick

Thank you for taking the time to report this bug and helping to make
> Ubuntu better. You reported this bug a while ago and there hasn't been
> any activity in it recently. We were wondering if this is still an issue
> for you. Can you try with the latest Ubuntu release? Thanks in advance.

I had to get something working sort of soon-ish at the time, since I was
using it for something at work; without a way forward in sight I had to
abandon using LDAP and use flat files instead.

I did briefly see another bug on Launchpad which was about mod_ssl causing
suspiciously similar sounding problems, so it might not be slapd as I was
thinking. My setup was serving svn over an https connection, and using slapd
for authentication.

I had it working in a test on a single-processor machine, but on a multi
Xeon processor server (an HP G-something, post-Compaq era, can't remember
what) the problem happened.

Cheers
Brian