loggerhead uses bdb

[David Allouche]

loggerhead, that is currently tested on codebrowse.launchpad.net, uses bdb to maintain its indexes.

bdb is notoriously fragile, and has the annoying habit of corrupting its database and stopping to work. The Subversion developers started using bdb but moved away from it because of this problem.

Today (2007-02-23) we had to clear the (painfully generated) loggerhead indexes because it was no longer working correctly and was spewing errors like the following:
Exception bsddb.db.DBRunRecoveryError: (-30977, 'DB_RUNRECOVERY: Fatal error, run database recovery -- PANIC: fatal region error detected; run recovery') in ignored

Before we can use loggerhead as a production service, we must replace bdb by something more robust than three pieces of cardboard assembled with spit and duct tape.

[David Allouche (ddaa)]

High priority. Code browsing is shiny and makes users happy.

[Robey Pointer (robey)]

i agree that bdb sucks but since loggerhead uses shelve, it ought to be using "anydbm", which translates to gdbm (i would assume). is that not the case? or does gdbm suck just as bad?

i've had good experiences with both sleepycat and sqlite, so those are potential alternatives.

[David Allouche (ddaa)]

I do not know how gdbm compares to bdb. In the last launchpad-bazaar meeting somebody suggested that the corruption we observe might be caused by faulty locking in the client code. If this is true (no opinion, really), then this probably is probably (just speculating) shared by all bdb clones.

In the same meeting, good things were said about sqlite. Actually, I ONLY ever heard good things about sqlite. The only downside I can see is that it "Supports databases up to 2 tebibytes", which could conceivably be a limitation in the forseeable future for codebrowse.launchpad.net since loggerhead apparently caches full-text diffs. But when we get there, we'll probably need to get creative anyway, and use a more heavy-weight DBM such as Postgres.

In short, I really have no clue. But sqlite has good karma among the launchpad developers.

[James Henstridge (jamesh)]

Note that anydbm picks "dbhash" in preference to "gdbm", which is Berkeley DB. I have no idea whether gdbm would be better or worse for this.

[Robey Pointer (robey)]

Loggerhead just uses shelve to store processed revision entries and
search keys, so it's always in the form (key => pickled blob). It
would probably make baby Tux cry to use a relational db like sqlite
to store those blobs, but if it would be more stable, it's worth trying.

I'm also holding out a tiny hope that the current bazaar
optimizations will end up making the revision cache a lot less
important, maybe even to the point where I could remove it
completely. The search index could be made relational anyway. Just
kinda thinking out loud...

robey

[Nicholas Allen (nick-allen)]

I ran into this problem too. Is this likely to change anytime soon?

[Michael Hudson-Doyle (mwhudson)]

Depends on your value of soon :)

Within the next few months, at least.

[Michael Hudson-Doyle (mwhudson)]

Getting somewhere with this now.

[Michael Hudson-Doyle (mwhudson)]

loggerhead RF revision 138.

[Michael Hudson-Doyle (mwhudson)]

We rolled out the no-more-bdb branch to codebrowse.launchpad.net, yay.

[Michael Hudson-Doyle (mwhudson)]

Released in 1.2.