bzr-svn stores sqlite db in ~/.bazaar => REALLY SLOW on NFS

[James Y Knight]

My homedir is on NFS. Bzr-svn apparently stores a sqlite database in ~/.bazaar.

While this appears to have been working, it took over 10 minutes to get through 39k (out of 130k) revisions in the "analyzing repository layout" stage. I believe this is because of the excessive numbers of F_SETLK operations that were being done on the database file. I imagine (not an NFS protocol hacker) each one of those requires a synchronous roundtrip to the NFS server and is not too highly optimized.

In contrast, copying the whole 350MB ~/.bazaar to a local disk and symlinking ~/.bazaar to it took about 10s, and then after restarting the bzr branch operation, doing the whole "analyzing repository layout" took less than 30s.

I conclude that either you should use sqlite in some more efficient way, or else not store these files in my homedir. Howabout putting it in the bzr checkout directory instead?

PS: Grey text for this bug entry field? WTF? I had to use an external editor to write this so I wouldn't go blind during the process. Not Good UI.

[James Y Knight (foom)]

(sorry, wrong submitted to wrong project)

[Jelmer Vernooij (jelmer)]

Hi James,

On Wed, 2007-08-08 at 01:35 +0000, James Y Knight wrote:
> My homedir is on NFS. Bzr-svn apparently stores a sqlite database in
> ~/.bazaar.
>
> While this appears to have been working, it took over 10 minutes to get
> through 39k (out of 130k) revisions in the "analyzing repository layout"
> stage. I believe this is because of the excessive numbers of F_SETLK
> operations that were being done on the database file. I imagine (not an
> NFS protocol hacker) each one of those requires a synchronous roundtrip
> to the NFS server and is not too highly optimized.
>
> In contrast, copying the whole 350MB ~/.bazaar to a local disk and
> symlinking ~/.bazaar to it took about 10s, and then after restarting the
> bzr branch operation, doing the whole "analyzing repository layout" took
> less than 30s.
>
> I conclude that either you should use sqlite in some more efficient way,
> or else not store these files in my homedir. Howabout putting it in the
> bzr checkout directory instead?
I guess we could make it configurable, but I think the homedirectory is
the sanest default. The data in this directory is basically a cache and
it is specific for a particular Subversion repository. If we'd put it in
a bzr checkout directory that means it would have to be regenerated for
each bzr repository that uses it.

An alternative would be to not use sqlite but some format optimized
specifically for Bazaar so we wouldn't have to do that locking. Coming
up with such a format would be a significant effort though.

  status confirmed

Cheers,

Jelmer

[Jelmer Vernooij (jelmer)]

  importance low
  status triaged
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

[Jelmer Vernooij (jelmer)]

This is now at least partially fixed - file id maps are no longer stored
in sqlite but using knits.

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

[James Y Knight (foom)]

The revid map still stored in the sqlite database is still causing a big speed hit. I suspect that if everything was done in one (or a few) transactions it'd speed up sqlite on nfs a lot.

[Jelmer Vernooij (jelmer)]

The sqlite database will hopefully disappear in 0.4.10.