nautilus integration too slow in large trees

[mati]

I don't know if it's actually Nautilus' fault or bzr-gtk's, but navigating through a large project is painfully slow. Either way, in order for Bazaar to take over the world, the culprit should be fixed.

Also: why does it require to manually copy files to user's home directory? It should be done automatically.

[JeffV (jeff-launchpad-tanasity)]

I find it unusably slow too, taking minutes to navigate between directories when browsing.

But in my case, this may not be directly attributable to Olive....

I have a Virtualbox VM with Ubuntu Hardy Heron running on it. This acts as a development server, mimicing a live environment. The files on this Virtualbox are under Bazaar control. The project consists of about

To make changes to the code I mount the development directory using sshfs, and alter the code. Obviously sshfs is an overhead that is redundant since the Virtualbox is on the local machine. This may be what is slowing Olive down so much. Or it could be that Olive is hugely inefficient over networks, even if it is a virtual network.

I'm running a Q6600 quad core Intel with 8GB memory. Other operations with the virtual machine are speedy. The virtual machine has 2GB of the 8 dedicated to it.

I have just under 15,000 files under Bazaar, stored in about 900 directories.

[Martin Pool (mbp)]

So the complaint is that when run on a branch mounted over sshfs? Is
it adequate when running locally? How about running Olive inside the
vm development environment?
--
Martin <http://launchpad.net/~mbp/>

[mati (mati-wroc)]

Martin, for me it's slow locally. It's a ~600 MB project. If I can get any useful information, let me know what it would be.

[Martin Pool (mbp)]

2009/12/2 mati <email address hidden>:
> Martin, for me it's slow locally. It's a ~600 MB project. If I can get
> any useful information, let me know what it would be.

A good way start is to run say

 bzr --lsprof-file=/tmp/lsprof.out viz

and attach the trace file to a bug.

--
Martin <http://launchpad.net/~mbp/>

[JeffV (jeff-launchpad-tanasity)]

Martin, re comment #2: I can't run it in the VM - it's a server VM almost exactly matching the live server: there's no X installed on it.

I don't think it's sshfs that's at fault, but I have very little proof of it. I suspect, because of reports about bzr, that it's something about moving bzr data across a network. Or possibly that bzr is inefficient at diff on a large project.

But this is only a suspicion - I've not done any proper measurements to test this, and one reason it might not be this is that in the case of this VM, there's no real network since the VM is on the physical machine that I'm using to develop. But the traffic does go through the network stack, as far as I know. Though I'm not sure how the final step of the transfer of the network data from the VM to the physical machine happens, but I'm guessing it's through shared memory.

Anyhow did a brief, unscientific experiment. I installed bzr on the server and ran bzr diff on the project. At this point there are probably about 50 files amongst the 15,000 that have been changed. After about 5 minutes I killed it. This wasn't a very fair test - there was a lot running on this machine at the same time, and I think that it ended up using swap. But this suggests that the problem isn't Olive, or the network, but bzr diff.

When I have a less stressed machine, I'll try again.

[Martin Pool (mbp)]

Thanks for following up Jeff.

So to progress this I think we need to work out if the bug is:

- bzr on a large tree and a thrashing machine is just slow (not really a bug)
- bzr over sshfs is slower than local - not necessarily our bug, but might be
- bzr diff is unreasonably slow
- bzr diff performance is ok but bzr-gtk and/or bzr nautilus integration is slow
- bzr gtk should cope better when reading the directory is very slow,
by eg doing it in another thread

--
Martin <http://launchpad.net/~mbp/>

[mati (mati-wroc)]

Martin, I got:
bzr: ERROR: no such option: --lsprof-file

and found nothing in the docs which could help me launch it.

But I found a quick way to reproduce this behaviour:
mkdir test
cd test
dd if=/dev/urandom of=0 bs=1k count=1
for a in `seq 1000`; do cp 0 $a; done
bzr init
bzr add
Now open the folder in nautilus and watch all nautilus windows freeze for a long time (depending on urandom output, I got several seconds time on a fast, dual core machine with ext4 fs).

I hope that will help, otherwise I'm here to provide more useful info :)

[Martin Pool (mbp)]

2009/12/4 mati <email address hidden>:
> Martin, I got:
> bzr: ERROR: no such option: --lsprof-file
>
> and found nothing in the docs which could help me launch it.

It's not in 'bzr help global-options'?

>
> But I found a quick way to reproduce this behaviour:
> mkdir test
> cd test
> dd if=/dev/urandom of=0 bs=1k count=1
> for a in `seq 1000`; do cp 0 $a; done
> bzr init
> bzr add
> Now open the folder in nautilus and watch all nautilus windows freeze for a long time (depending on urandom output, I got several seconds time on a fast, dual core machine with ext4 fs).

That's great, but it doesn't really answer my question in #6. I
suppose if someone works on this bug they can dig down through there.

--
Martin <http://launchpad.net/~mbp/>

[mati (mati-wroc)]

I'm sorry, I didn't dig much. I'm attaching results from
bzr --lsprof-file prof.out viz

but I think it won't be worth much, as the viz window was empty and there was error message:
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/gtk/viz/branchwin.py", line 394, in _treeselection_changed_cb
    self.revision_menu.set_revision_ids([revision.revision_id])
AttributeError: 'NoneType' object has no attribute 'revision_id'
Profile data written to "prof.out".

[mati (mati-wroc)]

Ok, after the commit it worked well (and was fast).

Bug 287988 include some hing in comment #2 ("Unfortunately it's painfully slow on big working trees, because of Nautilus' polling mechanism.")

[Jelmer Vernooij (jelmer)]

FWIW, nautilus-bzr is now a separate binary package in Debian and
Ubuntu.

I'm also in contact with the rabbitvcs developers, they're keen on
integrating nautilus-bzr in the long run.

[Jorge Q (jquiroga2005)]

Yes, the same happens to me but with Olive (bzr-gtk). My environment is two real machines into LAN, the repository is on Windows XP SP3 and and I need a copy into the other that runs Ubuntu 10.10.

With the gtk tool, that mimics Tortoise, I selected "Branch..." into new and empty directory. I use bzr+ssh and ask me twice the password (a weird thing should be only one) then all Nautilius windows freeze and the Olive window too and I cannot kill the process with the Force Quit applet.

Another little thing is if I have a typo error in the Branch location Olive doesn't complaint, and should say anything like "Branch location doesn't exist".

I attached the output I got when I try to do the same from command line (and Olive previously freezed). In some occasion bzr branch works from the CLI (and ask only once my ssh password), then Olive wake up as if Olive would have done the process and show a success message. So I think that is something wrong in bzr-gtk and not in the network or protocol used because it was a really successful operation, that unfortunately I cannot reproduce after :-(.

In Windows everything goes Ok in Ubuntu I have installed this bzr-related stuff (obtained from the Ubuntu Software Center)
- Bazaar explorer (but before I installed the problem was present)
-bzr-search
-bzr-fastimport
-bzr-svn
-bzr-upload
-bzr-rewrite
-bzr-email
-loggerhead
-nautilus-bzr
-bzr-loom
-bzr-doc
-bzr-gtk
-bzr-pipeline
-bzr-dbus
-qbzr (Installed when installed bazaar explorer)
-qct

My Ubuntu and Windows are fully updated and Windows has an OpenSSH server (Yes the server should be Ubuntu but I'm working on it).

I expect that the file really helps. If something else you need, please ask me. As a suggestion try to do the same thing with a big directory on a fresh installed Ubuntu to try to reproduce the problem.

Thanks in advance.