bzr-explorer scans entire repo when opening a folder inside the repo

[Alexandre Garnier]

$ bzr init-repo repo
$ mkdir repo/stuff
$ mkdir repo/real-work

When opening repo/real-work, bzr-explorer walk through repo/stuff.
If this folder is big and/or deep (or is linked to a system folder in my case) the opening can be very long even if I don't care about other folders in the repo...

Seems to be something between bzrdir and transport.

Walking up the directory tree seems normal, but why walking down through siblings ?

[Alexander Belchenko (bialix)]

Alexandre Garnier пишет:
> Public bug reported:
>
> $ bzr init-repo repo
> $ mkdir repo/stuff
> $ mkdir repo/real-work
>
> When opening repo/real-work, bzr-explorer walk through repo/stuff.

How can you know this?

> If this folder is big and/or deep (or is linked to a system folder in my case) the opening can be very long even if I don't care about other folders in the repo...
>
> Seems to be something between bzrdir and transport.

Is there something suspicious in .bzr.log

> Walking up the directory tree seems normal, but why walking down through
> siblings ?

Can you check that repo/stuff is not listed as parent or submit location
for repo/real-work (see the output of bzr info).

I agree Explorer should not scan all branches in the shared repo if
you're opening only one branch. But I need more info on what's going on
and how you can see it.

[Alexandre Garnier (zigarn)]

  Le 27/08/2010 17:34, Alexander Belchenko a écrit :
> Alexandre Garnier пишет:
>> Public bug reported:
>>
>> $ bzr init-repo repo
>> $ mkdir repo/stuff
>> $ mkdir repo/real-work
>>
>> When opening repo/real-work, bzr-explorer walk through repo/stuff.
> How can you know this?

First it was because I got an encoding error from nowhere :
File
"/users/garniera/system/boulard/lib/python/bzrlib/transport/local.py",
line 163, in get
     return osutils.open_file(path, 'rb')
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc9' in
position 30: ordinal not in range(128)

So I modify to log 'path' to find where it was failing and I got logs of
all subfolders of repo.

Then when trying to reproduce in an other sandbox place (and environment
to avoid encoding problem from Xming connection), opening repo/real-work
opens repo.

>> If this folder is big and/or deep (or is linked to a system folder in my case) the opening can be very long even if I don't care about other folders in the repo...
>>
>> Seems to be something between bzrdir and transport.
> Is there something suspicious in .bzr.log

Nothing, just logs about launching bzr explorer.

>> Walking up the directory tree seems normal, but why walking down through
>> siblings ?
> Can you check that repo/stuff is not listed as parent or submit location
> for repo/real-work (see the output of bzr info).
>
> I agree Explorer should not scan all branches in the shared repo if
> you're opening only one branch. But I need more info on what's going on
> and how you can see it.

In fact, repo/real-work is not a branch, just a folder containing
branches. And there is no connection between repo/real-work (or
sub-branches) and repo/stuff.
When opening repo/real-work/my_branch (a real branch), no problem.

I'll try to create better use-case with logs and everything at home tonight.

--
Alex

[Alexandre Garnier (zigarn)]

Here is a complete use-case (maybe not relevant but reproducing my problem) :

Initialization :

/tmp $ bzr init-repo repo
Shared repository with trees (format: 2a)
Location:
  shared repository: repo
/tmp $ cd repo
/tmp/repo $ mkdir datas
/tmp/repo $ bzr init datas/dev_datas
Created a repository tree (format: 2a)
Using shared repository: /home/alex/tmp/repo/
/tmp/repo $ mkdir work
/tmp/repo $ cd work/
/tmp/repo/work $ bzr init branch
Created a repository tree (format: 2a)
Using shared repository: /home/alex/tmp/repo/
/tmp/repo/work $ bzr co branch/ checkout

Launching bzr explorer :
$ bzr explorer
Ctrl+o --> /tmp/repo/work/

Result : it opens the whole repository /tmp/repo and not just the folder /tmp/repo/work/ (cf. attachment)

Logs when printing 'path' at line 163 of bzrlib/transport/local.py :
/home/alex/.bzr/branch-format
/home/.bzr/branch-format
/.bzr/branch-format
/tmp/repo/work/.bzr/branch-format
/tmp/repo/.bzr/branch-format
/tmp/repo/.bzr/checkout/format
/tmp/repo/.bzr/branch/format
/tmp/repo/.bzr/repository/format
/tmp/repo/.bzr/branch-format
/tmp/repo/.bzr/.bzr/branch-format
/tmp/repo/.bzr/README/.bzr/branch-format
/tmp/repo/.bzr/branch-format/.bzr/branch-format
/tmp/repo/.bzr/branch-lock/.bzr/branch-format
/tmp/repo/.bzr/repository/.bzr/branch-format
/tmp/repo/.bzr/repository/format/.bzr/branch-format
/tmp/repo/.bzr/repository/indices/.bzr/branch-format
/tmp/repo/.bzr/repository/lock/.bzr/branch-format
/tmp/repo/.bzr/repository/obsolete_packs/.bzr/branch-format
/tmp/repo/.bzr/repository/pack-names/.bzr/branch-format
/tmp/repo/.bzr/repository/packs/.bzr/branch-format
/tmp/repo/.bzr/repository/shared-storage/.bzr/branch-format
/tmp/repo/.bzr/repository/upload/.bzr/branch-format
/tmp/repo/datas/.bzr/branch-format
/tmp/repo/datas/dev_datas/.bzr/branch-format
/tmp/repo/datas/dev_datas/.bzr/branch/format
/tmp/repo/datas/dev_datas/.bzr/repository/format
/tmp/repo/datas/.bzr/branch-format
/tmp/repo/.bzr/branch-format
/tmp/repo/.bzr/repository/format
/tmp/repo/datas/dev_datas/.bzr/branch/branch.conf
/tmp/repo/datas/dev_datas/.bzr/branch/format
/tmp/repo/datas/dev_datas/.bzr/branch/branch.conf
/tmp/repo/.bzr/repository/pack-names
/tmp/repo/datas/dev_datas/.bzr/branch/branch.conf
/tmp/repo/work/.bzr/branch-format
/tmp/repo/work/branch/.bzr/branch-format
/tmp/repo/work/branch/.bzr/branch/format
/tmp/repo/work/branch/.bzr/repository/format
/tmp/repo/work/.bzr/branch-format
/tmp/repo/.bzr/branch-format
/tmp/repo/.bzr/repository/format
/tmp/repo/work/branch/.bzr/branch/branch.conf
/tmp/repo/work/branch/.bzr/branch/format
/tmp/repo/work/branch/.bzr/branch/branch.conf
/tmp/repo/.bzr/repository/pack-names
/tmp/repo/work/branch/.bzr/branch/branch.conf
/tmp/repo/work/checkout/.bzr/branch-format
/tmp/repo/work/checkout/.bzr/branch/format
/tmp/repo/work/checkout/.bzr/repository/format
/tmp/repo/work/.bzr/branch-format
/tmp/repo/.bzr/branch-format
/tmp/repo/.bzr/repository/format
/tmp/repo/work/checkout/.bzr/branch/bran...

[Alexander Belchenko (bialix)]

Ok, so it's not quite the bug per se. It's a combination of intended behavior and another bug in explorer (slow on open).

So, when you trying to open directory `work` inside shared repository `repo` Explorer tries to open either branch or repo in `work` directory. But there is no branch nor repo, so bzr search upwards. It found repo in the `../repo`. So here everything is correct.
But then explorer sees shared repo and trying to find *all* branches inside this repo. And therefore it walks down in all directories inside `repo` to be able to show you content of shared repository. The search method (inside bzrlib) is not very optimal IMO.

So to understand how to fix it we need to decide what should be a correct behavior.

[Alexandre Garnier (zigarn)]

From my point of view, when I open 'repo/work' I would like to open only this folder, not all the repo.
--> Explorer behavior should be to hide everything outside the folder I want to open.

But I'm still wondering why opening a subfolder of a repo should scan *all* branches inside the repo. Isn't it possible to filter the scan on the subfolder ? If I open this folder and not the repo it's because I don't care about other branches of the repo.
Opening a branch inside a repo doesn't scan the repo...

[Alexander Belchenko (bialix)]

Alexandre Garnier пишет:
>>From my point of view, when I open 'repo/work' I would like to open only this folder, not all the repo.
> --> Explorer behavior should be to hide everything outside the folder I want to open.

Makes sense. But it's hard to read the user mind without optional python
module named "telepathy" :-) What if the user actually *want* open
entire repo and just missed the right folder?

> But I'm still wondering why opening a subfolder of a repo should scan *all* branches inside the repo.

Because that's how repo.find_branches() method currently implemented in bzr.

 > Isn't it possible to filter the scan on the subfolder ?

We need to write our own find_branches() implementation, I guess.

 > If I open this folder and not the repo it's because
 > I don't care about other branches of the repo.

See my the first comment. Sometimes you don't but sometimes you do.

> Opening a branch inside a repo doesn't scan the repo...

Because in this case bzr already has a branch and therefore explorer can
  open only it. There is 3 modes to open location in the explorer:

1) open branch/checkout -> no scan performed
2) open shared repo (with upward searching) -> scan for all nested
branches, checkouts, etc performed
3) open the plain directory without shared repo -> scan for all nested
branches, checkouts, repositories performed.

Your case is 2. The case 3 could be even slower, I guess.

I have a plan to change how the explorer opens locations. I want to put
the open process in the background to keep the UI responsive. Actually I
want to extract all accesses to bzr commands from explorer into separate
process (1 process for each branch/repo), something like Chrome browser
does. But this is the long term plan, and will require a lot of work.

I will try to invent some workaround for your situation in the short
term. Or maybe you have in mind something and going to write a patch?

[Alexandre Garnier (zigarn)]

> What if the user actually *want* open entire repo and just missed the right folder?

It opens the folder. If it's not the right one, the user can then redo
the opening action.

> We need to write our own find_branches() implementation, I guess.

Or maybe change it to filter on really opened folder when going upward
to find shared repo.

> I will try to invent some workaround for your situation in the short
> term. Or maybe you have in mind something and going to write a patch?

No idea at the moment, and not enough python/bzr skills for a patch...
(And no time at work for this work problem)

My real problem is that we use symlinks but work on Windows, and because
of that the shared repo is the home folder in which we have other
symlinks to libraries and some datas (large and deep tree).
When opening my ~/work folder, it scans all my home and follows symlinks
and finally opens all the shared repo with my ~/bazaar/plugins/* and
other useless things like that.
I'll try to find an other branch organization compatible with our work
process and systems (in)compatibilities.