Refresh branch state takes too long time

[Alexander Belchenko]

I have been experimenting with a large project branch.
The whole project contains several source code projects, some of which have been deleted from this test branch.

I recently updated QBZR and Bazaar Explorer. Now when I start or refresh, the program hangs for around 4 minutes then shows the lower message ' Branch has changes not present in its submit branch' followed by a list of ~5 thousand deleted files and a few modified files. However the top message 'Working tree is up to date at revision #' and there is nothing to commit.

It is quite possible my branch is not configured correctly. I just don't expect to keep seeing the files I've deleted or get the massive pause on each refresh. Whatever version software I had previously been using, I'm sure it did not behave like this.

This is intended to be a general question since I'm confused as to what is going on here.

Here are the program versions anyway:
bzrlib 2.0.0
bzrtools 2.0.0
exporer 0.10.0
qbzr 0.17.0
rebase 0.5.3
Python 2.5.4
PyQt 4.4.3
Qt 4.4.1

[Alexander Belchenko (bialix)]

As a workaround edit .bzr/branch/branch.conf file and remove line that started with

submit_branch = ...

[Greg (gregspecialsource)]

removing 'submit_branch = ...' from both source and destination branches had no effect on the 4min delay or the 'Branch has changes not present in its submit branch' message and pseudo change list.
No other references between the two branches exist (that I could find).

[Alexander Belchenko (bialix)]

Greg пишет:
> removing 'submit_branch = ...' from both source and destination branches had no effect on the 4min delay or the 'Branch has changes not present in its submit branch' message and pseudo change list.
> No other references between the two branches exist (that I could find).
>

Something really wrong here. We need more info how to reproduce this
problem.

Does your project is open source?

[Greg (gregspecialsource)]

I'm afraid the project is not open source.
The behavior I am seeing now only occurred after updating QBZR and Bazaar Explorer from the global release 2.0.
I updated QBZR in order to test a bug for that project and had to update Bazaar Explorer also due to compatibility issues. I have not updated any other modules / plugins.

The main steps I took were:
1) Branch from a large project with thousands of files
2) Delete a few thousand files from the new branch
3) Test updating 2 source code files between the two branches, back and forth
4) Update qbzr, explorer from official v2.0 file set.

The parent branch behaves fine, the new branch behaves strangely as described earlier.

[Greg (gregspecialsource)]

I just uninstalled bazaar and explorer and reinstalled v2.0.
The issue went away. So definitely related to something new.

[Ian Clatworthy (ian-clatworthy)]

Greg wrote:
> I just uninstalled bazaar and explorer and reinstalled v2.0.
> The issue went away. So definitely related to something new.
>

Bazaar Explorer is effecting doing "bzr status -rsubmit:..-1" in that
feature branch. Try running that at the command line please. How long
does it take?

Ian C.

[Greg (gregspecialsource)]

Just reinstalled Bazaar Explorer v0.10.0...
Executing "bzr status -rsubmit:..-1" logged (I presume) the same list of deleted and modified files in ~2.5 seconds.
Opening the same project in Bazaar Explorer took ~4 minutes.

[Craig Hewetson (craighewetson-deactivatedaccount)]

I've noticed that "refresh action" seems to hold up the UI thread, can't this action rather be run in the background?
Also can't the progress bar be shown or some indication that the action is busy (maybe status view shows a message "Reloading..." etc) while the refresh action is running?

I experience similar performance and UI thread hold ups in the embedded qbrowse widget.

[Ian Clatworthy (ian-clatworthy)]

I wonder if I should make the "submit changes" panel an optional feature rather than always turned on. It would be good to get to the bottom of the slowness regardless.

[Ian Clatworthy (ian-clatworthy)]

I've looked into adding a preference here to control whether the submit report is included or not. I think it will be better to make the set of reports configurable rather than make special preferences to enable/disable each report. I'm pretty concerned about the performance issue though. I guess having thousands of files changed in a branch is not a common occurrence though.

[Greg (gregspecialsource)]

I must admit I haven't got a clue what you are talking about with reports, special preferences etc. :)
I just see a bunch of changes that I don't want to see and get a long pause for some reason.
I have to comment that branches containing thousands of files were quite common with my previous employers projects.

[Greg (gregspecialsource)]

Clarifying last comment... I meant changes / additions / deletions of thousands of files, not just lots of files themselves.

[Ian Clatworthy (ian-clatworthy)]

To explain my comment re "reports", my vision is for the bottom half of the repository view and the bottom half (third?) of the status view to show useful "reports". Right now:

* the repo view has 3 of these: Recent History, Local Changes and Missing Revisions
* the status view has one: Submit Delta.

The Submit Delta report only appears for feature branches too.

Looking forward, the idea is that the user will be able to say, for each report, whether they want to see it on each view or not. So if you want to see Recent History on the status view, that will be easy to configure. Furthermore, developers will be able to add their own reports defined in XML. The decision as to where to render those custom reports, if anywhere, will be left to the user to configure. Make sense?

I'm curious about the frequent use of branches with 1000s of changes. What sort of data were you managing?

[Greg (gregspecialsource)]

Ahh! Thank you for explaining that. I was wondering what that bottom window was doing and why it sometimes showed those file lists.

In that case, selecting reports when needed makes sense and investigating any poor performance is also great.

As for 'frequent uses of branches with 1000s of changes'. Perhaps I exaggerate a little, but those were projects related to video games and CAD. There was a combination of individual developers with their own infrequently merged branches and periodic very large asset changes.

By the way, you are doing a top job Ian :)

[Greg (gregspecialsource)]

Just installed v2.1 which includes BZR Explorer v1.0rc1
Opening this particular project takes the usual several minutes. I see a progress bar visible in the status region however this bar never showed any progress during the load period. (The bar was empty the whole time.)
Just reporting in, I do not know if you have made changes regarding this issue.

[Greg (gregspecialsource)]

I did mention this in the first post, but the cause of this issue is the fact that I branched a multi project solution, and with the new branch, deleted a few sub projects totaling 5800 files.

Executing "bzr status -rsubmit:..-1" from the command line takes no significant time, but the explorer reports window still takes the 4 minutes per refresh, making the whole application unusable for the time being.

Just updated to v1.0 final and there is no change, which is expected since no fix was logged.

[Greg (gregspecialsource)]

I could possibly privately share my file-set, or better yet run a instrumented build to profile or log what is going on, if you want to provide such a build.

[Ian Clatworthy (ian-clatworthy)]

Hi Greg,

Collecting some profiling information would be great. Here's what to do ...

1. Change to the directory holding the branch.
2. Run this command ...

   bzr --lsprof-file callgrind.out.500072 explorer .

3. Wait for the output to appear then close the application.
4. Attach the callgrind.out.500072 file to this bug report.

Thanks.

[Greg (gregspecialsource)]

I launched with that command, opened the slow project then closed the application.
I've attached the output as callgrind.out.zip

[Ian Clatworthy (ian-clatworthy)]

Hi Greg,

Thanks for the profiling data. It's not telling me much unfortunately. I'm going to be out of the office for most of the next week so I'd like to get things working for you until I can dig deeper. Let's switch off the "Submit delta" report altogether and see if that addresses things. To do that, you'll need to patch the lib/view_workingtree.py file. Around line 40, immediately after this line:

    self.detail_views = []

add this line:

    return

Restart the app and see if things improve or not.

Thanks,
Ian C.

[Greg (gregspecialsource)]

That work around worked. The bottom window was gone and the slow project opened instantly.

[Matt Wilkie (maphew)]

I have this same problem. The code I'm using is public: lp:~leo-editor-team/leo-editor/contrib. Screenshot of what I see attached.

[Alexander Belchenko (bialix)]

Some improvements re speed of refresh landed to trunk. Please test.

[Alexander Belchenko (bialix)]

As analisys of https://bugs.launchpad.net/bzr-explorer/+bug/779529 reveals there is very poor performance of qt widget we're using for status/submit panes when we need to show many files with many links to different actions.

Can you confirm that you have worse performance when there are many files changed in your tree?