bzr push 0.8 is REALLY slow

[Nicholas Allen]

I added some files to a repository that had already been pushed (with only 1 revision containing 2 files). I then added a bunch of files to the repositiory, committed and then did a push to push the added files to the server.
Bzr gave the following message for at least 15 minutes:

 [============== ] fetch phase 0/4

When I looked at my network connection it was indeed fetching a lot of data. I don't know what though as at this stage the repository on the server only contained 2 files each about 10 lines long! What data is it fetching?

I would have thought it would just send the new files I had added and it would be a quick operation.

[John A Meinel (jameinel)]

Nicholas Allen wrote:
> Public bug reported:
>
> I added some files to a repository that had already been pushed (with only 1 revision containing 2 files). I then added a bunch of files to the repositiory, committed and then did a push to push the added files to the server.
> Bzr gave the following message for at least 15 minutes:
>
> [============== ] fetch
> phase 0/4
>
> When I looked at my network connection it was indeed fetching a lot of
> data. I don't know what though as at this stage the repository on the
> server only contained 2 files each about 10 lines long! What data is it
> fetching?
>
> I would have thought it would just send the new files I had added and it
> would be a quick operation.
>
> ** Affects: bzr (upstream)
> Severity: Normal
> Priority: (none set)
> Status: Unconfirmed
>

If this is an existing repository (pre 0.8) then it probably is in Weave
format.

What does 'bzr info' say about the repository format?

Format:
...
    repository: Knit repository format 1

The weave format requires the entire file be replaced each time, which
means it has to download the old version to make sure we aren't missing
anything, create a new version, and upload it.
Knits are designed to have an index and an appendable data store, so we
only read the index, and then append new information to both.

If you are still using the Weave repository format, a 'bzr upgrade
--format=knit' should make things faster.

John
=:->

[Nicholas Allen (nick-allen)]

bzr info says that the format is:

Format:
       control: Meta directory format 1
  working tree: Working tree format 3
        branch: Branch format 5
    repository: Knit repository format 1

So it seems to be in knit format. I created the repository with version 0.8.

[Nicholas Allen (nick-allen)]

I have repeated the test with 0.8.2 and my upload never goes above 2kb/sec but I can ftp to the same server and transfer files at 30kb/s. Bzr also seems to be retreiving as much data from the server as it sends even if you add new files to an empty repository. Why does it do receive so much data during a push and why can't it use the full bandwidth of the connection?

[Nicholas Allen (nick-allen)]

I just tried transferring the exact same repository manually by sftp and it is also slow. Could it be that bzr stores the repository in many small files and this causes the ftp overhead to be quite large (sending file names back and forth)? If I transfer a single large file I get a fast data rate but with the bzr repository it requires receiving about as much data as is sent. My upload never goes above about 2kb/s and download is around 1.7 kb/s.

[John A Meinel (jameinel)]

There are some problems because of the number of files. But I've also seen that we have too many round trips. I have a branch which handles most of this, but it is blocked on some other changes.

However, the fixes should be in bzr.dev before 0.11 is released.

(The branches involved are transport-nonatomic-put and reduce-knit-churn)

[John A Meinel (jameinel)]

First fixes should be in bzr-0.11