rich-root stream spends too much time in heads()

Bug #487632 reported by Hakan Küçükyılmaz on 2009-11-24
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Wishlist
John A Meinel

Bug Description

bzr --version
Bazaar (bzr) 1.18.1

How to reproduce:

First way:
  bzr init-repo /tmp
  cd /tmp
  bzr branch lp:maria maria-local-master
  bzr upgrade --rich-root-pack

run time is 30 - 40 hours

Second way:
  bzr init-repo /tmp
  cd /tmp
  bzr upgrade --rich-root-pack
  bzr branch lp:maria maria-local-master

run time is again 30 - 40 hours

Related branches

Why are you trying to upgrade to rich-root-pack ? It's an outdated,
obsolete format. If you'd like to use a newer (faster) format, try 2a.

I am also using lp:ourdelta, which does not like the default format of bzr 1.18.1. "Upgrading" to rich-root-pack helped.
My colleague has the same long branch time when using:
    bzr init --1.9 . ; bzr branch lp:maria

I will test 2a, to see whether it works.

Best,

Hakan

On Tue, 2009-11-24 at 15:49 +0000, Hakan Küçükyılmaz wrote:
> I am also using lp:ourdelta, which does not like the default format of bzr 1.18.1. "Upgrading" to rich-root-pack helped.
> My colleague has the same long branch time when using:
> bzr init --1.9 . ; bzr branch lp:maria
>
> I will test 2a, to see whether it works.
2a has a superset of the features of rich-root-pack and is also faster.
The only disadvantage of 2a is that it doesn't work with bzr < 2.0.

Cheers,

Jelmer

Hi,

I think realistically it's better to do a coordinated upgrade to 2a once all the developers are on bzr 2.0 or later. This conversion could be faster, but we're unlikely to work on the speed of upgrade to old formats.

Changed in bzr:
status: New → Won't Fix

Now I tried to upgrade to 2a:

bzr 1.18.1

bzr init-repo /tmp
Shared repository with trees (format: pack-0.92)

time bzr branch lp:maria maria-local-master
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data. See "bzr help launchpad-login".
Branched 2765 revision(s).

real 12m59.216s
user 7m29.336s
sys 0m49.795s

time bzr upgrade --2a

(this is running for > 24 hours now)

Next I tried is to use bzr 2.0.2 (on a different machine)

bzr init-repo /tmp
Shared repository with trees (format: 2a)

This works:
time bzr branch lp:ourdelta ourdelta-local-master
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data. See "bzr help launchpad-login".
Branched 67 revision(s).

real 2m38.653s
user 0m1.513s
sys 0m15.658s

This ends with OOM:

time bzr branch lp:maria maria-local-master
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data. See "bzr help launchpad-login".
[- ] 4KB 0KB/s | Finding revisions
Killed##############|] 1619430KB 313KB/s | Fetching revisions:Inserting stream:Walking content 59458/59458

real 616m23.099s
user 421m24.961s
sys 11m52.201s

Conclusion up to now: I can't find a usable way for lp:maria and 2a

Changed in bzr:
status: Won't Fix → New

Switching status back to new, because upgrading to 2a format is unusable slow, too:

How to reproduce:
Version used: bzr 1.18.1

bzr init-repo /tmp
Shared repository with trees (format: pack-0.92)

time bzr branch lp:maria maria-local-master
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data. See "bzr help launchpad-login".
Branched 2765 revision(s).

real 12m59.216s
user 7m29.336s
sys 0m49.795s

time bzr upgrade --2a

(this is running for > 24 hours now)

On Wed, 2009-11-25 at 15:22 +0000, Hakan Küçükyılmaz wrote:
> Switching status back to new, because upgrading to 2a format is unusable
> slow, too:

 status wontfix

Hi Hakan. We've spent considerable time making the conversion the speed
that it is and do not plan to make it faster.

Well over 50% of the time of the import is extracting texts from the old
version: improvements there would only help the conversion and users
that haven't upgraded - we want to focus on helping users that are
upgrading.

I note that you are working on mysql branches, which are very big.

Upgrading that project should be handled with some coordination, as you
can prepare a single upgraded branch, which people can use to seed new
repositories and migrate much more quickly in that fashion.

Cheers,
Rob

Changed in bzr:
status: New → Won't Fix

For what it's worth, I have a 2a copy of MySQL 5.1 up at lp:~mnordhoff/mysql-server/mysql-5.1-2a (and 5 other branches, too). Starting from that, it takes me 40 minutes to convert Maria on my Athlon 64 X2 3800+.

(I might upload a copy of Maria, too.)

Matt Nordhoff (mnordhoff) wrote :

lp:~mnordhoff/maria/5.2-2a pushed.

But I saw something concerning and need to run "bzr check", which will take...um, a bit less than 36 hours.

To everyone else, I'm really sorry about all of this off-topic chatter. I'll stop now. I hope.

Oh, if you want, we could mark this wishlist. However I don't think
anyone will step up to improve it further - its hard, and we've spent
weeks on it already, and as I mention the relevance of improving this is
decaying with every day as people upgrade.

-Rob

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> Oh, if you want, we could mark this wishlist. However I don't think
> anyone will step up to improve it further - its hard, and we've spent
> weeks on it already, and as I mention the relevance of improving this is
> decaying with every day as people upgrade.
>
> -Rob
>

 status: confirmed
 importance: wishlist
 assigned: jameinel

I'll note that I found a bad code path, and fixing it can at least drop
the time in half. And we still have the 'xml8' patch which is another
20% or so.

Once I get back to work next week I'll post my findings. But it dropped
the time to convert mysql/cluster-6.4 about 3:1. (66k seconds down to
22k seconds.)

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksRiyEACgkQJdeBCYSNAAP8JQCcC4sQagJPnx1Ink24KcvcAxWe
o80An2dbrZJ7FWg+hfGaZu2j9qqzxIWy
=IjBh
-----END PGP SIGNATURE-----

Changed in bzr:
importance: Undecided → Wishlist
status: Won't Fix → Confirmed
John A Meinel (jameinel) on 2009-11-29
summary: - bzr upgrade --rich-root-pack is extremely slow
+ rich-root stream spends too much time in heads()
John A Meinel (jameinel) on 2009-11-30
Changed in bzr:
status: Confirmed → Fix Committed
assignee: nobody → John A Meinel (jameinel)
John A Meinel (jameinel) wrote :

I'm uploading the conversion improvents from the associated branch. The first graph is the time to convert 1000 revisions (as shown as the time between the "Auto-packing" statements in the log file.)

bzr.dev is the 'stock' bzr
xml8 has this patch
=== modified file 'bzrlib/xml8.py'
--- bzrlib/xml8.py 2009-07-07 04:32:13 +0000
+++ bzrlib/xml8.py 2009-11-30 03:40:50 +0000
@@ -433,10 +433,10 @@
                 pass
             else:
                 # Only copying directory entries drops us 2.85s => 2.35s
- # if cached_ie.kind == 'directory':
- # return cached_ie.copy()
- # return cached_ie
- return cached_ie.copy()
+ if cached_ie.kind == 'directory':
+ return cached_ie.copy()
+ return cached_ie
+ # return cached_ie.copy()

         kind = elt.tag
         if not InventoryEntry.versionable_kind(kind):

xml8+heads has the above patch and the associated branch

John A Meinel (jameinel) wrote :

I should clarify that the bottom graph is the 'cumulative' time for the conversion.
There is also a bit of noise in the 'xml8' entry, because I manually interrupted it from time to time to try to figure out why it was going slow. But the overall impact is quite clear.

John A Meinel (jameinel) on 2009-12-13
Changed in bzr:
milestone: none → 2.1.0b4
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers