2a fetch is very cpu intensive

Bug #562666 reported by Michael Hudson-Doyle
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Bazaar
Confirmed
High
Unassigned
Breezy
Triaged
Medium
Unassigned

Bug Description

Because 2a fetch validates and filters the data it transfers, it is uses a lot of cpu -- for example, fetching an imported kernel (over a high bandwidth low latency link) uses about an hour of cpu time. git does it in about 20 seconds.

There are reasons for done the validation and filtering, but perhaps there's a place for a dumb copy? I think Something Needs To Be Done, in any case.

Martin Pool (mbp)
Changed in bzr:
status: New → Confirmed
importance: Undecided → High
tags: added: 2a fetch performance
Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 562666] [NEW] 2a fetch is very cpu intensive

On Tue, 2010-04-13 at 23:55 +0000, Michael Hudson wrote:
> Public bug reported:
>
> Because 2a fetch validates and filters the data it transfers, it is uses
> a lot of cpu -- for example, fetching an imported kernel (over a high
> bandwidth low latency link) uses about an hour of cpu time. git does it
> in about 20 seconds.

It would be good to validate that this is actually a full fetch and sha
validate by git; it may not be.

> There are reasons for done the validation and filtering, but perhaps
> there's a place for a dumb copy? I think Something Needs To Be Done, in
> any case.

I agree; in particular the use case Michael cares about here is fetching
over a LAN to make a temporary branch copy to work on. So local disk
'tricks' like hardlinking won't help at all: we need to make actual
fetch more efficient.

-Rob

Revision history for this message
John A Meinel (jameinel) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Hudson wrote:
> Public bug reported:
>
> Because 2a fetch validates and filters the data it transfers, it is uses
> a lot of cpu -- for example, fetching an imported kernel (over a high
> bandwidth low latency link) uses about an hour of cpu time. git does it
> in about 20 seconds.
>
> There are reasons for done the validation and filtering, but perhaps
> there's a place for a dumb copy? I think Something Needs To Be Done, in
> any case.
>
> ** Affects: bzr
> Importance: Undecided
> Status: New
>

Note that moreso than just validating the data, it is probably
triggering the 'repack-on-the-fly' code, and extracting everything
(should be fairly cheap) and re-compressing it (generally pretty expensive).

This could also be improved using fetch_order='unordered' and weakening
the 'needs-repacking' check.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkvFL8kACgkQJdeBCYSNAAOf+gCgnmGVBZXyfW9349XoUJsbjmTD
A58An02Fe9P7c5RCIhFHbrI9GR9T3jiU
=E/ra
-----END PGP SIGNATURE-----

Revision history for this message
William Grant (wgrant) wrote :

A checkout of the above imported kernel over my 10Mbps high-latency Internet connection took a little under four hours. Even a local branch is slow -- an lsprof'd run (output attached) took 96 minutes, while a non-lsprof'd one looks like it will take around an hour (it's not done yet, though).

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 562666] Re: 2a fetch is very cpu intensive

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

William Grant wrote:
> A checkout of the above imported kernel over my 10Mbps high-latency
> Internet connection took a little under four hours. Even a local branch
> is slow -- an lsprof'd run (output attached) took 96 minutes, while a
> non-lsprof'd one looks like it will take around an hour (it's not done
> yet, though).
>
> ** Attachment added: "lsprof of local kernel branch"
> http://launchpadlibrarian.net/44115461/lsprof
>

Note that lsprof is showing that the bulk of the time is spent
processing the inventory pages, looking for (file_id, revision_id) keys
that need to be fetched.

Specifically, it is claiming that of the 5.8k seconds to run the whole
branch op, 4.2ks are in _filter_text_keys, w/ 1.7k spent deserialising
the pages to pull out the text keys.

There seem to be 1.4M pages, which triggers 110M text key checks.

Some of that time we can't avoid. However, I'm also guessing that a huge
part of that time is the overhead of hitting so many text keys that the
gc overhead becomes large. (an expense partially caused by fetching
everything-at-once.)

I don't know that there is a lot of tuning we can do here, but there
would probably be some.

The other issue is that probably a lot of the keys are duplicates, so we
end up spending a fair amount of time determining that.

One option might be to store a 'text_keys' like set with the serialized
form, and have a quicker check to see if the serialized form has already
been read, thus avoiding deserializing...

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkvFUMEACgkQJdeBCYSNAAOL+ACgywYy7226U/sPNWh0NGONvWu1
9zAAn3elUU3gf0Mi+eCrSHCsV8XO62/X
=kzgY
-----END PGP SIGNATURE-----

Jelmer Vernooij (jelmer)
tags: added: check-for-breezy
Jelmer Vernooij (jelmer)
tags: removed: check-for-breezy
Changed in brz:
status: New → Triaged
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.