lightweight checkout over hpss is slow, many roundtrips

[Martin Pool]

'bzr checkout bzr+ssh://bazaar.launchpad.net/~launchpad/lp-source-dependencies/trunk download-cache --lightweight' does nearly all hpss-vfs operations, and does an unreasonably large number of round trips:

427.952 Transferred: 140628kB (328.7kB/s r:140607kB w:21kB)
427.952 return code 0
[13818] 2011-04-16 10:08:26.471 INFO: HPSS calls: 113 (103 vfs) SmartSSHClientMedium(bzr+ssh://<email address hidden>/)

the final tree is 88MB.

[Dimitrios Apostolou (jimis)]

Likewise, 'bzr checkout --lightweight lp:gcc' downloads almost 8GB of data while project data is less than 1GB (metadata excluded, since it's lightweight).

On the mailing list I got told that lightweight checkouts are not intended for network use, but for local checkouts. In that case light checkouts should be disabled for network protocols, or at least show a big fat warning to the user. It is *very* discomforting for someone to try "lightweight" checkout to be faster than heavy operations, only to wait 5h-6h without progress percentage being displayed to have a clue that it won't finish anytime soon.

I know now that lightweight checkouts are not intended for use over network, but my first impression was that it meant downloading only data, no metadata at all, with all logic being computed server side. I was actually expecting it to be as lightweight as wget...

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/06/2011 03:26 PM, Dimitrios Apostolou wrote:
> Likewise, 'bzr checkout --lightweight lp:gcc' downloads almost 8GB of
> data while project data is less than 1GB (metadata excluded, since it's
> lightweight).
>
> On the mailing list I got told that lightweight checkouts are not
> intended for network use, but for local checkouts. In that case light
> checkouts should be disabled for network protocols, or at least show a
> big fat warning to the user. It is *very* discomforting for someone to
> try "lightweight" checkout to be faster than heavy operations, only to
> wait 5h-6h without progress percentage being displayed to have a clue
> that it won't finish anytime soon.

The 8GB is a strict bug which has been fixed in bzr 2.4.
https://bugs.launchpad.net/bzr/+bug/737234

You weren't getting a good progress indication because that particular
code path wasn't supposed to be downloading that much data. (It still
downloads enough that we should be doing a progress bar, but it now does
~150MB rather than 8GB.)

The issue was that we have an in-memory cache for particular content,
the GCC tree was large enough to blow-out the cache, so instead of
hitting cache for everything, new requests which push out old content,
and we would re-read all content for every request. We changed the order
of reads to be in-order rather than cross-order, which both improves how
much data is downloaded, and performance across the board.

I certainly apologize that this wasn't fixed sooner. I won't say that
lightweight checkouts are still optimized for network performance, but
8GB was a bug which has been fixed.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk3EAW4ACgkQJdeBCYSNAAMgPACeIclAitChdHF/BevbqWET6a1j
eMYAoLVPBbx9OQKtuq+krTj1ugmBaUbL
=bt5O
-----END PGP SIGNATURE-----

[Vincent Ladeuil (vila)]

>>>>> Dimitrios Apostolou writes:

    > In that case light checkouts should be disabled for network
    > protocols, or at least show a big fat warning to the user.

Certainly not, many use cases involves a network, a *local* network
generally with low latency and high bandwidth, but still a network.

But anyway, if you got confused, others will too.

Can you tell us where we could fix the help/doc so you will have
realized sooner that the lightweight was referring to disk not to
network ?

[Martin Pool (mbp)]

Well I think just the name and probably the option help imply it will be a
fast history free checkout.
On May 6, 2011 4:56 PM, "Vincent Ladeuil" <email address hidden> wrote:
>>>>>> Dimitrios Apostolou writes:
>
> > In that case light checkouts should be disabled for network
> > protocols, or at least show a big fat warning to the user.
>
> Certainly not, many use cases involves a network, a *local* network
> generally with low latency and high bandwidth, but still a network.
>
> But anyway, if you got confused, others will too.
>
> Can you tell us where we could fix the help/doc so you will have
> realized sooner that the lightweight was referring to disk not to
> network ?
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/762330
>
> Title:
> lightweight checkout over hpss is slow, many roundtrips
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/bzr/+bug/762330/+subscribe

[Dimitrios Apostolou (jimis)]

I applied the patch from bug #737234 to bzr v2.3.1 and I can verify that the issue I was facing is fixed. Both lightweight checkout and stacked branch for lp:gcc now download ~540MB over only 20min, a huge improvement which makes these operations truly lightweight over network. Thank you for the fix.

The directories downloaded in both cases are 625MB size as reported by du. So I don't see an issue about using these operations over network any more.

On another note, only thing that remains for these operations to be as lightweight as wget, is reducing their memory usage which now peaks at 500MB.