newly converted repositories are not properly packed

[John A Meinel]

Symptoms
========

A newly converted repository has more than one pack and not as much space savings as anticipated.

Cause
=====

A bug inside the 2a format code when an autopack occurs meant that the packs written during autopack were not selected for later repacking at the end of the conversion.

Workaround
==========

Running 'bzr pack' will cause a full pack to take place.

[John A Meinel (jameinel)]

Post 'bzr pack' I get:
101M .bzr/repository/obsolete_packs
1.0K .bzr/repository/pack-names
8.8M .bzr/repository/indices
30M .bzr/repository/packs

In other words, I was correct about expecting ~30MB of packed content. As the size dropped from 101MB => 38.8MB (packs + indices).

Do we just need to ask a LOSA to run 'bzr pack' since the branch is only accessible from pqm?

[Robert Collins (lifeless)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John A Meinel wrote:
> Post 'bzr pack' I get:
> 101M .bzr/repository/obsolete_packs
> 1.0K .bzr/repository/pack-names
> 8.8M .bzr/repository/indices
> 30M .bzr/repository/packs
>
> In other words, I was correct about expecting ~30MB of packed content.
> As the size dropped from 101MB => 38.8MB (packs + indices).
>
> Do we just need to ask a LOSA to run 'bzr pack' since the branch is only
> accessible from pqm?

Well the master copy was freshly upgraded yesterday; we should check
with spm how big it is. That same bzr was then used to push the new
bzr.dev branch. Lets get detail, but be sure to find the root cause;
otherwise users will need to manually pack etc too.

 summary "newly converted and pushed repository poorly packed."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkqgH1MACgkQ42zgmrPGrq6+dwCfUdDTj2ViEef5SdLeMmpVpIJS
+X0An2hKQwZOMY7YwOdyDXfSBkJlLUHV
=gXsz
-----END PGP SIGNATURE-----

[Robert Collins (lifeless)]

exporting rev 1111 of bzr 1.18 in 1.9 format and then converting to 2a results in two packs; current WAG is that its packing only the most recent not-packed stages: we don't hint that the *resulting pack* of earlier incremental packs should be packed too.

[Robert Collins (lifeless)]

Two buglets: when autopack happened we were not returning its result in _commit_write_group, and GCRepo's _execute_pack_operations was not returning a hint.

[John A Meinel (jameinel)]

So, the reason 'launchpad-code" was included in this, is because we don't have a way to get the *public side* of a Bazaar branch to get repacked, without either committing 10x the total amount of data, or poking a sysadmin.

Even if I could do 'bzr pack lp:bzr' (I'm not allowed because it is only owned by ~bzr-pqm) it still wouldn't fix the public branch.

With our new repack-on-the-fly and conversions doing a full repack when finished, this is probably going to be less of an issue. The issue does remain, though.

[Michael Hudson-Doyle (mwhudson)]

I'm still just a touch confused, I guess: will this be an ongoing problem on Launchpad, or once the various bzr problems are sorted out, will we just have a bunch of poorly packed branches on launchpad that we should sort out?

I guess the semi-oft-requested "force a total remirror" feature would help here, perhaps?

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Hudson wrote:
> I'm still just a touch confused, I guess: will this be an ongoing
> problem on Launchpad, or once the various bzr problems are sorted out,
> will we just have a bunch of poorly packed branches on launchpad that we
> should sort out?

It is a bit unclear whether this will be ongoing. We think we've sorted
out the key parts that were causing this. (fragmentation without
recombining.)

However, it will certainly still be a problem until Launchpad upgrades
to 2.0 (once it is released), as 1.17 and 1.18 will continue to fragment.

>
> I guess the semi-oft-requested "force a total remirror" feature would
> help here, perhaps?
>

That would certainly be an option. Note, however, that 1.17 is
fragmenting during the mirroring operation, so it wouldn't help *today*.
(If you look at ~bzr-pqm/bzr/bzr.dev on the hosted side, it was a decent
50MB or so, but the mirrored side was around 100MB. After forcing a
'pack', it dropped down to more 30-40MB.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkqnKI8ACgkQJdeBCYSNAANDxgCaAtGG7k0nGsap/bQYwa+78NHC
LuMAoLdu0oILcSVXkSW9+aZZLLuiIF/5
=6QWB
-----END PGP SIGNATURE-----