inconsistent parents in bzr 1.14 shared repository

[GuilhemBichot]

Hello. I am opening this here directly. An alternative would have been to first go through the support portal, but result would be the same.
I did a "bzr check -v" on a binary copy of the shared repository of our central machine, which is in 1.14 format, it ran for 51 hours and reported errors. The check was done with bzr 1.15.1 and not bzr.dev due to
https://bugs.launchpad.net/bzr/+bug/522296 .
The binary copy was done with "tar" after putting the central's machine repository offline, so the copy process isn't guilty.
I would like to know whether this is serious corruption, and how to eliminate it (from the central host, from the developers' own repositories...). I am attaching the errors.
Thank you.

[GuilhemBichot (guilhem-bichot)]

This "check -v" was done in preparation for an upgrade of our central repository to format 2a.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

GuilhemBichot wrote:
> This "check -v" was done in preparation for an upgrade of our central
> repository to format 2a.
>

The only thing I saw on a quick glance was 'inconsistent parents' which
IIRC can be resolved with 'bzr reconcile'.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkuFJ14ACgkQJdeBCYSNAAMgDwCgmMTxuItvbFY69BNY/cYxtXbl
TSEAnihE5YYwmYNqlZS7+zcPHjg+Y5O7
=CuqP
-----END PGP SIGNATURE-----

[GuilhemBichot (guilhem-bichot)]

Thanks John. So it's good to know that 'bzr reconcile' will fix this. There are additional questions:
- will "bzr upgrade" or on-the-fly 1.9/1.14->2a format conversion also fix this, or does "reconcile" needs to be explicitely invoked? - If it has to be invoked, should it be before/after the upgrade?
- should every developer run "reconcile" on his own repository?
- how did this happen? We checked the repository for corruption in Oct 2008, it had problems, they were fixed (https://bugs.launchpad.net/bzr/+bug/277537), there was no corruption anymore, now there is.

[Vincent Ladeuil (vila)]

The problem is a different than bug #277537, but using the script that parses the .bzr.log on your file gives:
# revisions involved: 2
# files involved: 498
Suspicious rev_id: '<email address hidden>'
Suspicious rev_id: '<email address hidden>'
2 suspicious revision ids were encountered

[Vincent Ladeuil (vila)]

'bzr reconcile' should be run *before* the upgrade.

The origin of the corruption is still unclear but may not be worth investigating because 'bzr check'
here is reporting that some file revisions are not referenced in the inventories (i.e. unreachable)
and should just be deleted.

[GuilhemBichot (guilhem-bichot)]

Thank you. I would still like to know:
- how could an unreachable revision come to life here? Normally, we use "bzr merge", "pull", "push", "commit" between bzr branches, I'm not aware of anything more fancy.
- will "bzr upgrade" or on-the-fly 1.9/1.14->2a format conversion also fix this, or does "reconcile" needs to be explicitely invoked? - should every developer run "reconcile" on his own repository?

[GuilhemBichot (guilhem-bichot)]

I have started a "bzr reconcile -v" (using bzr.dev) on the repository copy.

[GuilhemBichot (guilhem-bichot)]

reconcile finished after 5 hours. Next step will be to "bzr check -v" the result.

[GuilhemBichot (guilhem-bichot)]

I'm attaching the log of "bzr reconcile" (done with bzr.dev), which returned 0. I'm also attaching the log of the post-reconcile "bzr check -v" (done with bzr 1.15.1) which also returned 0; could you please have a look at those logs, especially the check's one, and say whether everything looks ok now, no more problems? Does it look consistent with what had been observed in the pre-reconcile check (log posted earlier)? Thanks.

[GuilhemBichot (guilhem-bichot)]

detail: in the log above, the interesting piece is the last command run (sorry for not cutting the useless other pieces).

[GuilhemBichot (guilhem-bichot)]

Note also that "bzr reconcile" left a 450MB file: .bzr/repository/uploads/y74vnm919853u01sunmn.reconcile which according to John A. Meinel is a temporary file and should have been automatically deleted.
If you say that my repo now looks ok, I guess I will run "bzr upgrade" on it and give the resulting repo to my colleagues.
I assume that if they instead wanted to upgrade their local repo, they would have to run reconcile too? "upgrade" doesn't automatically "reconcile"?

[Vincent Ladeuil (vila)]

All logs look fine.
> - how could an unreachable revision come to life here? Normally, we use "bzr merge",
> "pull", "push", "commit" between bzr branches, I'm not aware of anything more fancy.

AFAIK, there is no known way to produce inconsistent parents with bzr released versions.
There has been reports of such inconsistent parents though, hence the fix provided by 'bzr reconcile',
but I think they were due to some bzr.dev versions which were fixed before releases.
More importantly, these parents have no known consequences in normal bzr usage, so the right
thing to do is indeed to run 'bzr reconcile'.

Regarding the file left by reconcile, there is bug #373539 about adding
a bzr command to cleanp up both the obsolete and upload directories (I thought John or you
filed a more specific bug against reconcile itself but I can't find it back).

> I assume that if they instead wanted to upgrade their local repo, they would have to run reconcile too?

Yes.

> "upgrade" doesn't automatically "reconcile"?

No.

Can your report the repo sizes (with clean obsolete and upload dirs :) before and after the upgrade ?
Thanks in advance.

[Robert Collins (lifeless)]

On Mon, 2010-03-08 at 14:53 +0000, Vincent Ladeuil wrote:
> All logs look fine.
> > - how could an unreachable revision come to life here? Normally, we use "bzr merge",
> > "pull", "push", "commit" between bzr branches, I'm not aware of anything more fancy.
>
> AFAIK, there is no known way to produce inconsistent parents with bzr released versions.
> There has been reports of such inconsistent parents though, hence the fix provided by 'bzr reconcile',
> but I think they were due to some bzr.dev versions which were fixed before releases.
> More importantly, these parents have no known consequences in normal bzr usage, so the right
> thing to do is indeed to run 'bzr reconcile'.

many old bzr releases had a bug generating inconsistent parents. I don't
recall when we fixed it. 1.14 perhaps?

-Rob

[GuilhemBichot (guilhem-bichot)]

>Can your report the repo sizes (with clean obsolete and upload dirs :) before and after the upgrade ?

before: 950MB; after: 440MB, and after an additional "bzr pack": 341MB.
NB: I didn't use "bzr upgrade" but created a fresh 2a repo and re-branched each branch from the 1.14 repo to the 2a repo, which caused on-the-fly upgrade.
Next thing, is doing a "bzr check -v" of the 2a repo.

[GuilhemBichot (guilhem-bichot)]

>Regarding the file left by reconcile, there is bug #373539 about adding
>a bzr command to cleanp up both the obsolete and upload directories (I thought John or you
>filed a more specific bug against reconcile itself but I can't find it back).

It was bug 528412

[GuilhemBichot (guilhem-bichot)]

bzr check -v
on the 2a repo, seems ok:
...
[20305] 2010-03-09 11:06:02.358 INFO: Checking branch at 'file:///home/mysql_src/from_bk_internal/for_upgrade/guilhem
/saved_5.0_bk_trees/mysql-5.0.72sp1-bug46861/'.
[20305] 2010-03-09 11:06:02.358 INFO: No working tree found at specified location.
[20305] 2010-03-09 11:06:02.358 INFO: Checking repository at 'file:///home/mysql_src/from_bk_internal/for_upgrade/guilhem/'.
5537.612 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x2b3a310>, 129072803, 6148346) to an LRUSizeCache failed. value 165270510 is too big to fit in a the cache with size 41943040 52428800
5619.516 Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x2b3a310>, 234365681, 4634459) to an LRUSizeCache failed. value 43326957 is too big to fit in a the cache with size 41943040 52428800
[20305] 2010-03-11 16:32:13.309 INFO: checked repository <bzrlib.transport.local.LocalTransport url=file:///home/mysql_src/from_bk_internal/for_upgrade/guilhem/> format <RepositoryFormat2a>
[20305] 2010-03-11 16:32:13.327 INFO: 86618 revisions
[20305] 2010-03-11 16:32:13.328 INFO: 39252 file-ids
[20305] 2010-03-11 16:32:13.328 INFO: 0 unreferenced text versions
[20305] 2010-03-11 16:32:13.328 INFO: checked branch file:///home/mysql_src/from_bk_internal/for_upgrade/guilhem/mysql-6.0-mtr/ format Branch format 6
...
192373.540 Transferred: 0kB (0.0kB/s r:0kB w:0kB)
192373.540 return code 0
I used bzr.dev and it took 53 hours (700MB RAM).
The number of revisions is a bit lower than in the 1.14 repo, that is possible, because I used "bzr branch", so some revisions present in the repository but not in any branch (present in a deleted branch some time ago) could be lost.

[Vincent Ladeuil (vila)]

Since the 'bzr check' after the 'bzr reconcile' is ok, it confirms that reconcile fixed the inconsistent parents.