bzr merge fails: bzr: ERROR: No final name for trans_id 'new-1'

[Kristian Nielsen]

We are cherry-pick merging into a repository from another repoistory with no common base.

The merge succeeds, but on the resulting repository we see bzr failing on some (but not all) subsequent merges from the unrelated repository:

    $ bzr merge ../test-merge-source
    +N mysql-test/suite/pbxt/r/type_varchar.result.OTHER
    +N mysql-test/suite/pbxt/t/type_varchar.test.OTHER
     M storage/pbxt/src/ha_pbxt.cc
    Contents conflict in mysql-test/suite/pbxt/r/type_varchar.result
    Contents conflict in mysql-test/suite/pbxt/t/type_varchar.test
    /usr/lib/python2.5/site-packages/bzrlib/ui/__init__.py:91: UserWarning: ProgressTask(0/2, msg='Apply phase') is not the active task ProgressTask(None/None, msg='')
      % (task, self._task_stack[-1]))
    /usr/lib/python2.5/site-packages/bzrlib/ui/__init__.py:91: UserWarning: ProgressTask(2/3, msg='Merge phase') is not the active task ProgressTask(None/None, msg='')
      % (task, self._task_stack[-1]))
    bzr: ERROR: No final name for trans_id 'new-1'
    file-id: None
    root trans-id: 'new-0'

    $ bzr --version
    Bazaar (bzr) 1.14.1
      Python interpreter: /usr/bin/python 2.5.2
      Python standard library: /usr/lib/python2.5
      bzrlib: /usr/lib/python2.5/site-packages/bzrlib
      Bazaar configuration: /home/knielsen/.bazaar
      Bazaar log file: /home/knielsen/.bzr.log

This may be related to bug 364305.

The involved repositories are public on Launchpad, and the following steps can be used to reproduce:

    bzr branch 'lp:~knielsen/maria/mariadb-5.1-pbxt-merge-take2'
    bzr branch '-rrevid:<email address hidden>' lp:pbxt merge-source
    cd mariadb-5.1-pbxt-merge-take2/
    bzr co
    bzr merge ../merge-source

Interestingly, another merge from lp:pbxt that includes more of the history does not fail:

    bzr branch 'lp:~knielsen/maria/mariadb-5.1-pbxt-merge-take2'
    bzr branch '-rrevid:<email address hidden>' lp:pbxt merge-source
    cd mariadb-5.1-pbxt-merge-take2/
    bzr co
    bzr merge ../merge-source

(the conflicts are expected).

In case the problem is with the original merge, here are the steps that produced the branch 'lp:~knielsen/maria/mariadb-5.1-pbxt-merge-take2':

  mkdir pbxt-new-merge-take2
  cd pbxt-new-merge-take2/
  bzr branch -r'revid:<email address hidden>' lp:pbxt pbxt-rename
  cd pbxt-rename/
  bzr co
  bzr mkdir pbxt-to-be-renamed
  bzr rm aclocal.m4 config config.h.in configure configure-dzl.in configure.in COPYING Makefile-dzl.am Makefile.in plug-dzl.in bin src/Makefile-dzl.am src/Makefile.in
  (cd test/mysql-test/t && bzr rm --force <lots of files>)
  (cd test/mysql-test/r && bzr rm --force <lots of files>)
  bzr mv test/mysql-test/t pbxt-to-be-renamed/
  bzr mv test/mysql-test/r pbxt-to-be-renamed/
  bzr rm test
  bzr mv AUTHORS INSTALL NEWS plug.in src ChangeLog Makefile.am README TODO pbxt-to-be-renamed/
  bzr commit -m$'Prepare for merging PBXT into MariaDB tree:\n - Delete not needed files.\n - Move needed files into subdirectory to avoid conflicts.'
  cd ..
  bzr branch -r'revid:<email address hidden>' lp:maria mariadb-pbxt-merge
  cd mariadb-pbxt-merge/
  bzr co
  bzr rm storage/pbxt mysql-test/suite/pbxt/{t,r}
  bzr commit -m"Delete old PBXT files in preparation for re-merging PBXT with proper history and merge tracking."
  bzr merge ../pbxt-rename -r0..-1
  bzr ci -m"Re-merge PBXT with proper history and merge tracking, into temporary directory for later renaming into place."
  bzr mv pbxt-to-be-renamed/{t,r} mysql-test/suite/pbxt/
  bzr mkdir storage/pbxt/
  bzr mv pbxt-to-be-renamed/* storage/pbxt/
  bzr rm pbxt-to-be-renamed/
  bzr ci -m"Re-merge PBXT into MariaDB with proper history and merge tracking: move files into correct place."
  patch -p0 -s < /tmp/1
  (cd mysql-test/suite/pbxt/ && patch -p0 -s < /tmp/2)
  (cd mysql-test/suite/pbxt/ && patch -p0 -s < /tmp/3)
  bzr ci -m"After-merge fixes for re-merge of PBXT into MariaDB with proper history and merge tracking."

(I can supply the exact list of <lots of files> and exact patches /tmp/{1,2,3} on request).

The intention here is to merge a sub-project (pbxt) into a main project (mariadb) in a way that preserves history and allows to use bzr merge tracking to allow future merges of pbxt changes. It may be that this is unsupported abuse of bzr, but in that case the original merge attempt should fail rather than leave a broken merge (and possibly a corrupt repository?).

[John A Meinel (jameinel)]

So I would recommend doing the merge using the merge-into command provided by lp:bzr-merge-into, here are the steps that I did:

<email address hidden>'
<email address hidden>'
bzr branch lp:maria -r revid:$MYSQL_REV maria_base
bzr branch lp:pbxt -r revid:$PBXT_REV pbxt_base
cd maria_base
bzr co
bzr rm storage/pbxt mysql/test/suite/pbxt/{r,t}
bzr commit -m "Remove pbxt in preparation for bring in upstream"
bzr merge-into ../pbxt_base storage/pbxt
bzr mv storage/pbxt/test/mysql-test/t mysql-test/suite/pbxt/t
bzr mv storage/pbxt/test/mysql-test/r mysql-test/suite/pbxt/r

bzr commit -m "Merged pbxt upstream into storage/pbxt and mysql-test/suite/pbxt/{t,r}"

At this point, applying any patches, etc, is up to you.
And doing "bzr merge lp:pbxt" should mostly 'just work'.

If it doesn't, that is definitely a bug.

Also, since you never manually edited any structures, but only ran things via the 'bzr' client, there is obviously a real bug laying around. We shouldn't let you get into a situation where we will crash. If it is a data integrity problem, then we shouldn't have let you generate that.

Now, the fact that you are also *cherrypicking* may be an issue, I'm not really sure.

[Kristian Nielsen (knielsen)]

Unfortunately, I get the same problem using the bzr merge-into plugin.

$ bzr log --show-ids -r-1
------------------------------------------------------------
revno: 8 [merge]
revision-id: <email address hidden>
parent: <email address hidden>
parent: <email address hidden>
committer: John Arbash Meinel <email address hidden>
branch nick: merge_into
timestamp: Wed 2008-08-27 20:41:16 -0500
message:
  Merge in the changes from Scott Scriven

The merge goes without errors, but when I then try to merge the next commit from lp:pbxt, I get the error.

Here are exact steps to reproduce from public launchpad trees. It seems the error only occurs after applying our further patches to the merge, so I have attached the necessary patches to the bug. Below commands reproduce the problem for me:

  mkdir tmp-bug375898
  cd tmp-bug375898/
  bzr branch -r'revid:<email address hidden>' lp:pbxt pbxt-base
  bzr branch -r'revid:<email address hidden>' lp:maria mariadb-pbxt-merge
  bzr branch '-rrevid:<email address hidden>' lp:pbxt tmp-pbxt-test-merge-1
  cd mariadb-pbxt-merge/
  bzr co
  bzr rm storage/pbxt mysql-test/suite/pbxt/{r,t}
  bzr commit -m "Remove old pbxt merge in preparation for re-merge with proper history and merge tracking."
  bzr merge-into ../pbxt-base storage/pbxt
  bzr mv storage/pbxt/test/mysql-test/{t,r} mysql-test/suite/pbxt/
  bzr ci -m "Re-merge PBXT into MariaDB with proper history and merge tracking."
  bzr rm `cat ../../bug375898-deletelist.txt`
  patch -p0 -s < ../../bug375898.patch
  bzr ci -m"After-merge fixes for re-merge of PBXT into MariaDB with proper history and merge tracking."
  bzr merge ../tmp-pbxt-test-merge-1

Output from last merge command:

    +N mysql-test/suite/pbxt/r/type_varchar.result.OTHER
    +N mysql-test/suite/pbxt/t/type_varchar.test.OTHER
     M storage/pbxt/src/ha_pbxt.cc
    Contents conflict in mysql-test/suite/pbxt/r/type_varchar.result
    Contents conflict in mysql-test/suite/pbxt/t/type_varchar.test
    /usr/lib/python2.5/site-packages/bzrlib/ui/__init__.py:91: UserWarning: ProgressTask(0/2, msg='Apply phase') is not the active task ProgressTask(None/None, msg='')
      % (task, self._task_stack[-1]))
    /usr/lib/python2.5/site-packages/bzrlib/ui/__init__.py:91: UserWarning: ProgressTask(2/3, msg='Merge phase') is not the active task ProgressTask(None/None, msg='')
      % (task, self._task_stack[-1]))
    bzr: ERROR: No final name for trans_id 'new-1'
    file-id: None
    root trans-id: 'new-0'

So while I agree that merge-into is a better solution, it does not seem to solve the original problem :-(

[Frits Jalvingh (fjalvingh)]

John A Meinel: I would like to add that this *same* problem can occur when you're not cherrypicking: see bug 364305 second remark and on. I have a feeling the problem has something to do with having multiple independent roots.

[Robert Collins (lifeless)]

Tree-transform related I think, not history per-se, unless per-file 3-way bases are being used or something like that.

[Robert Collins (lifeless)]

nb: likely a dup with bug 364305

[GuilhemBichot (guilhem-bichot)]

We are hitting the same issue.
We have a branch of InnoDB code (call it "innodb"; InnoDB is a
sub-module of MySQL); we have merged this branch into a MySQL branch
(call it "mysql"), preserving the important history of the "innodb"
branch, we did it this way:

cd mysql
bzr merge ../innodb
# put the sub-module at its place in the tree:
for each directory merged from ../innodb do
   bzr mv directory storage/innodb/directory
bzr commit

It worked fine. Since then there has been new changes in the "innodb"
branch, and we want to merge them into the "mysql" branch:
cd mysql
bzr merge ../innodb
and we get this error:

+N storage/innobase/mysql-test/innodb_bug47621.result
+N storage/innobase/mysql-test/innodb_bug47621.test
 M storage/innobase/handler/ha_innodb.cc
 M storage/innobase/mysql-test/innodb_bug21704.result
 M storage/innobase/mysql-test/innodb_bug38231.test
 M storage/innobase/plug.in
 M storage/innobase/row/row0sel.c
 /usr/local/lib/python2.6/site-packages/bzrlib/ui/__init__.py:188: UserWarning: ProgressTask(0/2, msg='Apply phase') is not the active task ProgressTask(None/None, msg='')
  % (task, self._task_stack[-1]))
 /usr/local/lib/python2.6/site-packages/bzrlib/ui/__init__.py:188: UserWarning: ProgressTask(2/3, msg='Merge phase') is not the active task ProgressTask(None/None, msg='')
  % (task, self._task_stack[-1]))
bzr: ERROR: No final name for trans_id 'new-1'
file-id: None
root trans-id: 'new-0'

bzr --version
Bazaar (bzr) 2.1.0
  Python interpreter: /usr/local/bin/python2.6 2.6.4
(same with 2.2.0dev1).

which is the same error as in the present bug report 375898.

This is blocking progress of a developer, please help us.

"bzr info -v" in branch "mysql":
Format:
       control: Meta directory format 1
  working tree: Working tree format 6
        branch: Branch format 6
    repository: Packs 6 (uses btree indexes, requires bzr 1.9)
In the working tree:
      8681 unchanged
         0 modified
         0 added
         0 removed
         0 renamed
         0 unknown
         0 ignored
       406 versioned subdirectories
Branch history:
      3372 revisions
      1600 days old
   first revision: Thu 2005-10-27 07:02:45 +0000
  latest revision: Mon 2010-03-08 18:16:41 +0200
Repository:
     60965 revisions

"bzr info -v" in the shared repository containing the branch:
Shared repository with trees (format: 1.14)
Location:
  shared repository: .
Format:
       control: Meta directory format 1
    repository: Packs 6 (uses btree indexes, requires bzr 1.9)
Create working tree for new branches inside the repository.
Repository:
     60965 revisions

[GuilhemBichot (guilhem-bichot)]

correction: when I wrote that the first merge was done like this:
<quote>
we did it this way:

cd mysql
bzr merge ../innodb
</quote>
the last command was rather
bzr merge -r0.. ../innodb

[Vincent Ladeuil (vila)]

Following our discussions on IRC, I can't reproduce it with the attached script and bzr.dev@revno 5109.
Note that I used revno 798 as a starting point suspecting adding new files to be the bug cause
Can you:
1) spot an error there (or a difference)
2) provide me a more complete innodb branch (with the changes that triggered the bug) ?

[GuilhemBichot (guilhem-bichot)]

I have emailed the latest InnoDB branch. Looks like the one I sent previously was too old - I'm very sorry.
With this *and the attached script* I am able to reproduce the bug.

[Vincent Ladeuil (vila)]

I'm able to reproduce the bug in a smaller test case:
- merge a branch with no common ancestry (with -r0..)
- renamed all the merged files,
- delete a file present in the imported root directory
- merge subsequent changes from the branch (the deleted file doesn't need to be modified)

Possible workarounds:
- use merge-into (better in principle but as mentioned above some blocking bugs
  should be fixed there too :-/)
- *before* the initial merge, move all content of the branch to be merged into a single directory,
   the initial and subsequents merges will now avoid the problem.

I'm still investigating a fix for bzr but using the above workaround should unblock the situation.

[Kristian Nielsen (knielsen)]

About merge-into: please note comment #2, which has exact steps to reproduce using merge-into (no cherrypick or merge from unrelated tree). So I don't see how merge-into can be listed as a work-around? (maybe I misunderstood)

A work-around would be very welcome though. Are you saying that the problem occurs when the files merged from externally are in the root? And that if instead we merge from an external repository where everything is in a subdirectory, then the problem will be worked around?

    cd pbxt
    bzr mkdir temporary-subdir
    bzr mv `ls|grep -v temporary-subdir` temporary-subdir/
    bzr commit
    cd ../maria
    bzr merge-into ../pbxt storage/
    bzr mv storage/temporary-subdir storage/pbxt
    bzr commit

And then for the next merge, simply

    bzr merge lp:pbxt

Is this the workaround that was suggested?

[Vincent Ladeuil (vila)]

@Kristian: I know merge-into has bugs, I stated it in comment #11, what I was trying to say was:

if the merge-into bugs were fixed, then using it will be a workaround for this (bug #373898) bug.

The workaround I'm suggesting is not using merge-into, for your case it would be:
    cd pbxt
    bzr mkdir temporary-subdir
    bzr mv `ls|grep -v temporary-subdir` temporary-subdir/ # watch for unknowns here but that's the idea
    bzr commit
    cd ../maria
    bzr merge -r0.. ../pbxt
    bzr mv temporary-subdir storage/pbxt
    bzr commit

From there the merges from pbxt shouldn't crash, but 'temporary-subdir' has to stay (at least until either this bug is fixed or merge-into is fixed or both) so you may want to use another name :-/

[Vincent Ladeuil (vila)]

@Guilhem, the workaround I suggested for innodb is demonstrated in the attached script.

[Vincent Ladeuil (vila)]

I have a fix available and under review:
   https://code.edge.launchpad.net/~vila/bzr/375898-fix-root-more/+merge/22625

[Vasil Dimov (vasild)]

Thank you for fixing this bug!

In which 2.1.x version will the fix be available?

Thanks!

[Vincent Ladeuil (vila)]

>>>>> Vasil Dimov <email address hidden> writes:

    > Thank you for fixing this bug!
    > In which 2.1.x version will the fix be available?

It has landed in the 2.0 branch, so it should be available soon in
bzr.dev, as for the releases, the fix should be in 2.0.6, 2.1.2 and
2.2b2 when they become available.

[Dimitri John Ledkov (xnox)]

What version of bzr is deployed in the bzr recipe builder on launchpad?

added debian/source/local-options
Committed revision 3019.
Merging 'lp:xiphos/debian' in to '/home/buildd/build-bae7f17241ba7dab79ebd0dd03fe528f9c791b58/chroot-autobuild/home/buildd/work/tree/recipe-{debupstream}+{revno}[phone]ubuntu0'.
/usr/lib/python2.5/site-packages/bzrlib/ui/__init__.py:148: UserWarning: ProgressTask(0/2, msg='Apply phase') is not the active task ProgressTask(None/None, msg='')
  % (task, self._task_stack[-1]))
bzr: ERROR: No final name for trans_id 'new-4'
file-id: None
root trans-id: 'new-0'

Happened here: https://code.launchpad.net/~pkgcrosswire/+recipe/xiphos-daily/+build/10657

[Dimitri John Ledkov (xnox)]

Launchpad code team please see: https://bugs.launchpad.net/launchpad-code/+bug/375898/comments/18

[Vincent Ladeuil (vila)]

@Dmitrijs: Please file a new bug, adding comments to fix released bugs makes it order of magnitude harder to track progress otherwise.

'rmadison bzr' says karmic carries 2.0.0 while karmic-updates carries 2.0.2. Both are too old for the fix.