ValueError: Invalid value for commit message

[Peter Connolly]

When I attempt to migrate our Subversion repository using the latest svn2bzr code, I get:

Nothing changed in revision 169
Committing revision 170
Committing revision 171
Nothing changed in revision 172
Committing revision 173
Traceback (most recent call last):
  File "./svn2bzr.py", line 1083, in <module>
    main()
  File "./svn2bzr.py", line 1076, in main
    opts.prefix, opts.filter)
  File "./svn2bzr.py", line 1005, in svn2bzr
    creator.run()
  File "./svn2bzr.py", line 419, in run
    commit()
  File "./svn2bzr.py", line 410, in commit
    timestamp=timestamp)
  File "./svn2bzr.py", line 386, in commit
    verbose=False)
  File "<string>", line 4, in commit_write_locked
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree_4.py", line 225, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "<string>", line 4, in commit_write_locked
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 228, in commit
    *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commit.py", line 384, in commit
    self.rev_id = self.builder.commit(self.message)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 149, in commit
    self._validate_unicode_text(message, 'commit message')
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 132, in _validate_unicode_text
    raise ValueError('Invalid value for %s: %r' % (context, text))
ValueError: Invalid value for commit message: u'Copied Chapter 5 (SiteMesh) complete over ZacksTool. Defines new baseline. \r\nRenamed org.appfuse to org.alembix.\n'
peterc@popeye:~/bin/svn2bzr$

Examining the svn dump file, it appears to have the following at that point in the conversion:

02453880 62 61 73 65 6c 69 6e 65 2e 20 26 23 31 30 3b 52 |baseline. &#10;R|
02453890 65 6e 61 6d 65 64 20 6f 72 67 2e 61 70 70 66 75 |enamed org.appfu|
024538a0 73 65 20 74 6f 20 6f 72 67 2e 61 6c 65 6d 62 69 |se to org.alembi|
024538b0 78 2e 22 20 2f 3e 0a 20 20 20 20 3c 6f 70 74 69 |x." />. <opti|

[Peter Connolly (peterc-alembixengineering)]

Note that this is different from https://bugs.launchpad.net/bugs/345247. I was attempting to use the --scheme=trunk on the migration:

sudo ./svn2bzr.py --scheme=trunk /home/peterc/svn-repos-dump-full_20081001-0130 /var/bzr

bzr version: 1.14dev
svn version: 1.5.1

[Martin Spacek (mspacek)]

Peter, can you try using the latest rev of svn2bzr? Also, could you maybe include a little more of the dump file around where the error happens? (You can just post it in text mode, no need for the binary.) It's possible the \r and \n are causing the problem.

[Peter Connolly (peterc-alembixengineering)]

I was running the 0.9 version previously. I upgraded to the 0.10-dev version before rerunning the conversion:

peterc@popeye:~/bin/svn2bzr$ sudo ./svn2bzr.py ~/tmp/svn-repos-dump-full_20090401-0130 /var/bzr

I get the same error:

Committed revision 169
Committed revision 170
Committed revision 171
Nothing changed in revision 172
...but committing revision properties (revno, message, committer, and timestamp) anyways
Committed revision 172
Changing dir: /var/bzr/trunk/myusers
warning: Code for changing dir properties is unimplemented.
 dir: /var/bzr/trunk/myusers
 property: {'svn:ignore': 'UserFormControllerTest_output.txt\nZacksTool.sptl'} remains unchanged
Changing dir: /var/bzr/trunk/myusers/web/WEB-INF/classes
warning: Code for changing dir properties is unimplemented.
 dir: /var/bzr/trunk/myusers/web/WEB-INF/classes
 property: {'svn:ignore': 'images\nMETA-INF\norg'} remains unchanged
Traceback (most recent call last):
  File "./svn2bzr.py", line 1637, in <module>
    main()
  File "./svn2bzr.py", line 1630, in main
    log=log)
  File "./svn2bzr.py", line 1531, in svn2bzr
    creator.run()
  File "./svn2bzr.py", line 682, in run
    self.commit_revision(revno, revision)
  File "./svn2bzr.py", line 664, in commit_revision
    timestamp=timestamp, timezone=offset)
  File "./svn2bzr.py", line 547, in commit
    timestamp, timezone)
  File "./svn2bzr.py", line 584, in commit_brt
    verbose=False)
  File "<string>", line 4, in commit_write_locked
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree_4.py", line 225, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "<string>", line 4, in commit_write_locked
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 228, in commit
    *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commit.py", line 384, in commit
    self.rev_id = self.builder.commit(self.message)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 149, in commit
    self._validate_unicode_text(message, 'commit message')
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 132, in _validate_unicode_text
    raise ValueError('Invalid value for %s: %r' % (context, text))
ValueError: Invalid value for commit message: u'Copied Chapter 5 (SiteMesh) complete over ZacksTool. Defines new baseline. \r\nRenamed org.appfuse to org.alembix.\n'

Here are two portions of the SVN dump file around the above line that is causing the error:

  <component name="VCS.FileViewConfiguration">
    <option name="SELECTED_STATUSES" value="DEFAULT" />
    <option name="SELECTED_COLUMNS" value="DEFAULT" />
    <option name="SHOW_FILTERS" value="true" />
    <option name="CUSTOMIZE_VIEW" value="true" />
    <option name="SHOW_FILE_HISTORY_AS_TREE" value="true" />
  </component>
  <component name="VcsManagerConfiguration">
    <option name="CHECK_CODE_SMELLS_BEFORE_PROJECT_COMMIT" value="true" />
    <option name="PUT_FOCUS_INTO_COMMENT" value="false" />
    <option name="FORCE_NON_EMPTY_COMMENT" value="false" />
    <option name="LAST_COMMIT_MESSAGE" value="Copied Chapter 5 (SiteMesh) complete over ZacksTool. Defines new baseline. &#10;Renamed o
...

[Peter Connolly (peterc-alembixengineering)]

Those XML lines cut-and-pasted rather funkily. Here's a 2nd stab at those two lines:

<option name="LAST_COMMIT_MESSAGE" value="Copied Chapter 5 (SiteMesh) complete over ZacksTool. Defines new baseline. &#10;Renamed org.appfuse to org.alembix." />

<MESSAGE value="Copied Chapter 5 (SiteMesh) complete over ZacksTool. Defines new baseline. &#10;Renamed org.appfuse to org.alembix." />

[Peter Connolly (peterc-alembixengineering)]

Another portion of the SVN dump file:

Revision-number: 171
Prop-content-length: 154
Content-length: 154

K 7
svn:log
V 80
This commit was manufactured by cvs2svn to create branch
'SPRING_LIVE_PAGE_139'.
K 8
svn:date
V 27
2006-08-03T03:23:19.000000Z
PROPS-END

Node-path: branches/SPRING_LIVE_PAGE_139/ZacksTool/web/taglibs.jsp
Node-kind: file
Node-action: add
Node-copyfrom-rev: 170
Node-copyfrom-path: trunk/ZacksTool/web/taglibs.jsp

Revision-number: 172
Prop-content-length: 160
Content-length: 160

K 7
svn:log
V 59
baselining ZacksTool from working myusers, with appfuse db

K 10
svn:author
V 6
peterc
K 8
svn:date
V 27
2006-08-03T03:23:19.000000Z
PROPS-END

Revision-number: 173
Prop-content-length: 215
Content-length: 215

K 7
svn:log
V 113
Copied Chapter 5 (SiteMesh) complete over ZacksTool. Defines new baseline.
Renamed org.appfuse to org.alembix.

K 10
svn:author
V 6
peterc
K 8
svn:date
V 27
2006-08-03T05:37:49.000000Z
PROPS-END

Node-path: trunk/myusers
Node-kind: dir
Node-action: change
Prop-content-length: 80
Content-length: 80

K 10
svn:ignore
V 48
UserFormControllerTest_output.txt
ZacksTool.sptl
PROPS-END

Node-path: trunk/myusers/.classpath
Node-kind: file
Node-action: change
Prop-content-length: 142
Text-content-length: 2537
Text-content-md5: ec8bdee2f5378630451ee342d70c089b
Content-length: 2679

K 14
svn:executable
V 1
*
K 15
cvs2svn:cvs-rev
V 3
1.3
K 12
svn:keywords
V 23
Author Date Id Revision
K 13
svn:eol-style
V 6
native
PROPS-END
<?xml version="1.0" encoding="UTF-8"?>
<classpath>
        <classpathentry kind="src" path="src"/>
        <classpathentry output="build/test/classes" kind="src" path="test"/>
        <classpathentry excluding="WEB-INF/classes/|WEB-INF/lib/c3p0-0.8.4.5.jar/|WEB-INF/lib/cglib-full-2.0.1.jar/|WEB-INF/lib/ehcache
        <classpathentry kind="lib" path="web/WEB-INF/lib/spring.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/standard.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/sitemesh-2.0.2.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/odmg-3.0.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/oscache-2.0.1.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/struts-el.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/commons-collections.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/commons-digester.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/commons-lang.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/dom4j-1.4.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/struts.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/commons-beanutils.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/hibernate2.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/commons-fileupload.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/commons-logging.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/commons-validator.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/aopalliance.jar"/>
        <classpathentry kind="lib" path="web/WEB-INF/lib/jstl.jar"/>
        <classpathentry kind="lib"...

[Martin Spacek (mspacek)]

Peter, the bzrlib.errors.PathNotChild error should now be fixed. Try the latest trunk rev.

I'm a bit confused as to the actual contents of your dump file. If the svn:log entry for Revision-number: 173 is as you say in your last post:

K 7
svn:log
V 113
Copied Chapter 5 (SiteMesh) complete over ZacksTool. Defines new baseline.
Renamed org.appfuse to org.alembix.

...then that should work fine. But according to your previous posts, it looks like you're getting a unicode error. svn2bzr doesn't (yet) support unicode in svn:log and svn:author fields. I'm not sure where you're getting those XML lines describing the commit message from. If the svn:log entry doesn't actually look exactly like above, then go in and delete any unicode characters. You should also decrement the 113 value appropriately (this describes the minimum length of the message field). You might also want to add a few extra blank lines after the entry, just to be safe. If you do modify your dump file, make sure your text editor doesn't strip trailing spaces or replace line endings.

Could you re-post that region of your dump file as an attachment?

[Peter Connolly (peterc-alembixengineering)]

I'm attaching a snippet of the dump file that contains the offending revision. I think the unicode was lost in the cut-and-paste operation.

[Peter Connolly (peterc-alembixengineering)]

I also downloaded the latest rev of svn2bzr and tried running it against an edited version of the dump file. I deleted the ^M character at the end of the problematic commit log message and adjusted the length accordingly. This time I got thrown into the debugger:

Committed revision 171
Nothing changed in revision 172
...but committing revision properties (revno, message, committer, and timestamp) anyways
Committed revision 172
Changing dir: /var/bzr/trunk/myusers
warning: Code for changing dir properties is unimplemented.
 dir: /var/bzr/trunk/myusers
 property: {'svn:ignore': 'UserFormControllerTest_output.txt\nZacksTool.sptl'} remains unchanged
Changing dir: /var/bzr/trunk/myusers/web/WEB-INF/classes
warning: Code for changing dir properties is unimplemented.
 dir: /var/bzr/trunk/myusers/web/WEB-INF/classes
 property: {'svn:ignore': 'images\nMETA-INF\norg'} remains unchanged
Committed revision 173
error: Encountered error: global name 'Error' is not defined
while processing dump entry <DumpEntry "{'content-length': 28940, 'text-content-length': 28824, 'node-action': 'change', 'node-path': 'trunk/myusers/ZacksTool.iws', 'text-content-md5': 'b54274e224b71afd262c01371f6d83e0', 'prop-content-length': 116, 'node-kind': 'file'}">
Dropping into debugger...
> /home/peterc/bin/svn2bzr/branchcreator.py(843)execute_revision_entries()
-> for entry in entries:
(Pdb) quit
Traceback (most recent call last):
  File "./svn2bzr.py", line 194, in <module>
    main()
  File "./svn2bzr.py", line 187, in main
    log=log)
  File "./svn2bzr.py", line 88, in svn2bzr
    creator.run()
  File "/home/peterc/bin/svn2bzr/branchcreator.py", line 620, in run
    self.execute_revision_entries(entries)
  File "/home/peterc/bin/svn2bzr/branchcreator.py", line 843, in execute_revision_entries
    for entry in entries:
  File "/home/peterc/bin/svn2bzr/branchcreator.py", line 843, in execute_revision_entries
    for entry in entries:
  File "/usr/lib/python2.5/bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python2.5/bdb.py", line 67, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit

[Martin Spacek (mspacek)]

Oops. Unfortunately, the true error was masked by the fact that the "Error" class wasn't available in branchcreator.py. My mistake. That happened when I split the code up into separate (smaller) modules. That's fixed in trunk rev 23. Could you try running it again? That should unveil the true error message and what line it occurs on...

Also, it seems that your revision 173 does get committed successfully, and that the error is happening in revision 174. Could you post a little more of your (edited) dump file including rev 174?

[Peter Connolly (peterc-alembixengineering)]

Status update: got the latest rev 23 and ran again. Ran into a 2nd unicode-ish problem like the first one. Edited that out of the SVN dump file and re-ran again. It's now into the 200s of revisions. Will report on the results later when/if they complete.

[Peter Connolly (peterc-alembixengineering)]

I was able to get through revision #607 out of the total of 1878 by editing out the ^M characters from the commit log messages. But in revision #608 I encountered a new problem that I've described in bug #362059.