UnknownErrorFromSmartServer: Server sent an unexpected error: ('error', 'math range error') during commit

[byron jones]

when committing a change, the following error is produced:
bzr: ERROR: Server sent an unexpected error: ('error', 'math range error')
the server has been updated successfully, however my local repo needs a 'bzr up' to bring it into sync with the server.

server version is 2.3.1, i am waiting on the sysadmin to see if there's anything in the server logs.

$ bzr -Dhpss commit skins/custom/show_bug.css -m 'stop resizing comment column when all comments are hidden'
Committing to: bzr+ssh://bjones%<email address hidden>/bmo/4.0/
modified skins/custom/show_bug.css
bzr: ERROR: Server sent an unexpected error: ('error', 'math range error')
HPSS calls: 18 (0 vfs) SmartSSHClientMedium(bzr+ssh://<email address hidden>@bzr.mozilla.org/)
$ bzr st
modified:
  skins/custom/show_bug.css
$ bzr up
All changes applied successfully.
Updated to revision 7580 of branch bzr+ssh://bjones%<email address hidden>/bmo/4.0
$ bzr st
$

Wed 2011-03-30 00:25:17 +0800
0.127 bazaar version: 2.3.1
0.128 bzr arguments: [u'-Dhpss', u'commit', u'skins/custom/show_bug.css', u'-m', u'stop resizing comment column when all comments are hidden']
0.184 looking for plugins in /home/byron/.bazaar/plugins
0.187 looking for plugins in /usr/lib/python2.6/site-packages/bzrlib/plugins
0.212 encoding stdout as sys.stdout encoding 'UTF-8'
0.379 opening working tree '/cygdrive/c/dev/bugzilla/repo/bmo/4.0'
0.579 hpss: Built a new medium: SmartSSHClientMedium
0.581 hpss call: 'BzrDir.open_2.1', 'bmo/4.0/'
0.582 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
0.745 ssh implementation is OpenSSH
5.458 result: ('yes', 'no')
5.459 hpss call: 'BzrDir.open_branchV3', 'bmo/4.0/'
5.460 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
5.739 result: ('branch', 'Bazaar Branch Format 7 (needs bzr 1.6)\n')
5.740 hpss call: 'BzrDir.find_repositoryV3', 'bmo/4.0/'
5.742 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
6.002 result: ('ok', '', 'yes', 'no', 'yes', 'Bazaar repository format 2a (needs bzr 1.16 or later)\n')
6.007 hpss call: 'Branch.get_stacked_on_url', 'bmo/4.0/'
6.009 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
6.283 result: ('NotStacked',)
6.285 hpss call: 'Branch.get_config_file', 'bmo/4.0/'
6.286 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
6.548 result: ('ok',)
6.549 38 body bytes read
6.551 hpss call: 'Branch.get_config_file', 'bmo/4.0/'
6.553 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
6.814 result: ('ok',)
6.815 38 body bytes read
6.816 preparing to commit
6.822 hpss call: 'BzrDir.open_2.1', 'bmo/4.0/'
6.823 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
7.082 result: ('yes', 'no')
7.083 hpss call: 'BzrDir.open_branchV3', 'bmo/4.0/'
7.084 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
7.349 result: ('branch', 'Bazaar Branch Format 7 (needs bzr 1.6)\n')
7.350 hpss call: 'BzrDir.find_repositoryV3', 'bmo/4.0/'
7.352 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
7.612 result: ('ok', '', 'yes', 'no', 'yes', 'Bazaar repository format 2a (needs bzr 1.16 or later)\n')
7.614 hpss call: 'Branch.get_stacked_on_url', 'bmo/4.0/'
7.615 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
7.884 result: ('NotStacked',)
7.885 hpss call: 'Branch.last_revision_info', 'bmo/4.0/'
7.886 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
8.154 result: ('ok', '7579', '<email address hidden>')
8.157 hpss call: 'Branch.lock_write', 'bmo/4.0/', '', ''
8.158 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
8.427 result: ('ok', '488pgs2c45fs6u2glaes', '')
8.428 hpss call: 'Branch.last_revision_info', 'bmo/4.0/'
8.430 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
8.696 result: ('ok', '7579', '<email address hidden>')
[ 5476] 2011-03-30 00:25:26.684 INFO: Committing to: bzr+ssh://bjones%<email address hidden>/bmo/4.0/
8.757 Selecting files for commit with filter [u'skins/custom/show_bug.css']
[ 5476] 2011-03-30 00:25:26.786 INFO: modified skins/custom/show_bug.css
9.268 Using fetch logic to copy between CHKInventoryRepository('file:///cygdrive/c/dev/bugzilla/repo/bmo/4.0/.bzr/repository/')(RepositoryFormat2a()) and RemoteRepository(bzr+ssh://bjones%<email address hidden>/bmo/4.0/.bzr/)(RemoteRepositoryFormat(_network_name='Bazaar repository format 2a (needs bzr 1.16 or later)\n'))
9.272 fetch up to rev {<email address hidden>}
9.285 hpss call w/body: 'Repository.get_parent_map', 'bmo/4.0/', 'include-missing:', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>', '<email address hidden>' ('\n\n0'...)
9.307 3 bytes
10.478 result: ('ok',)
11.762 56200 body bytes read
11.796 hpss call w/body stream: ('Repository.insert_stream_1.19', 'bmo/4.0/', '')
12.063 result: ('ok',)
12.063 hpss call w/body stream: ('Repository.insert_stream_1.19', 'bmo/4.0/', '')
12.544 result: ('ok',)
12.545 hpss call: 'Branch.set_last_revision_info', 'bmo/4.0/', '488pgs2c45fs6u2glaes', '', '7580', '<email address hidden>'
12.549 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
12.949 result: ('error', 'math range error')
12.951 hpss call: 'Branch.unlock', 'bmo/4.0/', '488pgs2c45fs6u2glaes', ''
12.952 (to bzr+ssh://bjones%<email address hidden>/bmo/4.0/)
13.222 result: ('ok',)
13.277 Transferred: 72kB (5.8kB/s r:58kB w:14kB)
13.478 Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 926, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 1126, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 691, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 713, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/builtins.py", line 3241, in run
    exclude=tree.safe_relpath_files(exclude))
  File "/usr/lib/python2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/workingtree_4.py", line 197, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/mutabletree.py", line 201, in commit
    *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/commit.py", line 286, in commit
    possible_master_transports=possible_master_transports)
  File "/usr/lib/python2.6/site-packages/bzrlib/cleanup.py", line 131, in run
    self.cleanups, self.func, self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/commit.py", line 450, in _commit
    self.branch.repository, new_revno, self.rev_id)
  File "/usr/lib/python2.6/site-packages/bzrlib/branch.py", line 1049, in import_last_revision_info
    self.set_last_revision_info(revno, revid)
  File "/usr/lib/python2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/remote.py", line 2728, in set_last_revision_info
    str(revno), revision_id)
  File "/usr/lib/python2.6/site-packages/bzrlib/remote.py", line 59, in _call
    self._translate_error(err, **err_context)
  File "/usr/lib/python2.6/site-packages/bzrlib/remote.py", line 2314, in _translate_error
    self.repository._translate_error(err, branch=self, **context)
  File "/usr/lib/python2.6/site-packages/bzrlib/remote.py", line 825, in _translate_error
    self.bzrdir._translate_error(err, repository=self, **context)
  File "/usr/lib/python2.6/site-packages/bzrlib/remote.py", line 175, in _translate_error
    _translate_error(err, bzrdir=self, **context)
  File "/usr/lib/python2.6/site-packages/bzrlib/remote.py", line 3029, in _translate_error
    raise errors.UnknownErrorFromSmartServer(err)
UnknownErrorFromSmartServer: Server sent an unexpected error: ('error', 'math range error')

13.528 return code 3
[ 5476] 2011-03-30 00:25:31.458 INFO: HPSS calls: 18 (0 vfs) SmartSSHClientMedium(bzr+ssh://<email address hidden>@bzr.mozilla.org/)

[Andrew Bennetts (spiv)]

So it's happening on the call to Branch.set_last_revision. That operation itself is very simple, but it also triggers pre/post_change_branch_tip hooks on the server. So my suspicion is that there's a plugin installed on the server that uses those hooks, and the error is originating there. Given that branch is actually updated, it's almost certainly something that occurs during the post_change_branch_tip hook.

Can you find out which plugins are installed on the server? Although probably there's a traceback in the server's log that will make it clear which one is the culprit.

[byron jones (glob)]

the error started happening after it was upgraded from 2.1.1 to 2.2.2, and affects all users. it's running 2.3.1 currently.

server-side log:

7.454 Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/bzrlib/smart/request.py", line 355, in
_call_converting_errors
    return callable(*args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/smart/request.py", line 140, in
execute
    return self.do(*args)
  File "/usr/lib/python2.4/site-packages/bzrlib/smart/branch.py", line 52, in
do
    return self.do_with_branch(branch, *args)
  File "/usr/lib/python2.4/site-packages/bzrlib/smart/branch.py", line 72, in
do_with_branch
    return self.do_with_locked_branch(branch, *args)
  File "/usr/lib/python2.4/site-packages/bzrlib/smart/branch.py", line 182, in
do_with_locked_branch
    return self.do_tip_change_with_locked_branch(branch, *args)
  File "/usr/lib/python2.4/site-packages/bzrlib/smart/branch.py", line 289, in
do_tip_change_with_locked_branch
    branch.set_last_revision_info(int(new_revno), new_last_revision_id)
  File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line 194, in
write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.4/site-packages/bzrlib/branch.py", line 2870, in
set_last_revision_info
    self._run_post_change_branch_tip_hooks(old_revno, old_revid)
  File "/usr/lib/python2.4/site-packages/bzrlib/branch.py", line 1219, in
_run_post_change_branch_tip_hooks
    hook(params)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/search/__init__.py",
line 59, in auto_index_branch
    search_index.index_branch(result.branch, result.new_revid)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/search/index.py", line
321, in index_branch
    self.index_revisions(branch, revs_to_index)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/search/index.py", line
336, in index_revisions
    outer_bar)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/search/index.py", line
384, in _index_revisions
    self._add_index(builder)
  File "/usr/lib/python2.4/site-packages/bzrlib/plugins/search/index.py", line
440, in _add_index
    max_components = int(math.ceil(math.log(total_revisions, 2)))
OverflowError: math range error

[byron jones (glob)]

this looks like a duplicate of https://launchpad.net/bugs/627202

[Martin Pool (mbp)]

Yes, I think you're right. You'll need to disable, uninstall, or
upgrade bzr-search on the server.

Martin