"ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem" when pushing a branch

[Julian Edwards]

I was pushing up a branch change with 1.9rc1-1 and got this exception:

UnknownErrorFromSmartServer: Server sent an unexpected error: ('error',
'<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

OOPS-2030DZ37 seems to be an example of this.

Doing a push, wait 30 seconds, push again, is likely to trigger this while the scanner is running.

[Martin Albisetti (beuno)]

Happened to me again in: https://code.edge.launchpad.net/~beuno/launchpad/ui_bugs_n_branches_revenge

bzr: ERROR: Server sent an unexpected error: ('error', '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

Traceback from .bzr.log:

9.270 fetch up to rev {<email address hidden>}
46.975 Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 849, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 795, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 495, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 913, in run
    use_existing_dir=use_existing_dir)
  File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 162, in _show_push_branch
    stop_revision=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1749, in push
    _override_hook_source_branch=_override_hook_source_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 2476, in _run_with_write_locked_target
    target.unlock()
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1520, in unlock
    self._unlock(branch_token, repo_token)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1488, in _unlock
    self._translate_error(err, token=str((branch_token, repo_token)))
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1361, in _translate_error
    self.repository._translate_error(err, branch=self, **context)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 359, in _translate_error
    self.bzrdir._translate_error(err, repository=self, **context)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 90, in _translate_error
    _translate_error(err, bzrdir=self, **context)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1796, in _translate_error
    raise errors.UnknownErrorFromSmartServer(err)
UnknownErrorFromSmartServer: Server sent an unexpected error: ('error', '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

46.976 return code 3

[Tim Penhey (thumper)]

On Sat, 08 Nov 2008 00:25:01 Martin Albisetti wrote:
> Happened to me again in:
> https://code.edge.launchpad.net/~beuno/launchpad/ui_bugs_n_branches_r
>evenge
>
> bzr: ERROR: Server sent an unexpected error: ('error',
> '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')
>
> Traceback from .bzr.log:
>
> 9.270 fetch up to rev
> {<email address hidden>}
> 46.975 Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line
> 849, in run_bzr_catch_errors return run_bzr(argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line
> 795, in run_bzr ret = run(*run_argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line
> 495, in run_argv_aliases return self.run(**all_cmd_args)
> File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line
> 913, in run use_existing_dir=use_existing_dir)
> File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 162,
> in _show_push_branch stop_revision=revision_id)
> File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line
> 138, in read_locked result = unbound(self, *args, **kwargs)
> File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line
> 1749, in push
> _override_hook_source_branch=_override_hook_source_branch) File
> "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 2476, in
> _run_with_write_locked_target target.unlock()
> File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line
> 1520, in unlock self._unlock(branch_token, repo_token)
> File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line
> 1488, in _unlock self._translate_error(err, token=str((branch_token,
> repo_token))) File
> "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1361, in
> _translate_error self.repository._translate_error(err, branch=self,
> **context) File "/usr/lib/python2.5/site-packages/bzrlib/remote.py",
> line 359, in _translate_error self.bzrdir._translate_error(err,
> repository=self, **context) File
> "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 90, in
> _translate_error _translate_error(err, bzrdir=self, **context)
> File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line
> 1796, in _translate_error raise
> errors.UnknownErrorFromSmartServer(err)
> UnknownErrorFromSmartServer: Server sent an unexpected error:
> ('error', '<ProtocolError for
> xmlrpc.lp.internal:8097/branchfilesystem: -1 >')
>
> 46.976 return code 3
>
>
> ** Changed in: launchpad-bazaar
> Importance: Undecided => High
> Status: New => Triaged

Can you pass this information on to Aaron? Preferably also giving him
access to the branch. Please don't delete the branch yet.

[Tim Penhey (thumper)]

Jono, can you take a look at this?

[Michael Hudson-Doyle (mwhudson)]

I expect it's the timeout due to the scanner processing the same branch thing.

[Martin Pool (mbp)]

bzr 2.4 no longer uses xmlrpc before accessing the branch over ssh, so this bug should become less important.

[Robert Collins (lifeless)]

This probably should be critical as it would OOPS today; it also has nothing to do with bzr's xmlrpc usage - note the *internal* name: this is the codehosting use of the internal xmlrpc service (part of our existing SOA infrastructure).

[Martin Pool (mbp)]

oh of course

[Martin Pool (mbp)]

Escalating to critical on the grounds it's a user-affecting external failure.

This is also impacting the udd package importer in for example http://package-import.ubuntu.com/status/language-pack-gnome-ca.html#2011-07-18 07:27:28.730318

Though it's a slightly different error:

bzrlib.errors.UnknownErrorFromSmartServer: Server sent an unexpected error: ('error', '<ProtocolError for xmlrpc.lp.internal:8097/codehosting: 503 Service Unavailable>')

 * i don't know if this is recording an oops
 * if there is an oops, the bzr server is failing to pass it through
 * the ssh server could possibly retry a couple of times (I don't know if it would help)

[Robert Collins (lifeless)]

You can grep on carob to find an oops; if there isn't one that should also be a bug (critical too) as thats crucial signal for us.

I think the ssh server probably wants to fail fast here and let the client try again (or not). However, the 503 may be due to asking a server that is being deployed to at the time, in which case, until we have the graceful-deploy tickets completed, this will be a rare, but expected situation.

[Martin Pool (mbp)]

One possible oops for this is OOPS-2022I35, where it's timing out after 9s updating the branch last-mirror time. It doesn't seem like it ought to be slow so perhaps the database was heavily loaded or locked at the time it happened...

I don't at the moment see any failures from the ssh front end but I may just have not looked.

[edA-qa (eda-qa)]

I got possibly this error using "Bazaar (bzr) 2.5.1" pushing existing source to a new branch. It then left the target directory in a broken state. Trying the same push again to a different title then suceedeed (so the error appears to be ephemeral).

edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_type
bzr: ERROR: Server sent an unexpected error: ('error', 'xmlrpclib.ProtocolError', '<ProtocolError for xmlrpc.lp.internal:8097/codehosting: 502 Bad Gateway>')
edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_type
bzr: ERROR: At lp:~eda-qa/leaflang/primordial_type you have a valid .bzr control directory, but not a branch or repository. This is an unsupported configuration. Please move the target directory out of the way and try again.
edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_types
Using default stacking branch /+branch-id/725500 at chroot-87084816:///~eda-qa/leaflang/
Created new stacked branch referring to /+branch-id/725500.