"ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem" when pushing a branch

Bug #294726 reported by Julian Edwards on 2008-11-06
38
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Unassigned

Bug Description

I was pushing up a branch change with 1.9rc1-1 and got this exception:

UnknownErrorFromSmartServer: Server sent an unexpected error: ('error',
'<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

OOPS-2030DZ37 seems to be an example of this.

Doing a push, wait 30 seconds, push again, is likely to trigger this while the scanner is running.

Revision history for this message
Martin Albisetti (beuno) wrote :

Happened to me again in: https://code.edge.launchpad.net/~beuno/launchpad/ui_bugs_n_branches_revenge

bzr: ERROR: Server sent an unexpected error: ('error', '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

Traceback from .bzr.log:

9.270 fetch up to rev {<email address hidden>}
46.975 Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 849, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 795, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 495, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 913, in run
    use_existing_dir=use_existing_dir)
  File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 162, in _show_push_branch
    stop_revision=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1749, in push
    _override_hook_source_branch=_override_hook_source_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 2476, in _run_with_write_locked_target
    target.unlock()
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1520, in unlock
    self._unlock(branch_token, repo_token)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1488, in _unlock
    self._translate_error(err, token=str((branch_token, repo_token)))
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1361, in _translate_error
    self.repository._translate_error(err, branch=self, **context)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 359, in _translate_error
    self.bzrdir._translate_error(err, repository=self, **context)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 90, in _translate_error
    _translate_error(err, bzrdir=self, **context)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1796, in _translate_error
    raise errors.UnknownErrorFromSmartServer(err)
UnknownErrorFromSmartServer: Server sent an unexpected error: ('error', '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')

46.976 return code 3

Changed in launchpad-bazaar:
importance: Undecided → High
status: New → Triaged
Revision history for this message
Tim Penhey (thumper) wrote : Re: [Bug 294726] Re: Exception when pushing a change

On Sat, 08 Nov 2008 00:25:01 Martin Albisetti wrote:
> Happened to me again in:
> https://code.edge.launchpad.net/~beuno/launchpad/ui_bugs_n_branches_r
>evenge
>
> bzr: ERROR: Server sent an unexpected error: ('error',
> '<ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem: -1 >')
>
> Traceback from .bzr.log:
>
> 9.270 fetch up to rev
> {<email address hidden>}
> 46.975 Traceback (most recent call last):
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line
> 849, in run_bzr_catch_errors return run_bzr(argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line
> 795, in run_bzr ret = run(*run_argv)
> File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line
> 495, in run_argv_aliases return self.run(**all_cmd_args)
> File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line
> 913, in run use_existing_dir=use_existing_dir)
> File "/usr/lib/python2.5/site-packages/bzrlib/push.py", line 162,
> in _show_push_branch stop_revision=revision_id)
> File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line
> 138, in read_locked result = unbound(self, *args, **kwargs)
> File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line
> 1749, in push
> _override_hook_source_branch=_override_hook_source_branch) File
> "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 2476, in
> _run_with_write_locked_target target.unlock()
> File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line
> 1520, in unlock self._unlock(branch_token, repo_token)
> File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line
> 1488, in _unlock self._translate_error(err, token=str((branch_token,
> repo_token))) File
> "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 1361, in
> _translate_error self.repository._translate_error(err, branch=self,
> **context) File "/usr/lib/python2.5/site-packages/bzrlib/remote.py",
> line 359, in _translate_error self.bzrdir._translate_error(err,
> repository=self, **context) File
> "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 90, in
> _translate_error _translate_error(err, bzrdir=self, **context)
> File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line
> 1796, in _translate_error raise
> errors.UnknownErrorFromSmartServer(err)
> UnknownErrorFromSmartServer: Server sent an unexpected error:
> ('error', '<ProtocolError for
> xmlrpc.lp.internal:8097/branchfilesystem: -1 >')
>
> 46.976 return code 3
>
>
> ** Changed in: launchpad-bazaar
> Importance: Undecided => High
> Status: New => Triaged

Can you pass this information on to Aaron? Preferably also giving him
access to the branch. Please don't delete the branch yet.

Revision history for this message
Tim Penhey (thumper) wrote : Re: Exception when pushing a change

Jono, can you take a look at this?

Changed in launchpad-bazaar:
assignee: nobody → jml
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I expect it's the timeout due to the scanner processing the same branch thing.

Jonathan Lange (jml) on 2008-12-13
Changed in launchpad-bazaar:
assignee: jml → nobody
Revision history for this message
Martin Pool (mbp) wrote :

bzr 2.4 no longer uses xmlrpc before accessing the branch over ssh, so this bug should become less important.

Revision history for this message
Robert Collins (lifeless) wrote :

This probably should be critical as it would OOPS today; it also has nothing to do with bzr's xmlrpc usage - note the *internal* name: this is the codehosting use of the internal xmlrpc service (part of our existing SOA infrastructure).

Revision history for this message
Martin Pool (mbp) wrote : Re: [Bug 294726] Re: "ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem" when pushing a branch

oh of course

Revision history for this message
Martin Pool (mbp) wrote :

Escalating to critical on the grounds it's a user-affecting external failure.

This is also impacting the udd package importer in for example http://package-import.ubuntu.com/status/language-pack-gnome-ca.html#2011-07-18 07:27:28.730318

Though it's a slightly different error:

bzrlib.errors.UnknownErrorFromSmartServer: Server sent an unexpected error: ('error', '<ProtocolError for xmlrpc.lp.internal:8097/codehosting: 503 Service Unavailable>')

 * i don't know if this is recording an oops
 * if there is an oops, the bzr server is failing to pass it through
 * the ssh server could possibly retry a couple of times (I don't know if it would help)

Changed in launchpad:
importance: High → Critical
Revision history for this message
Robert Collins (lifeless) wrote :

You can grep on carob to find an oops; if there isn't one that should also be a bug (critical too) as thats crucial signal for us.

I think the ssh server probably wants to fail fast here and let the client try again (or not). However, the 503 may be due to asking a server that is being deployed to at the time, in which case, until we have the graceful-deploy tickets completed, this will be a rare, but expected situation.

Revision history for this message
Martin Pool (mbp) wrote :

One possible oops for this is OOPS-2022I35, where it's timing out after 9s updating the branch last-mirror time. It doesn't seem like it ought to be slow so perhaps the database was heavily loaded or locked at the time it happened...

I don't at the moment see any failures from the ssh front end but I may just have not looked.

description: updated
Curtis Hovey (sinzui) on 2012-11-29
tags: added: codehosting-ssh
tags: added: oops
Revision history for this message
edA-qa (eda-qa) wrote :

I got possibly this error using "Bazaar (bzr) 2.5.1" pushing existing source to a new branch. It then left the target directory in a broken state. Trying the same push again to a different title then suceedeed (so the error appears to be ephemeral).

edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_type
bzr: ERROR: Server sent an unexpected error: ('error', 'xmlrpclib.ProtocolError', '<ProtocolError for xmlrpc.lp.internal:8097/codehosting: 502 Bad Gateway>')
edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_type
bzr: ERROR: At lp:~eda-qa/leaflang/primordial_type you have a valid .bzr control directory, but not a branch or repository. This is an unsupported configuration. Please move the target directory out of the way and try again.
edA-qa@silent-red:/opt/leaf/primordial_type$ bzr push lp:~eda-qa/leaflang/primordial_types
Using default stacking branch /+branch-id/725500 at chroot-87084816:///~eda-qa/leaflang/
Created new stacked branch referring to /+branch-id/725500.

Airkm (airkm) on 2019-02-24
information type: Public → Private
William Grant (wgrant) on 2019-02-25
information type: Private → Public
Changed in launchpad:
importance: Critical → High
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers