hpss traceback with fault 8002 error

[Vincent Ladeuil]

Trying to commit to branch bound to lp, I got the following traceback:bzr commit -m 'selftest --starting-with now accepts multiple values'
Committing to: bzr+ssh://bazaar.launchpad.net/~vila/bzr/bzr.integration2/
modified NEWS
modified bzrlib/builtins.py
modified bzrlib/tests/__init__.py
modified bzrlib/tests/test_selftest.py
modified bzrlib/tests/blackbox/test_selftest.py
Committed revision 3669.
bzr: ERROR: bzrlib.errors.ErrorFromSmartServer: Error received from smart server: ('error', "<Fault 8002: 'error'>")

Traceback (most recent call last):
  File "/v/home/vila/src/bzr/trunk/bzrlib/commands.py", line 857, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/v/home/vila/src/bzr/trunk/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/v/home/vila/src/bzr/trunk/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/Volumes/home/vila/src/bzr/trunk/bzrlib/builtins.py", line 2318, in run
    exclude=safe_relpath_files(tree, exclude))
  File "/v/home/vila/src/bzr/trunk/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/v/home/vila/src/bzr/trunk/bzrlib/workingtree_4.py", line 242, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/v/home/vila/src/bzr/trunk/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/v/home/vila/src/bzr/trunk/bzrlib/mutabletree.py", line 197, in commit
    revprops=revprops, *args, **kwargs)
  File "/v/home/vila/src/bzr/trunk/bzrlib/commit.py", line 416, in commit
    self._cleanup()
  File "/v/home/vila/src/bzr/trunk/bzrlib/commit.py", line 600, in _cleanup
    raise e
ErrorFromSmartServer: Error received from smart server: ('error', "<Fault 8002: 'error'>")

bzr 1.7dev on python 2.5.2 (linux2)
arguments: ['/home/vila/bin/bzr', 'commit', '-m', 'selftest --starting-with now accepts multiple values']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /home/vila/.bazaar/plugins/bzrtools [1.6.0]
  defaultToUrllib /home/vila/.bazaar/plugins/defaultToUrllib [unknown]
  difftools /home/vila/.bazaar/plugins/difftools [0.91.0]
  email /home/vila/.bazaar/plugins/email [unknown]
  fastimport /home/vila/.bazaar/plugins/fastimport [unknown]
  grep /home/vila/.bazaar/plugins/grep [unknown]
  gtk /home/vila/.bazaar/plugins/gtk [0.96.0dev1]
  hgrep /home/vila/.bazaar/plugins/hgrep [unknown]
  launchpad /v/home/vila/src/bzr/trunk/bzrlib/plugins/launchpad [unknown]
  local_test_server /home/vila/.bazaar/plugins/local_test_server [unknown]
  loom /home/vila/.bazaar/plugins/loom [1.4.0dev0]
  pqm /home/vila/.bazaar/plugins/pqm [1.4.0dev0]
  stats /home/vila/.bazaar/plugins/stats [unknown]
  testprefixes /home/vila/.bazaar/plugins/testprefixes [unknown]
  transportstats /home/vila/.bazaar/plugins/transportstats [unknown]
  upload /home/vila/.bazaar/plugins/upload [0.1.0]
  usertest /home/vila/.bazaar/plugins/usertest [unknown]
  webdav /home/vila/.bazaar/plugins/webdav [1.6.0]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Note that I just committed to another branch (lp:~vila/bzr/bzr.integration) seconds before that one.

[Vincent Ladeuil (vila)]

Issuing the same command I got:

bzr commit -m 'selftest --starting-with now accepts multiple values'
Unable to obtain lock lp-140211244:///~vila/bzr/bzr.integration2/.bzr/branch/lock
held by <email address hidden> on host vostok [process #23631]
locked 6 minutes, 44 seconds ago
Will continue to try until 07:33:40, unless you press Ctrl-C
If you're sure that it's not being modified, use bzr break-lock lp-140211244:///~vila/bzr/bzr.integration2/.bzr/branch/lock
bzr: ERROR: Could not acquire lock "(remote lock)"

Note the ugly url.

And then:
bzr break-lock lp:~vila/bzr/bzr.integration2
Break lock bzr+ssh://<email address hidden>/%7Evila/bzr/bzr.integration2/.bzr/branch/lock
held by <email address hidden> on host vostok [process #23631]
locked 7 minutes, 58 seconds ago? [y/n]: y
vila:~/src/bzr/integration/trunk2 :) $ bzr commit -m 'selftest --starting-with now accepts multiple values'
Committing to: bzr+ssh://bazaar.launchpad.net/~vila/bzr/bzr.integration2/
bzr: ERROR: no changes to commit. use --unchanged to commit anyhow
vila:~/src/bzr/integration/trunk2 :( $ bzr missing lp:~vila/bzr/bzr.integration2
Branches are up to date.
vila:~/src/bzr/integration/trunk2 :) $

So, no harm done, but yet, we may want to avoid that kind of backtrace :)

[Andrew Bennetts (spiv)]

Judging from the error (it looks like a generic xmlrpclib.Fault happened on the server), I'm pretty sure this is a launchpad-bazaar issue rather than bazaar itself.

So, I'm reassigning this. launchpad-bazaar guys: feel free to add bzr back to this bug if you think it's appropriate, so long as you explain why :)

[Andrew Bennetts (spiv)]

Well, perhaps there is also a bzr issue... I suppose there's an argument that the server sending unexpected error messages like ('error', "<Fault 8002: 'error'>") shouldn't cause a traceback on the client, just a single ERROR line. Probably it would be clearest to have 2 bug reports:

 * one for launchpad-bazaar sending a "<Fault 8002: 'error'>" as an error for a perfectly acceptable operation, and
 * one for bzr giving a big ugly traceback when the server sends unintelligible generic error responses. I've filed bug 253527 for this.

[Jonathan Lange (jml)]

There should be an oops report for this in my inbox tomorrow :)

[Michael Hudson-Doyle (mwhudson)]

I don't think the authserver logs oopses, unfortunately.

I would _guess_ that this is a serialization error. Rummaging through the authserver logs can confirm or deny this.

There's already a bug about how break-lock gives you the wrong URL, I think.

[Jonathan Lange (jml)]

What a pity. I had thought we'd added OOPS logging. Maybe that was just improving 8002 error messages in tests.

Anyway, vila, is this error regularly reproducible or is it a once off? Did you get the error the same day you filed the bug?

[Vincent Ladeuil (vila)]

It occurred only once.

I filed the bug minutes after it occurred.

[Jonathan Lange (jml)]

Thanks vila. That mean we need to look in the authserver logs for 2008-09-01 at around 0630 UTC.

[Jonathan Lange (jml)]

Looks like you were right, Michael:
        psycopg2.extensions.TransactionRollbackError: could not serialize access due to concurrent update

Full trace at https://pastebin.canonical.com/8825/ for those with access.

I guess this should go away once we switch over to the zope XMLRPC server?

[Stuart Bishop (stub)]

On Thu, Sep 4, 2008 at 2:28 PM, Jonathan Lange <email address hidden> wrote:
> Looks like you were right, Michael:
> psycopg2.extensions.TransactionRollbackError: could not serialize access due to concurrent update
>
> Full trace at https://pastebin.canonical.com/8825/ for those with
> access.
>
> I guess this should go away once we switch over to the zope XMLRPC
> server?

Yes - the Zope3 environment correctly handles serialize exceptions.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

[Jonathan Lange (jml)]

In that case, the bug should be fixed in the next rollout.