TransactionRollbackError on /auth/complete causes OOPS and OpenID login failure

[Roman Yepishev]

This bug report serves as a master bug report for the issues with logging in that were caused by TransactionRollbackError on the server-side.

These errors are temporary, usually next attempt will succeed but this definitely needs to be looked into and find why the transaction took that long.

Traceback is:
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/sql/query.py, line 1734, in execute_sql
    cursor.execute(sql, params)
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/sql/subqueries.py, line 34, in do_query
    self.execute_sql(None)
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/sql/subqueries.py, line 86, in delete_batch
    self.do_query(self.model._meta.db_table, where)
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/query.py, line 870, in delete_objects
    del_query.delete_batch(pk_list)
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/base.py, line 443, in delete
    delete_objects(seen_objs)
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/contrib/sessions/backends/db.py, line 71, in delete
    Session.objects.get(session_key=session_key).delete()
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/contrib/sessions/backends/base.py, line 250, in cycle_key
    self.delete(key)
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/contrib/auth/__init__.py, line 64, in login
    request.session.cycle_key()
  Module /usr/lib/python2.5/site-packages/django_openid_auth/views.py, line 209, in login_complete
    auth_login(request, user)
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/core/handlers/base.py, line 86, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/core/handlers/wsgi.py, line 239, in __call__
    response = self.get_response(request)
  Module /var/lib/python-support/python2.5/paste/translogger.py, line 67, in __call__
    return self.application(environ, replacement_start_response)
TransactionRollbackError: ' could not serialise access due to concurrent update'

[John O'Brien (jdobrien)]

This appears to be coming from the django-openid code package and not any of our specific code.

On 07/27/2010 05:33 AM, Roman Yepishev wrote:
> Public bug reported:
>
> This bug report serves as a master bug report for the issues with
> logging in that were caused by TransactionRollbackError on the server-
> side.
>
> These errors are temporary, usually next attempt will succeed but this
> definitely needs to be looked into and find why the transaction took
> that long.
>
> Traceback is:
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/sql/query.py, line 1734, in execute_sql
> cursor.execute(sql, params)
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/sql/subqueries.py, line 34, in do_query
> self.execute_sql(None)
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/sql/subqueries.py, line 86, in delete_batch
> self.do_query(self.model._meta.db_table, where)
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/query.py, line 870, in delete_objects
> del_query.delete_batch(pk_list)
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/db/models/base.py, line 443, in delete
> delete_objects(seen_objs)
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/contrib/sessions/backends/db.py, line 71, in delete
> Session.objects.get(session_key=session_key).delete()
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/contrib/sessions/backends/base.py, line 250, in cycle_key
> self.delete(key)
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/contrib/auth/__init__.py, line 64, in login
> request.session.cycle_key()
> Module /usr/lib/python2.5/site-packages/django_openid_auth/views.py, line 209, in login_complete
> auth_login(request, user)
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/core/handlers/base.py, line 86, in get_response
> response = callback(request, *callback_args, **callback_kwargs)
> Module /srv/ubuntuone.com/production/ubunet-rev-2946/utilities/../lib/django/core/handlers/wsgi.py, line 239, in __call__
> response = self.get_response(request)
> Module /var/lib/python-support/python2.5/paste/translogger.py, line 67, in __call__
> return self.application(environ, replacement_start_response)
> TransactionRollbackError: ' could not serialise access due to concurrent update'
>
> ** Affects: ubuntuone-servers
> Importance: Medium
> Assignee: Ubuntu One web team (ubuntuone-web)
> Status: New
>
> ** Visibility changed to: Public
>

[Elliot Murphy (statik)]

On Tue, Jul 27, 2010 at 7:42 AM, John O'Brien <email address hidden> wrote:
> This appears to be coming from the django-openid code package and not any of our specific code.

jdo probably already knows all this, but I want to clarify this so
everyone who wasn't around when we took over the project understands
our relationship with it.

The django-openid code was originally written by Simon Willison, and
is maintained as a separate open source project outside the
ubuntuone-servers tree so it might not feel like "our" code. However,
it's important for everyone to realize that the django-openid code
should be viewed as "our" code, and we are responsible for maintaining
the project and the package in Ubuntu, fixing bugs, merging patches
from the community, uploading new versions to Ubuntu, etc. James
Henstridge is an expert on this code and was working on doing a new
release for Ubuntu last week.

This project is also used by several other teams at Canonical and
several Ubuntu community sites (LoCo directory, etc.), so bugfixes to
it benefit a lot of sites.
--
Elliot Murphy | https://launchpad.net/~statik/

[John O'Brien (jdobrien)]

On 07/27/2010 11:02 AM, Elliot Murphy wrote:
> On Tue, Jul 27, 2010 at 7:42 AM, John O'Brien<email address hidden> wrote:
>
>> This appears to be coming from the django-openid code package and not any of our specific code.
>>
> jdo probably already knows all this, but I want to clarify this so
> everyone who wasn't around when we took over the project understands
> our relationship with it.
>
> The django-openid code was originally written by Simon Willison, and
> is maintained as a separate open source project outside the
> ubuntuone-servers tree so it might not feel like "our" code. However,
> it's important for everyone to realize that the django-openid code
> should be viewed as "our" code, and we are responsible for maintaining
> the project and the package in Ubuntu, fixing bugs, merging patches
> from the community, uploading new versions to Ubuntu, etc. James
> Henstridge is an expert on this code and was working on doing a new
> release for Ubuntu last week.
>
> This project is also used by several other teams at Canonical and
> several Ubuntu community sites (LoCo directory, etc.), so bugfixes to
> it benefit a lot of sites.
>
On that thought, it is also very likely that a part of our code is not
handling a transaction correctly and is locking the user record causing
this problem during authentication.

[Martin Albisetti (beuno)]

I am seeing dozens of these in our logs. Seems to be the most popular oops.