Payment notification processor should handle DB connection resets

[Anthony Lenton]

On 7/03 at 9:09AM the prod DB was reset. At that point the payment notification processor service failed with the following error:

2012-07-03 09:09:09,690 - root - ERROR - Process payment notifications failed:
2012-07-03 09:09:09,850 - root - ERROR - Traceback (most recent call last):
2012-07-03 09:09:09,850 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/src/softwarecenteragent/scripts/process_pa
yment_notifications", line 37, in <module>
2012-07-03 09:09:09,850 - root - ERROR - call_command('process-payment-notifications')
2012-07-03 09:09:09,850 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/core/management/_
_init__.py", line 166, in call_command
2012-07-03 09:09:09,850 - root - ERROR - return klass.execute(*args, **defaults)
2012-07-03 09:09:09,850 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/core/management/b
ase.py", line 220, in execute
2012-07-03 09:09:09,850 - root - ERROR - output = self.handle(*args, **options)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/src/softwarecenteragent/management/command
s/process-payment-notifications.py", line 35, in handle
2012-07-03 09:09:09,851 - root - ERROR - include_initiated=include_initiated)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/src/softwarecenteragent/managers.py", line 176, in process_payment_notifications
2012-07-03 09:09:09,851 - root - ERROR - % subscriptions.count())
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/models/query.py", line 334, in count
2012-07-03 09:09:09,851 - root - ERROR - return self.query.get_count(using=self.db)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/models/sql/query.py", line 401, in get_count
2012-07-03 09:09:09,851 - root - ERROR - number = obj.get_aggregation(using=using)[None]
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/models/sql/query.py", line 367, in get_aggregation
2012-07-03 09:09:09,851 - root - ERROR - result = query.get_compiler(using).execute_sql(SINGLE)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/models/sql/compiler.py", line 735, in execute_sql
2012-07-03 09:09:09,851 - root - ERROR - cursor.execute(sql, params)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
2012-07-03 09:09:09,852 - root - ERROR - return self.cursor.execute(query, args)
2012-07-03 09:09:09,852 - root - ERROR - DatabaseError: terminating connection due to administrator command
2012-07-03 09:09:09,852 - root - ERROR - SSL connection has been closed unexpectedly
2012-07-03 09:09:09,852 - root - ERROR -
2012-07-03 09:09:09,852 - root - ERROR -

And continue to fail like that for the rest of the day, until the process was reset due to log rotation.

The payment notification processor should handle DB connection resets. I'll file a separate bug to improve monitoring so that we find out about this kind of issues faster.