send-bug-notifications.py can't handle rejections due to oversized messages

[Steve McInerney]

has been script_activity alerting for some hours now.

Am seeing entries in the log like this:
2012-01-15 21:18:16 ERROR Unhandled exception
 -> http://launchpadlibrarian.net/90161643/qUTipcmcOXKO7I6lfms6IPKVeCn.txt ((552, '5.3.4 Message size exceeds fixed limit', '<email address hidden>'))
2012-01-15 21:18:16 ERROR Error while building email notifications.
 -> http://launchpadlibrarian.net/90161646/vu4qEui592LWTIbDgDDQeuotIOK.txt (GeneratorExit)
2012-01-15 21:20:13 INFO Logging INFO and higher messages to /srv/launchpad.net/production-logs/send-bug-notifications.log
2012-01-15 21:20:13 INFO Creating lockfile: /var/lock/launchpad-send-bug-notifications.lock

which may account for failing to successfully complete; yet other runs appear to complete successfully, so tis rather odd.

Additionally, while it is running and processing at least some of the time; scriptactivity seems devoid of starter information:
launchpad_prod_1=# SELECT date_started, date_completed FROM scriptactivity WHERE name= 'send-bug-notifications' ORDER BY date_started DESC LIMIT 20;
        date_started | date_completed
----------------------------+----------------------------
 2012-01-15 11:45:31.402454 | 2012-01-15 11:45:43.339172
 2012-01-15 11:40:42.31134 | 2012-01-15 11:41:01.144317
 2012-01-15 11:35:28.838547 | 2012-01-15 11:35:45.528171
 2012-01-15 11:30:36.711617 | 2012-01-15 11:30:43.799517

I'd expect to see a new entry for each startup; just a lack of date_completed.

[Graham Binns (gmb)]

The error was as follows:

SMTPSenderRefused: (552, '5.3.4 Message size exceeds fixed limit', '<email address hidden>')

Steve and I used https://pastebin.canonical.com/58139/ to find the offending message (turned out to be message 11441321). We confirmed this with https://pastebin.canonical.com/58140/. The message is about 18M characters in size where everything else is ~500.

Our quick fix for the problem was to mark the bugnotification linked to the message as date_emailed = now() at time zone 'UTC'. However, send-bug-notifications.py should at least OOPS when it encounters this problem, rather than just exiting.

[Graham Binns (gmb)]

Steve also sez:

<spm> sounds like a plethora of bugs there tho. unable to reject bad data and continue on - wjhile somehow alerting a human that bad stuff needs poking at; no oopsing; no scriptactivity startup time

[Robert Collins (lifeless)]

This smells like fallout from the last fix for this... I forget the
bug number but you can probably find it from the incident report.

[Robert Collins (lifeless)]

See bug 788874

[Launchpad QA Bot (lpqabot)]

Fixed in stable r17682 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/17682>.