send-bug-notifications.py can't handle rejections due to oversized messages

Bug #916939 reported by Steve McInerney on 2012-01-15
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Critical
Colin Watson

Bug Description

has been script_activity alerting for some hours now.

Am seeing entries in the log like this:
2012-01-15 21:18:16 ERROR Unhandled exception
 -> http://launchpadlibrarian.net/90161643/qUTipcmcOXKO7I6lfms6IPKVeCn.txt ((552, '5.3.4 Message size exceeds fixed limit', '<email address hidden>'))
2012-01-15 21:18:16 ERROR Error while building email notifications.
 -> http://launchpadlibrarian.net/90161646/vu4qEui592LWTIbDgDDQeuotIOK.txt (GeneratorExit)
2012-01-15 21:20:13 INFO Logging INFO and higher messages to /srv/launchpad.net/production-logs/send-bug-notifications.log
2012-01-15 21:20:13 INFO Creating lockfile: /var/lock/launchpad-send-bug-notifications.lock

which may account for failing to successfully complete; yet other runs appear to complete successfully, so tis rather odd.

Additionally, while it is running and processing at least some of the time; scriptactivity seems devoid of starter information:
launchpad_prod_1=# SELECT date_started, date_completed FROM scriptactivity WHERE name= 'send-bug-notifications' ORDER BY date_started DESC LIMIT 20;
        date_started | date_completed
----------------------------+----------------------------
 2012-01-15 11:45:31.402454 | 2012-01-15 11:45:43.339172
 2012-01-15 11:40:42.31134 | 2012-01-15 11:41:01.144317
 2012-01-15 11:35:28.838547 | 2012-01-15 11:35:45.528171
 2012-01-15 11:30:36.711617 | 2012-01-15 11:30:43.799517

I'd expect to see a new entry for each startup; just a lack of date_completed.

Related branches

Steve McInerney (spm) on 2012-01-15
Changed in launchpad:
importance: Undecided → Critical
tags: added: canonical-losa-lp
Graham Binns (gmb) on 2012-01-15
summary: - cronscripts/send-bug-notifications.py is failing to successfully
- complete
+ send-bug-notifications.py can't handle rejections due to oversized
+ messages
Graham Binns (gmb) wrote :

The error was as follows:

SMTPSenderRefused: (552, '5.3.4 Message size exceeds fixed limit', '<email address hidden>')

Steve and I used https://pastebin.canonical.com/58139/ to find the offending message (turned out to be message 11441321). We confirmed this with https://pastebin.canonical.com/58140/. The message is about 18M characters in size where everything else is ~500.

Our quick fix for the problem was to mark the bugnotification linked to the message as date_emailed = now() at time zone 'UTC'. However, send-bug-notifications.py should at least OOPS when it encounters this problem, rather than just exiting.

tags: added: oops
Changed in launchpad:
status: New → Triaged
Graham Binns (gmb) wrote :

Steve also sez:

<spm> sounds like a plethora of bugs there tho. unable to reject bad data and continue on - wjhile somehow alerting a human that bad stuff needs poking at; no oopsing; no scriptactivity startup time

This smells like fallout from the last fix for this... I forget the
bug number but you can probably find it from the incident report.

Robert Collins (lifeless) wrote :
William Grant (wgrant) on 2012-10-22
tags: added: email
Colin Watson (cjwatson) on 2015-07-15
Changed in launchpad:
assignee: nobody → Colin Watson (cjwatson)
status: Triaged → In Progress
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
Colin Watson (cjwatson) on 2015-08-06
tags: added: qa-untestable
removed: qa-needstesting
William Grant (wgrant) on 2015-08-10
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers