send-bug-notifications.py needs to be parallelised or made more efficient

[Graham Binns]

send-mail-notifications.py can't handle large numbers of pending notifications. For example, in the case of bug #223962, large numbers of notifications were being generated by checkwatches and send-bug-notifications couldn't process them fast enough to keep up.

We need to do one of two things:

 1. Re-write send-bug-notifications so that it can be forked and parallelised safely.
 2. Make it efficient enough so that it can deal with high volumes of notifications sequentially.

Option 1 seems more likely to be doable that option 2.

[Stuart Bishop (stub)]

Do we know where the bottleneck is?

If in the SMTP sending, a pool of senders pulling items off a queue and sending them is easy enough to implement using threads or just asking the admins if the SMTP server can be made to accept deliveries faster.

If it is generating the notifications to send, multiple processes or threads might not help as they have multiple caches that need to be populated each transaction. Running the script in autocommit mode might be a quick way of getting the cache shared across transactions.

[Björn Tillenius (bjornt)]

Marking this bug as Incomplete, since we need to figure out what is taking time, before trying to optimize it.

Having a quick look at the log, the actual sending of the notifications take quite a while. Having that go faster would improve things. Another thing that is taking long is to actually build the notification to be sent. Not sure what's taking long there, we need to debug more.

[Björn Tillenius (bjornt)]

It seems that a lot of time is spent talking to the SMTP server. On forster, it takes a few seconds to just send the e-mails. When setting send_email to false on staging, so that send-bug-notifications.py do all its work, except for talking to the SMTP server, it takes less than a second: https://pastebin.canonical.com/4785/

On forster, a similar notification about the same bug took around 8 seconds to send out.

The main reason we see send-bug-notifications.py taking longer than normal, is that quite a lot people are subscribing to all Ubuntu bugs. Since most of our bug notifications are about Ubuntu bugs, the run time increases a bit with each subscriber.

Our best bet is probably to send the same copy of the bug notification to all Ubuntu subscribers; so that we talk to the SMTP server only once, no matter how many Ubuntu subscribers there are.

This would have been easy, if we didn't set the To header to the e-mail we're sending to. The SMTP might be able to inject this header, but we'll have to check. Another option is to change the To address to be the bug address. This is something I think we should do anyway, since it's more correct.

[Stuart Bishop (stub)]

Tom - can you have a quick look into speeding up the SMTP server that this script is talking too? It likely has anti-spam-relay delays and doing DNS checks synchronously. As this script is a trusted sender on the LAN and is too dumb to deal with failures due to dud MX records and whatnot anyway, we might be able to fix this issue with a few tweaks on the SMTP server.

[Stuart Bishop (stub)]

An SMTP server on localhost might solve the issue too if the script is talking to some central server. Our script can talk inefficiently to the local SMTP server and it talks efficiently to the main outgoing SMTP relay.

[Björn Tillenius (bjornt)]

Lowered the priority, since the run times are usually well below 5 minutes. It's not due to cause any real issues in the short term. If there are issues, we could probably get away with unsubscribing people from Ubuntu, and subscribe them to the ubuntu-bugs mailing list.