Some great discussion about this from our ops folk:

06:31 < elmo> lifeless: I do categorically think what I said in the RT, namely that sending mail in-request is crazy town
06:32 < elmo> lifeless: and while i'm OK with us doing some level of investigation about the current MTA performance, I don't think 230ms is all that bad
06:32 < elmo> lifeless: and fundamentally if you guys want to continue to send 100 mails and not time out, I think it's got to be made async from the request
06:33 < lifeless> elmo: so, yes, agreed.
06:33 < gary> We do have a planned story for that, on the basis of what elmo wrote on the RT--basically, back to the original plan, in fact.
06:33 < lifeless> elmo: working on all angles at once.
06:34 < lifeless> elmo: OTOH I don't want to reinvent postfix as a module in LP.
06:34 < lifeless> elmo: putting mails-to-send into the DB and actually spooling them elsewhere is equally nuts. We should definitely make them async.
06:34 < gary> yeah, that was basically the original argument.
06:35 < lifeless> putting a 'please notify about this <action>' statement into a work queue makes a lot of sense to me
06:36 < elmo> so I don't know enough about postfix, but with exim, there is certainly a way of using sendmail and saying "just queue this, do not do anything that might slow me down (e.g. any network stuff)"
06:36 < elmo> OTOH, even using that seems risky if you're going to have a 100+ people being mailed and a 1-2s timeout
06:36 < lifeless> elmo: what are we running on the appservers ?
06:36 < elmo> postfix
06:37 < lifeless> k
06:38 < gary> sending the emails is ideally transactionally aware.  I don't know if we do that now.
06:38 < gary> I think we do.
06:38 < gary> peripheral to this discussion; only pertinent if we have to write something fresh
06:39 < lamont> if you're mailing  with /usr/sbin/sendmail from postfix, it literally just drops a file in a directory and returns
06:39 < lamont> of course, CPU gets used elsewhere after that
06:39 < lifeless> lamont: we're using SMTP
06:40 < lifeless> hmm, perhaps we should be using the submission port - it might have different rules.
06:40  * lifeless wags
06:42 < lamont> like the good little MTA that it is, postfix puts the CPU intensive stuff squarely on port 25, so that spammers get slowed down too, not just him overloaded.
06:42 < lifeless> lamont: does it treat port 587 differently?
06:42 < lamont> submission port tends to get the same daemon, but with lighter rules in some ways
06:42 < lamont> and, of course, many things are tunable wrt what it checks in which instance of the smtpd process
06:43 < lifeless> lamont: if I filed an rt saying 'please make this as light as possible for lp on lp appservers', would that be a difficult task? or is 'very light' a well known thing ?
06:44 < lamont> do you connect to 127.0.0.1, or to the host's 91.189 iP?
06:44 < lifeless> localhost
06:44 < lifeless> (literally, thats whats configured)
06:44 < lamont> it's generally going to be ugly any way you come at it
06:44 < lifeless> so it will depend on /etc/hosts
06:45 < lamont> life would be simpler if you just used /usr/sbin/sendmail - even the interface is simpler
06:45 < lamont> why re-invent the MUA?
06:46 < lifeless> existing module from Zope.
06:46 < lamont> ah, zope reinvented it.  got it
06:47 < lamont> If the postfix config on the app servers is the default, and I suspect it is, it's about as light as it can get.
06:48 < elmo> it is the default
06:48 < elmo> modulo addition of a relayhost
06:49 < lamont> somewhere in the middle might be "use /usr/sbin/sendmail -bs, and pretend it's port 25"