E-Mail sending iteration stops after first failure

[Martin]

 1) Steps to reproduce the issue you have observed

    A project issue with multiple followers exist. Add a message ("send message") to the project issue. The SMTP server does, however, not accept more than five emails in a row.

 2) The result you observed

    Multiple emails are sent, one per recipient (= followers minus the current user). If there is an exception, such as SMTPServerDisconnected("Connection unexpectedly closed"), when sending to one of the recipients no attempt is taken to deliver the email to the remaining recipients.

 3) The result you expected

    If there is a problem with sending an email, this should not affect other recipients. Also, if the SMTP server closed the connection at this point, further attempts to send out the email must be taken.

 4) The platform you are using (and browser version, if relevant)

    Debian, MTA is a Microsoft product, AFAIK.

 5) The OpenERP version you are using

    7.0-20131126-00264

[Olivier Dony (Odoo) (odo-openerp)]

Firstly, the feedback to your related bug reports (bug 1255611 and bug 1255615) is relevant here.

Secondly, this bug report is correct in the sense that when sending out a multi-recipient mail_mail, the mail_mail.send() method does not keep track of which of the recipients have successfully been sent an email.

In the presence of an outgoing SMTP server that does not reliably accept outgoing messages, this means that all recipients are considered failed at once, with 2 consequences:
 - by default the recipients that have not yet received the email will never receive it unless a manual action is taken
 - if a manual retry of the failed delivery is attempted by the system administrator (via /Settings/Technical/Emails/Emails), all the recipients will be delivered again, possibly leading to duplicate messages *and* probably triggering the throttle limit again, making no further progress in the delivery.

So, in case of partial delivery failure the system could keep track of recipients who have already been delivered, and only retry the remaining ones. This does not solve the problem of handling the automated retry though.
However once we are able to partially retry a failed mail, we could imagine to distinguish permanent (5xx) and temporary (4xx) delivery failures, and automatically retry the temporary ones.

Note that this is considered a "nice-to-have" improvement rather than a bug, but mail delivery is sufficiently important to qualify it as a Low priority bug rather than a wishlist.

Thanks for your detailed bug report!

[Martin (debacle)]

If I understand correctly, temporarily failures of an SMTP server are still "correct behaviour", i.e. a correctly configured SMTP server might react that way. The client needs to retry. Assuming an SMTP server that is able to accept email all of the time is wrong. Note: My knowledge of SMTP servers is rather limited, so correct me if I'm wrong etc.

[Olivier Dony (Odoo) (odo-openerp)]

On 11/28/2013 04:35 PM, W. Martin Borgert wrote:
> If I understand correctly, temporarily failures of an SMTP server are
> still "correct behaviour", i.e. a correctly configured SMTP server might
> react that way.

Sure, it's a valid SMTP protocol situation, but not a common nor expected
behavior for a relay SMTP (i.e. your own local post office). Especially for
valid, authenticated clients. If there is any reason to temporarily delay the
actual delivery, the relay SMTP should simply queue it internally until it can
be delivered, not push the burden back on the MUA. An authenticated relay
client is not supposed to be treated like an unknown spammer.
Basically we're talking about "relaying errors" here, not "delivery errors".
This is about an MTA not accepting an outbound email from one of its authorized
client, not a final delivery failure (recipient does not exist, etc.), which is
only known later *after* the mail has been accepted for relay.

In fact most email clients I know (e.g. Thunderbird) will throw up if the relay
SMTP does that, and give back the relay error message to the user, who will
panic and try to find someone who knows how to fix "the emails". While actually
everything could have been just fine if the relay did its job properly.

So yes it could happen (e.g. when you are trying to setup your outgoing SMTP
servers and the SMTP relay does not recognize you), but it is not supposed to
happen as soon as the SMTP relay settings are correct.

OpenERP being nothing more than a basic MUA here, it does not expect it, and
does not really know how to deal with the situation. However it must not block
all email-related operations because that's worse. Many email notifications are
sent indirectly by the system, e.g. when you change the status of a document,
and it would be unthinkable to block them all until the SMTP relay decides it's
time to accept messages again.

> The client needs to retry. Assuming an SMTP server that
> is able to accept email all of the time is wrong. Note: My knowledge of
> SMTP servers is rather limited, so correct me if I'm wrong etc.

OpenERP can retry, since the messages are all queued. But the retry is not
automatic, just like it would not be in Thunderbird. The difference is that
OpenERP currently leaves the responsibility of retrying to the admin instead of
the users, who should not need to care about this.
And again, the admin should normally never need to do anything if the SMTP
settings are correct and if the relay MTA does its job properly.

This is the current situation in OpenERP 7.0. It is not ideal but it will work
fine in most situations, for the reasons explained above.

And now we can further improve this behavior by automatically re-queuing the
non-permanent relay errors. For permanent relaying errors we could also post a
notification in the sender's Messaging/Inbox inside OpenERP, which they may
notice even if the SMTP config is totally broken.

[Martin (debacle)]

Thanks for the explanation!

Unfortunately, I'm not sure how I can solve the problem for me, because the MTA is outside of my control.

On the OpenERP side, there are two connected issues:

 1. Informing the admin about the problem. I cannot assume, that admins will proactively look into a queue at regular intervals. I'm not sure how to do this, because the admin might only login and check the queue, if a user reports a problem. Waiting until another person (maybe a client) gets angry for not receiving messages might be a bad idea business-wise. IMHO, the users themselves must be able to see the problem, so that they can tell the admin "please check, my message shows a strange error"

 2. When the admin got informed about the problem, they must be able to re-transmit any single email, that got stuck

[Martin (debacle)]

Just a note to other poor victims of MS mail server products: I don't know how, but it is possible to set throttle limit higher, which makes the problem less pressing.