zope.sendmail - stale lock links in queue prevent timely delivery after recovery from crash

[Matthew Grant]

After a zope.sendmail queue process thread delivery crash, stale lock links can remain in the queue, prevent timely delivery of mesages that are waiting to be sent - they become 'stuck'.

Solution is to implement a queue cleaning routine that deletes all the '.sending-*' lock links on thread startup, that is optionally turned on by a flag in the zope.sendmail configuration zcml.

[Benji York (benji)]

Matthew Grant wrote:
> Solution is to implement a queue cleaning routine that deletes all the
> '.sending-*' lock links on thread startup, that is optionally turned on
> by a flag in the zope.sendmail configuration zcml.

The problem with that approach is that if there are multiple
threads/processes sharing a queue, some non-stale locks will be cleared
unnecessarily.

There is already code for dealing with stale locks in zope.sendmail. It
may be flawed, but that's the first place to go for solving this.

[Matthew Grant (grantma)]

The stale locks code in zope.sendmail is flawed such that stale messages
wont be sent for 3 hours (MAX_SEND_TIME = 3*60*60, delivery.py) if the
mail server is restarted. For the business I am in we require that the
stale messages be sent on sever restart (For a lot of enterprise and ISP
environments this is an operational requirement). By default this
feature is turned off, so that the current status quo with the algorithm
is kept. It is there for when you need it, and it's functionality is
covered by the unit tests I have written. I am not forcing this
functionality on everyone, but just putting it there for those who need
it.

zope.sendmail is not written to be a full MTA, just an SMTP client that
sends its messages on to a proper MTA. It does not have a per message
delivery state tracking database(s)(Berkerley DB) like Exim4 or
sendmail. I am trying to keep it simple, but reliable for the SMTP
client role. If you want to run it directly deliver reliably to all the
end-point SMTP servers on the Internet, it is not going to work reliably
unless we build a full MTA SMTP agent with per message state machines.

On Mon, 2008-03-03 at 16:27 +0000, Benji York wrote:
> Matthew Grant wrote:
> > Solution is to implement a queue cleaning routine that deletes all the
> > '.sending-*' lock links on thread startup, that is optionally turned on
> > by a flag in the zope.sendmail configuration zcml.
>
> The problem with that approach is that if there are multiple
> threads/processes sharing a queue, some non-stale locks will be cleared
> unnecessarily.
>
> There is already code for dealing with stale locks in zope.sendmail. It
> may be flawed, but that's the first place to go for solving this.
>

[Benji York (benji)]

Matthew Grant wrote:
> The stale locks code in zope.sendmail is flawed such that stale messages
> wont be sent for 3 hours (MAX_SEND_TIME = 3*60*60, delivery.py) if the
> mail server is restarted.

If there is a flaw, we should repair it. OTOH, there has to be some
appreciable amount of time to leave lock links alone before declaring
them stale because another thread/process may be delivering it when a
process is restarted.

> For the business I am in we require that the
> stale messages be sent on sever restart (For a lot of enterprise and ISP
> environments this is an operational requirement).

That assumes that there is only one process servicing the queue. For
many installations that's not a safe assumption. Perhaps
DirectMailDelivery with a real, local MTA would be more appropriate for
your needs.

> By default this
> feature is turned off, so that the current status quo with the algorithm
> is kept. It is there for when you need it, and it's functionality is
> covered by the unit tests I have written. I am not forcing this
> functionality on everyone, but just putting it there for those who need
> it.

Code always has a cost. This code may be worth it, it doesn't seem so
to me.

> zope.sendmail is not written to be a full MTA, just an SMTP client that
> sends its messages on to a proper MTA. It does not have a per message
> delivery state tracking database(s)(Berkerley DB) like Exim4 or
> sendmail. I am trying to keep it simple, but reliable for the SMTP
> client role. If you want to run it directly deliver reliably to all the
> end-point SMTP servers on the Internet, it is not going to work reliably
> unless we build a full MTA SMTP agent with per message state machines.

I can't see the connection the above paragraph has with the present
discussion.

[Colin Watson (cjwatson)]

The zope.sendmail project on Launchpad has been archived at the request of the Zope developers (see https://answers.launchpad.net/launchpad/+question/683589 and https://answers.launchpad.net/launchpad/+question/685285). If this bug is still relevant, please refile it at https://github.com/zopefoundation/zope.sendmail.