ESTALE in cluster configuration
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
GNU Mailman |
New
|
Undecided
|
Unassigned |
Bug Description
I am running a Mailman cluster with two servers sharing archives/
qfiles/logs directories are placed on each server's local volumes.
Our environment is:
CentOS 7.6
Mailman 2.1.29
GlusterFS 5.2 for shared volume.
When I sent 1000 messages to the same list in 500 seconds (2 messages per second), about 20 messages has been shunted because of 'Stale file handle' error.
Aug 06 15:14:45 2019 (15817) Uncaught runner exception: [Errno 116] Stale file handle
Aug 06 15:14:45 2019 (15817) Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
mlist = self._open_
File "/usr/lib/
mlist = MailList.
File "/usr/lib/
self.Load()
File "/usr/lib/
dict, e = self.__load(file)
File "/usr/lib/
dict = loadfunc(fp)
IOError: [Errno 116] Stale file handle
Aug 06 15:14:45 2019 (15817) SHUNTING: 1565072084.
This happens when the recipient list's config.pck file is replaced by another server while reading it.
ESTALE could happen normally on shared volumes, and in most case, simply retrying open/read is sufficient to recover the error.
So I think a retry logic should be implemented in MailList._load() method.
ESTALE errors can also occures when closing a file, in LockFile.py and MailList.py:
Aug 28 12:13:14 2019 (25794) Uncaught runner exception: [Errno 116] Stale file handle mailman/ Mailman/ Queue/Runner. py", line 119, in _oneloop _onefile( msg, msgdata) mailman/ Mailman/ Queue/Runner. py", line 190, in _onefile mlist, msg, msgdata) mailman/ Mailman/ Queue/IncomingR unner.py" , line 115, in _dispose Lock(timeout= mm_cfg. LIST_LOCK_ TIMEOUT) mailman/ Mailman/ MailList. py", line 164, in Lock __lock. lock(timeout) mailman/ Mailman/ LockFile. py", line 288, in lock mailman/ Mailman/ LockFile. py", line 432, in __read
Aug 28 12:13:14 2019 (25794) Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
keepqueued = self._dispose(
File "/usr/lib/
mlist.
File "/usr/lib/
self.
File "/usr/lib/
elif self.__read() == self.__tmpfname:
File "/usr/lib/
fp.close()
IOError: [Errno 116] Stale file handle
Aug 28 12:13:14 2019 (25794) SHUNTING: 1566961993. 343576+ 720c68684e735da fb08c192202bfe1 454009a201
Aug 29 13:05:15 2019 (27163) Uncaught runner exception: [Errno 116] Stale file handle mailman/ Mailman/ Queue/Runner. py", line 119, in _oneloop _onefile( msg, msgdata) mailman/ Mailman/ Queue/Runner. py", line 165, in _onefile list(listname) mailman/ Mailman/ Queue/Runner. py", line 208, in _open_list MailList( listname, lock=False) mailman/ Mailman/ MailList. py", line 133, in __init__ mailman/ Mailman/ MailList. py", line 692, in Load mailman/ Mailman/ MailList. py", line 670, in __load
Aug 29 13:05:15 2019 (27163) Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
mlist = self._open_
File "/usr/lib/
mlist = MailList.
File "/usr/lib/
self.Load()
File "/usr/lib/
dict, e = self.__load(file)
File "/usr/lib/
fp.close()
IOError: [Errno 116] Stale file handle
Aug 29 13:05:15 2019 (27163) SHUNTING: 1567051501. 717527+ 7b52807884fed08 cbdb6f523d1edde f1779ddfdb