getmail's "delete_after" does not delete msgs from Yahoo server

[Bill Yikes]

Getmail is expected to delete messages older than 90 days under this configuration:

---8<--------------------
[options]
verbose = 0
message_log = ~/logs/getmail.log
message_log_verbose = true
delete_after = 90
max_messages_per_session=10

[retriever]
type=SimplePOP3SSLRetriever
server=pop.mail.yahoo.com
port=995
<email address hidden>
password=yadayada

[destination]
type = Mboxrd
path = ~/mail/yahoo.mbox
---8<--------------------

It fetched the first ten msgs from a box containing 10s of thousands of msgs, dating back 7 years ago. The resulting mbox file contained 9 messages dated 2013-2014, and 1 message with no "Date:" field. Then getmail was called again. The same 10 messages were fetched again and were also delivered to ~/mail/yahoo.mbox, so the mbox file contained 10 duplicates. The log file also indicates that the same 10 messages were fetched and the msg id's each appeared twice.

I don't know if getmail supports UIDL and whether it attempts to avoid fetching dupes, but certainly at least 9 of the messages should have been deleted from the server on the 1st session, and thus should not have even been available from the server thereafter.

Some badly acting POP3 servers will ignore delete requests because they want users to see ads on their web UI. So in order to rule out that possibility fetchmail was run with this config:

---8<--------------------
skip yahoo via pop.mail.yahoo.com
  protocol pop3
  port 995
  uidl
  username <email address hidden>
  password yadayada
  fetchlimit 10
  fetchall
  ssl
---8<--------------------

Fetchmail has no "delete_after" capability, but its default behavior is to delete every msg fetched from the server. And it did. Fetchmail grabbed the same 10 msgs that getmail fetched since they were still on the server after 2 getmail retrievals, then fetchmail deleted them from the server unconditionally. Then getmail was run for a 3rd time, and this time it fetched a different 10 messages, thus proving that fetchmail was able to delete the messages.

I hope it doesn't bother anyone that this is reported here despite the bug not being tested on Ubuntu. There is no upstream bug tracker and since it's not a security bug I prefer having the bug tracked and open to public feedback.

[Bill Yikes (yik3s)]

I tried the same test with:

delete_dup_msgids = true
type=BrokenUIDLPOP3SSLRetriever
move_on_delete=~/mail/yahoo_deleted.mbox

and then with IMAP:

type=SimpleIMAPSSLRetriever

and it made no difference. In both cases it still failed to send a DELE command to the server. It fetches the same msgs every time, without deleting them, and also creates copies in the destation mbox. I realize that POP + UIDL, POP w/out UIDL, and IMAP will each result in different IDs for the same message. But I ran the IMAP retrieval twice, thus ensuring that getmail would have a chance to match the ID of msgs already retrieved. Dupes were still created, and the move_on_delete folder remains empty.

[Bill Yikes (yik3s)]

Since it's unclear whether getmail is failing in the decision logic or whether server deletes are generally a problem, a test run was done with SimpleIMAPSSLRetriever and "delete = true". In this case, the error results:

'No mailbox exists with name "~/mail/yahoo_deleted.mbox"'

It's lying. "ls ~/mail/yahoo_deleted.mbox" reveals the file that was made with touch. So it looks like there are 3 bugs at this point:

1. "delete_after = 90" doesn't trigger detection of msgs with a "Date:" ~7 years past.
2. "delete_dup_msgids = true" is not inhibiting storage of msgs with duplicate IMAP-generated IDs.
3. "move_on_delete" fails with error falsely claiming the mailbox does not exist.

[Bill Yikes (yik3s)]

These findings are with Getmail version 5.13 (the Debian Buster distribution).