Ubuntu
fetchmail package

fetchmail can no longer connect to underwood & gives false error msg (TLS issues)

Bug #1934155 reported by Bill Yikes on 2021-06-30

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	fetchmail (Ubuntu)	Expired	Undecided	Unassigned

Bug Description

Version 6.4.16 is unable to fetch mail from the underwood onion site. This is the output when trying to connect:

fetchmail: normal termination, status 2
fetchmail: 6.4.16 querying underwood-onion (protocol IMAP) at Wed 30 Jun 2021 02:10:52 PM UTC: poll started
fetchmail: Trying to connect to 127.0.0.1/12345...connected.
fetchmail: IMAP< * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN
fetchmail: IMAP< A0001 OK Pre-login capabilities listed, post-login capabilities have more.
fetchmail: IMAP> A0002 STARTTLS
fetchmail: IMAP< A0002 BAD TLS support isn't enabled.
fetchmail: 127.0.0.1: upgrade to TLS failed.
fetchmail: Unknown login or authentication error on billyikes@127.0.0.1
fetchmail: socket error while fetching from billyikes@underwood-onion

This worked with past versions. To reproduce, use this stanza in .fetchmailrc:

skip underwood-onion via 127.0.0.1
        protocol imap
        port 12345
        username "billyikes"
        sslproto 'SSL3+'
no sslcertck
        fetchall

Note that past working stanzas did not need "sslproto" or "no sslcertck" but were introduced to after upgrading to 6.4.16.

run these commands:

$ socat TCP4-LISTEN:12345,reuseaddr,fork SOCKS4A:127.0.0.1:underwood2hj3pwd.onion:143,socksport=9050

$ fetchmail -v -d0 underwood-onion

$ pkill socat

It's the same outcome if "sslproto 'SSL23'" is used instead.

This is one report, but there are a few bugs here:

1) inability to connect to handshake with bad TLS protocols. It's an onion site, so SSL is not needed for crypto (it's there for a different purpose). So if fetchmail is judging the crypto to be insecure, it's overzealous in this case.

2) the "Unknown login or authentication error" is not only a false error, it's alarming. It's the worst kind of false error because it tells the user that there's a problem with their account.

3) there is no per-account SOCKS4a config parameter, so users are pushed into this inconvenient and ugly hack of running socat and piping through that. The "plugin" parameter does not help in this case because fetchmail still attempts to resolve the underwood2hj3pwd.onion outside of the proxy.

Bug (3) has always existed, but (1) & (2) are new regressions.

See original description

Bill Yikes (yik3s) on 2021-06-30

description:

updated

Bill Yikes (yik3s) on 2021-06-30

description:

updated

Revision history for this message

Matthias Andree (matthias-andree) wrote on 2021-06-30 (last edit on 2021-06-30):

Bill,

as to your report, I'll break it down along the same bug numbers.
Note I am speaking as the upstream maintainer here and am unaware of Ubuntu's or Debian's packaging decisions.

Bug #1. Fetchmail behaves properly. You requested some sslproto on a non-wrapped port (i. e. no "ssl" option), so fetchmail tries STARTTLS instead, which then fails. I suggest you re-read the manual for the relevant options that you are using and potential references from those options to others, if any. This is deliberately worded quite vaguely.

Bug #2 fetchmail has a 'login or authentication' error it cannot spell out in more detail for you, hence "unknown". Feel free to propose a better wording.

3a. I suggest you read the section "SOCKS" of the manual page. New options are quite high maintenance and I can't test them upstream for lack of socks proxies.

3b. From your earlier report [*] I suppose you might get away with something such as:

poll underwood2hj3pwd.onion via 127.0.0.1
plugin "socat STDIO SOCKS4A:127.0.0.1:%h:%p,socksport=9050"
# other options here

_____
[*] https://bugs.launchpad.net/ubuntu/+source/fetchmail/+bug/1924609

Bill Yikes (yik3s) on 2021-06-30

description:

updated

Revision history for this message

Paride Legovini (paride) wrote on 2021-07-01:

Hello Bill,

I agree with Matthias' reply: fetchmail seems to be working just as expected. Also please note that we use this bug tracker to work on bugs in Ubuntu, so feature requests like a per-account SOCKS4a config parameter don't really fit here; at a first look the right venue would be [1].

I'm marking this bug report as Incomplete for now. If you still believe you found a bug in how fetchmail handles ssl/tls/starttls connections please provide more details about your findings and change the bug status back to New, we'll look to it again. Otherwise if you agree we don't really have a bug here please change the report status to Invalid. Thanks!

[1] https://gitlab.com/fetchmail/fetchmail/

Changed in fetchmail (Ubuntu):
status:	New → Incomplete

Revision history for this message

Bill Yikes (yik3s) wrote on 2021-07-01 (last edit on 2021-07-01):

Download full text (6.8 KiB)

I appreciate the quick response, but I have to say both of you overlooked some of the data I presented, so I will elaborate. I was able to workaround the bug, but there are still bugs here. Having read the man page more times, I continue to find more anomalies than answers, which I will elaborate here.

To be clear, fetchmail changed, not my config. This config works on fetchmail 6.3.26 but fails on 6.4.16:

```
skip underwood-onion via 127.0.0.1
        protocol imap
        port 12345
        username "billyikes"
        fetchall
```
Nothing in your response suggests that I should expect this config to be treated differently from the two different versions of fetchmail. But it is. Verbose output shows that version 6.4.16 sends a "STARTTLS" even though the config does not call for SSL/TLS in any way. That's a bug. The man page says "sslproto auto" is a default which superficially seems reasonable until you notice that "auto" actually imposes TLS. This is a distortion of what users expect by the meaning of "automatic".

My first attempt at a fix for underwood was to add these lines (which fixed some of my stanzas for other servers):
~~~
sslproto 'SSL3+'
no sslcertck
~~~
Intution suggests this means: "If the server demands SSL, then be permissive and allow SSL3 or newer, but don't bother checking whether the cert is good or not, just use it if the server insists". But what we find is that the behavior is not intuitive w.r.t the options. And because it fails the rule of least astonishment, I'm calling it a bug. Moreover, the new manpage states:

"This option has a dual use, out of historic fetchmail behaviour. It controls both the SSL/TLS protocol version and, if --ssl is not specified, the STARTTLS behaviour (upgrading the protocol to an SSL or TLS connection in-band). Some other options may however make TLS mandatory."

To say sslproto "controls STARTTLS behaviour" without fully specifying how that behavior is controlled is a recipe for confusion. An empty argument is said to disable STARTTLS, but that was always the case. So there are two problems here. The first problem is that STARTTLS is technical under-the-hood jargon that end users probably should not be responsible for understanding. Users are basically told whether SSL/TLS is used, not what low-level commands are being passed to make that happen. The second problem is that auto states "require TLS", but does not mention the low-level "STARTTLS" that the section mentioned earlier. It's not clear if fetchmail is treating "TLS" and "STARTTLS" as one in the same, but users wouldn't generally know the underlying details expressed in RFC specs. We do see some ESPs making a distinction (e.g. https://danwin1210.me/mail/) IOW, STARTTLS is mentioned just enough to confuse, and not enough to be understandable.

Apart from the misleading & non-intuitive wording, some things are plainly false. E.g. this statement from the man page is simply not true:

"Only if this option [sslproto] and --ssl are both missing for a poll, there will be opportunistic TLS for POP3 and IMAP, where fetchmail will attempt to upgrade to TLSv1 or newer."

If that were a true statement, then...

I appreciate the quick response, but I have to say both of you overlooked some of the data I presented, so I will elaborate.  I was able to workaround the bug, but there are still bugs here.  Having read the man page more times, I continue to find more anomalies than answers, which I will elaborate here.

To be clear, fetchmail changed, not my config.  This config works on fetchmail 6.3.26 but fails on 6.4.16:

```
skip underwood-onion via 127.0.0.1
        protocol imap
        port 12345
        username "billyikes"
        fetchall
```
Nothing in your response suggests that I should expect this config to be treated differently from the two different versions of fetchmail.  But it is.  Verbose output shows that version 6.4.16 sends a "STARTTLS" even though the config does not call for SSL/TLS in any way.  That's a bug.  The man page says "sslproto auto" is a default which superficially seems reasonable until you notice that "auto" actually imposes TLS.  This is a distortion of what users expect by the meaning of "automatic".

My first attempt at a fix for underwood was to add these lines (which fixed some of my stanzas for other servers):
~~~
sslproto 'SSL3+'
no sslcertck
~~~
Intution suggests this means: "If the server demands SSL, then be permissive and allow SSL3 or newer, but don't bother checking whether the cert is good or not, just use it if the server insists".  But what we find is that the behavior is not intuitive w.r.t the options. And because it fails the rule of least astonishment, I'm calling it a bug.  Moreover, the new manpage states:

"This option has a dual use, out of historic fetchmail behaviour. It controls both the SSL/TLS  protocol  version  and,  if --ssl  is  not  specified, the STARTTLS behaviour (upgrading the protocol to an SSL or TLS connection in-band). Some other options may however make TLS mandatory."

To say sslproto "controls STARTTLS behaviour" without fully specifying how that behavior is controlled is a recipe for confusion.  An empty argument is said to disable STARTTLS, but that was always the case.  So there are two problems here.  The first problem is that STARTTLS is technical under-the-hood jargon that end users probably should not be responsible for understanding.  Users are basically told whether SSL/TLS is used, not what low-level commands are being passed to make that happen.  The second problem is that auto states "require TLS", but does not mention the low-level "STARTTLS" that the section mentioned earlier.  It's not clear if fetchmail is treating "TLS" and "STARTTLS" as one in the same, but users wouldn't generally know the underlying details expressed in RFC specs.  We do see some ESPs making a distinction (e.g. https://danwin1210.me/mail/)  IOW, STARTTLS is mentioned just enough to confuse, and not enough to be understandable.

Apart from the misleading & non-intuitive wording, some things are plainly false.  E.g. this statement from the man page is simply not true:

"Only if this option [sslproto] and --ssl are both missing for a poll, there will be opportunistic TLS for POP3  and  IMAP,  where  fetchmail will attempt to upgrade to TLSv1 or newer."

If that were a true statement, then the first stanza at the top of this post would work.  This is a contradiction in the man page.  That is, "sslproto auto" ironically states that TLS is mandated while simultaneously saying "auto" is also a default, and yet this behavior is not opportunistic.  Opportunistic means: accommodate (or even proactively request) higher levels of security, but ultimately settle for no security at all if availability demands it.

The "sslproto ''" empty string option is bizarre because it strictly disables STARTTLS without stating whether SSL is also disabled, apart from saying that it's incompatible with --ssl.  To expressly /force/ no encryption can be useful but only in limited & obscure test situations.  It's still good that it exists, but what about the much more common case of users who need opportunistic connections?  The empty string denies the opportunity for crypto while all the other sslproto options deny the possibility of no encryption.

Even more strange is "SSL23".  How often does someone want to insist on SSL2 or SSL3, but not the higher options?  There should be a "sslproto SSL2+".

And most importantly there should be a "sslproto none+" to truly deliver opportunistic sessions.

Regarding this suggestion:
```
poll underwood2hj3pwd.onion via 127.0.0.1
        plugin "socat STDIO SOCKS4A:127.0.0.1:%h:%p,socksport=9050"
```
which I expanded to:
```
skip underwood2hj3pwd.onion via 127.0.0.1
        plugin "socat STDIO SOCKS4A:127.0.0.1:%h:%p,socksport=9050"
        protocol       imap
        port           143
        username       "billyikes"
        fetchall
```
result was:
```
fetchmail: running socat STDIO SOCKS4A:127.0.0.1:%h:%p,socksport=9050 (host 127.0.0.1 service 143)
2021/07/01 16:25:44 socat[2202] E socks: connect request rejected or failed
fetchmail: socket error while fetching from billyikes@underwood2hj3pwd.onion
fetchmail: 6.4.16 querying underwood2hj3pwd.onion (protocol IMAP) at Thu 01 Jul 2021 04:25:44 PM UTC: poll completed
fetchmail: Query status=2 (SOCKET)
```
In the end, there are three configs that work.

(config 1)
```
skip underwood-onion via 127.0.0.1
        protocol       imap
        port           12345
        username       "billyikes"
        sslproto       ''
        no sslcertck
        fetchall
```
(config 2)
```
skip underwood-onion via 127.0.0.1
        protocol       imap
        port           12345
        username       "billyikes"
        no sslcertck
        fetchall
```
(config 3)
```
skip underwood-onion via 127.0.0.1
        protocol       imap
        port           12345
        username       "billyikes"
        sslproto       ''
        fetchall
```

It's nice that at least there is a possibility to still function.  Notice that on config 2, even though there is no explicit request for SSL, it's necessary to demand that the ssl cert is not checked.  It's a bit off that we should have to ask not to check certs despite SSL not being explicitly requested.  More importantly, Config 2 is not working as documented, because "auto" is the default and auto *requires* TLS.  So it's a bug that config 2 works to fetch msgs from a non-TLS server.

Config 3 is the most sensible and it is not a bug, strictly speaking.  However, what if underwood were to one day begin is permit TLS?  I might like to opportunistically benefit from that, but my config forcibly denies crypto.  This means I would have to periodically try "SSL23" and "SSL3+", because opportunistic SSL/TLS is no longer supported.  And if underwood starts requiring crypto, then later reverts back to no crypto, I have the maintenence burden of having to edit .fetchmailrc every time.  When in fact I'm happy to rely on the builtin crypto of onion circuits - I just need fetchmail to work at the best protocol the server supports.

Revision history for this message

Bill Yikes (yik3s) wrote on 2021-07-02:

I've also noticed that "auto" and "TLS1+" have the same meaning when passed to --SSLPROTO. That's part of the problem. So now there is duplication, and no way to get opportunistic crypto. If auto was previously opportunistic, then the change to remove it should be rolled back.

Bill Yikes (yik3s) on 2021-07-02

Changed in fetchmail (Ubuntu):
status:	Incomplete → New

Revision history for this message

Matthias Andree (matthias-andree) wrote on 2021-07-02:

Bill,

I can sense quite a deal of frustration on your end, and am sorry for that. I should have mentioned to also look into the NEWS file, which at least would have told you that sslcertck is now the default. Fetchmail's -vv verbose mode might also try to log more details on the decisions along the way...

Your feedback as to how you find documentation confusing or incomplete or incompletely referenced is helpful, thank you for that - although it will take a while before improvements will make it into Ubuntu, this is due to how software ends up for Ubuntu.

On the other hand, I must say that nobody promised that fetchmail 6.4 would be bug-for-bug-compatible with 6.3, and fetchmail has indeed changed in the years between the 6.3.26 and 6.4.1X releases. In particular, I was making fetchmail 6.4 protect users' credentials more thoroughly, and this is why fetchmail finally defaults to certificate validation, and that makes "opportunistic" encryption quite a bit more mandatory, and the failing STARTTLS after the server hadn't offered it was/is not helpful: In order to validate certificates, fetchmail will try to obtain one - and on a non-SSL connection this means starttls, and fetchmail tries it blind.

I see that it is less transparent to users than I would have hoped, so thanks for mirroring that - the certificate validation makes TLS (or STARTTLS in-band negotiation for that matter) less opportunistic and more mandatory than parts of the documentation suggest.

So, "no sslcertck" is currently required to make "opportunistic" TLS really opportunistic (your "config 2" - and not in an obscure way mandatory), and "sslproto ''" indeed defeats TLS negotiation at all (your config 3).

I need to think a bit more about which changes could go into 6.5, and which will have to wait for 7.0, where the version is more of a flag to users that reads "beware of incompatibilities".

Revision history for this message

Bill Yikes (yik3s) wrote on 2021-07-03:

I see that I overlooked the NEWS file. That's more detailed than I'm used to seeing. As I was just now skimming through it, it was clear that moves made to protect users from weak algorithms assumed they're on an untunneled connection, which is not always the case. Sometimes the SSL is just used for verification and the crypto is just redundant.

I should also mention that I struggled with the "no sslcertck" syntax. All the options I've been using to that point were "<key> <value>" format, and "no sslcertck" is an exceptional transpose of it. I first tried "sslcertck no" because I was sure the key-value wouldn't flip. But in fact the "keyword" really included a space. Coupled with the inconsistency of some keywords starting with "set", I felt I couldn't trust the man page. Adding to the confusion, some options take arguments and some do not, but the Keyword/Option Summary table doesn't show any BNF and it omits the <value> token, making so we have to work out from the wording of the "Function" column whether it's unary or takes an argument.

Anyway, I appreciate your help and I hope my feedback helps for future revisions.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-07-05:

Thanks for the report Bill and even more for the awesome and detailed explanations Matthias!
AFAICS there is nothing in here for Ubuntu to fix "right now". As versions 6.5/7.0 are releases we will pick them up but that isn't what this bug was about.

Gladly config based solutions were found to handle the problem, but I'll mark the bug as incomplete to auto-expire if nothing else needs to be added in your opinion.

Changed in fetchmail (Ubuntu):
status:	New → Incomplete

Revision history for this message

Launchpad Janitor (janitor) wrote on 2021-09-04:

[Expired for fetchmail (Ubuntu) because there has been no activity for 60 days.]

Changed in fetchmail (Ubuntu):
status:	Incomplete → Expired

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntufetchmail package

fetchmail can no longer connect to underwood & gives false error msg (TLS issues)

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
fetchmail package