Verify DNS fingerprints not working

[Andreas Tauscher]

When setting in /etc/ssh/ssh_config VerifyHostKeyDNS to yes the fingerprints are fetched, but the result is always:
debug1: found n insecure fingerprints in DNS
With dig +dnssec -tsshfp hostname the result is ok: ad flg is set.

[Andreas Tauscher (ta-lonestar-bbs)]

ssh version is OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020

[Seth Arnold (seth-arnold)]

Hello, dig will do dns lookups itself, it doesn't rely on the host resolver configuration. Does your host resolver configuration support dnssec? It might be worth using tcpdump or tshark or wireshark to see if the queries are properly formed, and if the replies are correct.

Thanks

[Andreas Tauscher (ta-lonestar-bbs)]

With @localhost as parameter it will use the local resolver.
Local resolver is unbound. The cr** systemd resolver is disabled.
Configuration is exactly same like on another machine where it is working like expected.
Only difference: Ubuntu 18.04 instead of 20.04.
On 18.04
debug1: found 3 secure fingerprints in DNS
With 20.04
debug1: found 3 insecure fingerprints in DNS
Also when I change the resolver this does not change. So the resolver seems not to be the problem.

[Andreas Tauscher (ta-lonestar-bbs)]

The DNS queries captured with wireshark ssh to unbound and unbound to world looking correct and allways the AD flag in the responses is set.

[Dan Streetman (ddstreet)]

this is likely a dup of bug 1897744

[Christian Ehrhardt  (paelzer)]

Turns out this seems to be a never ending story and you might have found a comeback of that issue for your particular configuration as you say this worked on 18.04 but fails on 20.04.

This goes way back
https://bugzilla.mindrot.org/show_bug.cgi?id=1455
Or half way back
https://trac.macports.org/ticket/49007
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=618863
https://bugzilla.mindrot.org/show_bug.cgi?id=2119

Other more recent similar issues were around "options edns0" being required to be set for this to work now:
https://github.com/NixOS/nixpkgs/issues/12470
https://exanames.typepad.com/blog/2009/06/one-more-thing-to-do-with-dnssec-ssh.html
https://bugzilla.redhat.com/show_bug.cgi?id=1630180
https://bugzilla.redhat.com/show_bug.cgi?id=1878166
Note: that option was the default for /etc/resolv.conf on Bionic/Focal for me.

Various working setups seem to have been affected by 7.5
https://lists.mindrot.org/pipermail/openssh-bugs/2017-April/017631.html
https://lists.mindrot.org/pipermail/openssh-unix-dev/2018-January/036600.html
https://bugzilla.mindrot.org/show_bug.cgi?id=2708

But Bionic -> Focal is openssh version 7.6 -> 8.3

Multiple of the above and some other references refer to requiring ldns support.
That clearly is in openssh since ~v6 but we don't enable it at build time
                   libldns support: no
Is that required and is it now more required than before - I don't know :-/

Sorry, all that I could provide so far was a collection of a (disturbing) history of that feature.

[Christian Ehrhardt  (paelzer)]

Something helpful for anyone looking into this later I found what seems a good testcase without requiring too much a local setup (of a dnssec dns server):

To get unbound (brute force) do:
apt install unbound
sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved
sudo systemctl enable unbound-resolvconf
sudo systemctl enable unbound
# set 127.0.0.1
vim /etc/resolv.conf

Now this should show the ad flag as reported:
$ dig salsa.debian.org -t sshfp
...
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

$ ssh -v -o "VerifyHostKeyDNS=yes" <email address hidden>

This indeed (as reported), does show the changed behavior (clean LXD containers, just changes as mentioned above, edns0 set by default):

Bionic:
debug1: found 4 secure fingerprints in DNS
debug1: matching host key fingerprint found in DNS

Focal:
debug1: found 4 insecure fingerprints in DNS
debug1: matching host key fingerprint found in DNS
The authenticity of host 'salsa.debian.org (209.87.16.44)' can't be established.
ED25519 key fingerprint is SHA256:OAD3pGSwcODIthxF+zIRvPTZ8UCJAYI75E42XDfGr84.
Matching host key fingerprint found in DNS.

[Christian Ehrhardt  (paelzer)]

As a first step I tried to make sure this actually is a change in openssh.
Because my reading of the issues referred above has shown that not all of the verification is done inside ssh but partially in glibc.

So I upgraded on the bionic test system step by step.

The upgrade dependency list for openssh-client is "libc-bin libc6 libgssapi-krb5-2 libkrb5-3 libkrb5support0 locales"

I found that the bug is reproducible with openssh-client 1:7.6p1-4ubuntu0.3 running against the newer glibc.

I can switch in/out the reported bug be switching on a Bionic system between glibc
2.27-3ubuntu1.2 - Bionic - Good
2.31-0ubuntu9.1 - Focal - Bad

[Christian Ehrhardt  (paelzer)]

ok up/downgrading just "libc6" is enough to trigger.

I also found that libc6 from Eoan version 2.30-0ubuntu2.2 is good.
So it is new in 2.31!

The changelog mentions soem DNSSEC
https://sourceware.org/legacy-ml/libc-announce/2020/msg00001.html

"* The DNS stub resolver will optionally send the AD (authenticated
  data) bit in queries if the trust-ad option is set via the options
  directive in /etc/resolv.conf (or if RES_TRUSTAD is set in
  _res.options). In this mode, the AD bit, as provided by the name
  server, is available to applications which call res_search and
  related functions. In the default mode, the AD bit is not set in
  queries, and it is automatically cleared in responses, indicating a
  lack of DNSSEC validation. (Therefore, the name servers and the
  network path to them are treated as untrusted.)"

Once I knew that it was a small step and I found that
  options edns0 trust-ad
in /etc/resolv.conf indeed fixes the issue.

I'm not sure if openssh would be entitled to set RES_TRUSTAD is set in _res.options.
Maybe not as that is more a decision of the admin setting up and configuring the system than the openssh software.

Therefore I think this is actually a little detail that upgraders that use dnssec for openssh (and maybe others) via libc6 resolv need to consider.

[Christian Ehrhardt  (paelzer)]

TL;DR:
one affected by this upgrade triggered behavior change needs to set
  options edns0 trust-ad
in /etc/resolv.conf to fix the issue.

And as usual, once you already know what things are about - then (but only then :-/ ) you find the important related issues - reported by Laney \o/
Subscribing him to the bug FYI

Debian bug
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960023
I'll add a bug link to Debian.
Discussed and declared to be systed issue and asked to file upstream.

Which led to ...

Systemd
https://github.com/systemd/systemd/issues/15767
This is fixed in newer systemd and will set trust-ad
I'll add a systemd task to consider backporting this to Focal

[Christian Ehrhardt  (paelzer)]

Summary:
- we understand what happened
- we have a workaround for users by changing a config
- we have a systemd change that can be considered to be backported by the systemd package maintainers

[Christian Ehrhardt  (paelzer)]

@Dan - now I saw your update - that might have shortened my dnssec trip :-)
It indeed is a duplicate of that - marking as such.