OVN: no DNS responses are generated for TCP queries

[Dr. Jens Harbott]

When a client sends a query via TCP instead of UDP, OVN does not generate an answer. The query is instead forwarded to the configured external resolvers. Example:

debian@vm1:~$ dig +tcp vm1

; <<>> DiG 9.18.16-1~deb12u1-Debian <<>> +tcp vm1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 24091
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;vm1. IN A

;; AUTHORITY SECTION:
. 2468 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2023080500 1800 900 604800 86400

;; Query time: 0 msec
;; SERVER: 9.9.9.9#53(9.9.9.9) (TCP)
;; WHEN: Sat Aug 05 19:44:16 UTC 2023
;; MSG SIZE rcvd: 107

For comparison check the same query via UDP:

debian@vm1:~$ dig +noedns vm1

; <<>> DiG 9.18.16-1~deb12u1-Debian <<>> +noedns vm1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41575
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;vm1. IN A

;; ANSWER SECTION:
vm1. 3600 IN A 10.128.0.77

;; Query time: 0 msec
;; SERVER: 9.9.9.9#53(9.9.9.9) (UDP)
;; WHEN: Sat Aug 05 19:45:39 UTC 2023
;; MSG SIZE rcvd: 40

(Note that the latter response does not actually come from the external server, but is spoofed by OVN.)

Marking as security bug since this may unexpectedly leak internal information (hostnames) to external third parties. Also see https://bugs.launchpad.net/neutron/+bug/2030294 for a related issue requiring the +noedns option needing to be used in the second example (it doesn't change the outcome for the first example).

[Brian Haley (brian-haley)]

Can you see if the patch I mentioned in the other bug referenced here resolves this?

https://patchwork<email address hidden>/

If so then we can close this and the other. Thanks.

[Dr. Jens Harbott (j-harbott)]

Well not generating an answer and instead forwarding the query to the upstream resolver is what is already happening in this case, I don't see how your patch would change this. It will also turn the situation with UDP+EDNS into the case described here, so if it does what the patch description says, it will resolve the "broken responses" issue at the price of converting it into another instance of the issue described here. (Which is also why I'm only responding here for now. I'm not 100% sure that this is actually severe enough to warrant privacy, but I'd like to hear others' opinion on it before opening.)

To double check, can you be more explicit about what version of OVN I should test this with? I have a simple devstack setup as reproducer, no special configuration needed.

Arguably both issues still are Neutron bugs, since it is a Neutron feature that instances are able to resolve addresses for other instances in the same tenant (or same network or same subnet at least, not sure about the exact scope), so forwarding requests to an upstream nameserver, which won't have the requested information (unless maybe when using external DNS integration), will solve neither of these two issues.

[Brian Haley (brian-haley)]

I know this patch is in v23.06.0, but I also think the latest Ubuntu 22.03.x version has it.

So is your VM setup to make EDNS queries or is this just TCP? Maybe I misread the description. I'm asking since OVN does not support EDNS, so there is nothing neutron can do to solve that.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

If I'm reading correctly, the concern is that guests may make some kinds of DNS queries which are then forwarded by Neutron's resolver to other outside resolvers. If Neutron has previously provided some guarantee of this as a security protection, for example if it documents publicly that it will protect DNS queries and avoid forwarding them, then this might qualify as a vulnerability. Even if so, I don't see that keeping this bug report secret is likely to prevent any particular exploit or abuse of that behavior.

[Dr. Jens Harbott (j-harbott)]

@Jeremy: Neutron in this case hasn't a real resolver. OVN sets up packet mangling rules that under certain conditions (the target of the query matches an instance hostname (A/AAAA) or address (PTR)) then transforms the query packet into an answer packet that is then returned to the sender, faking a response from the targeted resolver. Afaict this isn't explicitly documented, at least not in Neutron docs, so yes, I agree that we can make this public.

@Brian: It is not a configuration of the VM globally, each DNS client can choose its own options. The dig tool needs to be told to use TCP explicitly, it defaults to using UDP. Other clients may choose to do so on their own. Using the EDNS option is orthogonal to this, it can be enabled or disabled via options for dig, the default is enabled. Either way, all of these variants are valid DNS queries so if Neutron using the ML2/OVN backend is not able to handle them correctly, it should at least be listed in the OVN gaps document.

I will try to update my test scenario with 23.06 to verify the effect of your patch there.

[Jeremy Stanley (fungi)]

Thanks, for now I've switched this to a public hardening opportunity and marked our advisory task "won't fix" to indicate that the VMT doesn't expect to issue a security advisory about it. If the likely solutions involve better documentation, adding configuration options or fixing/improving non-OpenStack software like OVN, then this is one of class B1, C2 or D in our taxonomy: https://security.openstack.org/vmt-process.html#report-taxonomy

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/892578

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/892578
Committed: https://opendev.org/openstack/neutron/commit/0e5c91c4996418147cc329f60ce8b0d0eb539f72
Submitter: "Zuul (22348)"
Branch: master

commit 0e5c91c4996418147cc329f60ce8b0d0eb539f72
Author: Dr. Jens Harbott <email address hidden>
Date: Wed Aug 23 20:48:36 2023 +0200

    Add some more known issues to the OVN gap document

    See the related bugs.

    Related-Bug: #2030294
    Related-Bug: #2030295
    Change-Id: If90e4233c599b0ab4363d7eea6b00436bf7ab92c