systemd-resolved has issues when the answer is over 512 bytes with EDNS disabled

[Victor Tapia]

[Impact]

TCP stub is cutting down the payload to 512 bytes when EDNS is disabled. This makes non-EDNS clients (nslookup) receive a "shortened" answer even when UDP returns a truncated reply for a new TCP query. For instance,

- If the client supports EDNS:

$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

- If the client does not support EDNS:

$ dig +noedns +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
29

In the second case, no-EDNS, TCP should provide the complete answer, but it's capped at UDP's size.

[Test Case]

Query systemd-resolved with a domain name that resolves to multiple (lots.. 30+) A records. A client with EDNS support (dig) will receive all of them, a client without support (nslookup or dig +noedns) will have a truncated list. Using the example above:

EDNS: dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
non-EDNS: dig +noedns +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l

[Regression potential]

Minimal. This change only affects TCP requests, and the new size is already used in the code for other requests.

[Other Info]

Upstream bug: https://github.com/systemd/systemd/issues/10816
Fixed upstream with commit: https://github.com/systemd/systemd/commit/e6eed9445956cfa496e1db933bfd3530db23bfce

[Original Description]

Querying a domain name that has >512 bytes in records (e.g. 30+ A records), the number of results depends on the DNS client used:

- If the client supports EDNS:

$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

- If the client does not support EDNS:

$ dig +noedns +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
29

Normally a client that doesn't support EDNS would receive a truncated reply from the initial UDP connection (limited by the spec to 512 bytes) and a second query would be established via TCP to receive the complete results. In this case, the number of results is the same regardless of the protocol used (29).

Upstream bug: https://github.com/systemd/systemd/issues/10816

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "systemd-TCP-size-bionic.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Victor Tapia (vtapia)]

## VERIFICATION DISCO

- Before the fix:

ubuntu@disco:~$ dpkg -l | grep "ii systemd "
ii systemd 239-7ubuntu10 amd64 system and service manager

#EDNS query
ubuntu@disco:~$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query falling back to TCP
ubuntu@disco:~$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
29

#No EDNS query ignoring the truncate flag (stick to UDP only)
ubuntu@disco:~$ dig +noall +answer +noedns +ignore testing.irongiantdesign.com @127.0.0.53 | wc -l
29

- With the fix:

ubuntu@disco:~$ dpkg -l | grep "ii systemd "
ii systemd 239-7ubuntu15 amd64 system and service manager

#EDNS query
ubuntu@disco:~$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query falling back to TCP
ubuntu@disco:~$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query ignoring the truncate flag (stick to UDP only)
ubuntu@disco:~$ dig +noall +answer +noedns +ignore testing.irongiantdesign.com @127.0.0.53 | wc -l
29

[Brian Murray (brian-murray)]

Hello Victor, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Bryan Quigley (bryanquigley)]

Cosmic verifing
# Before the fix
ii systemd 239-7ubuntu10 amd64 system and service manager

#EDNS query
$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query falling back to TCP
$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
29

#No EDNS query ignoring the truncate flag (stick to UDP only)
$ dig +noall +answer +noedns +ignore testing.irongiantdesign.com @127.0.0.53 | wc -l
29

- With the fix:
systemd 239-7ubuntu10.5

#EDNS query
$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query falling back to TCP
$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query ignoring the truncate flag (stick to UDP only)
$ dig +noall +answer +noedns +ignore testing.irongiantdesign.com @127.0.0.53 | wc -l
29

[Dan Streetman (ddstreet)]

@vtapia, did you also test if 'ping' and/or 'telnet' DNS resolution was fixed as well? i.e. currently:

ubuntu@lp1804487:~$ ping testing.irongiantdesign.com
ping: testing.irongiantdesign.com: Temporary failure in name resolution
ubuntu@lp1804487:~$ telnet testing.irongiantdesign.com
telnet: could not resolve testing.irongiantdesign.com/telnet: Temporary failure in name resolution
ubuntu@lp1804487:~$ ssh testing.irongiantdesign.com
ssh: Could not resolve hostname testing.irongiantdesign.com: Temporary failure in name resolution
ubuntu@lp1804487:~$ host testing.irongiantdesign.com
testing.irongiantdesign.com has address 253.0.0.23
testing.irongiantdesign.com has address 253.0.0.6
testing.irongiantdesign.com has address 253.0.0.28
testing.irongiantdesign.com has address 253.0.0.25
testing.irongiantdesign.com has address 253.0.0.22
testing.irongiantdesign.com has address 253.0.0.5
testing.irongiantdesign.com has address 253.0.0.10
testing.irongiantdesign.com has address 253.0.0.4
testing.irongiantdesign.com has address 253.0.0.16
testing.irongiantdesign.com has address 253.0.0.2
testing.irongiantdesign.com has address 253.0.0.7
testing.irongiantdesign.com has address 253.0.0.26
testing.irongiantdesign.com has address 173.236.154.252
testing.irongiantdesign.com has address 253.0.0.21
testing.irongiantdesign.com has address 253.0.0.27
testing.irongiantdesign.com has address 253.0.0.13
testing.irongiantdesign.com has address 253.0.0.3
testing.irongiantdesign.com has address 253.0.0.1
testing.irongiantdesign.com has address 253.0.0.24
testing.irongiantdesign.com has address 253.0.0.12
testing.irongiantdesign.com has address 253.0.0.9
testing.irongiantdesign.com has address 253.0.0.20
testing.irongiantdesign.com has address 253.0.0.8
testing.irongiantdesign.com has address 253.0.0.15
testing.irongiantdesign.com has address 253.0.0.19
testing.irongiantdesign.com has address 253.0.0.17
testing.irongiantdesign.com has address 253.0.0.18
testing.irongiantdesign.com has address 253.0.0.14
testing.irongiantdesign.com has address 253.0.0.11

That doesn't appear fixed with the latest systemd...?

ubuntu@lp1804487:~$ dpkg -l |grep 'ii systemd '
ii systemd 239-7ubuntu10.5 amd64 system and service manager
ubuntu@lp1804487:~$ ping testing.irongiantdesign.com
ping: testing.irongiantdesign.com: Temporary failure in name resolution
ubuntu@lp1804487:~$ telnet testing.irongiantdesign.com
telnet: could not resolve testing.irongiantdesign.com/telnet: Temporary failure in name resolution
ubuntu@lp1804487:~$ ssh testing.irongiantdesign.com
ssh: Could not resolve hostname testing.irongiantdesign.com: Temporary failure in name resolution

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 239-7ubuntu15

---------------
systemd (239-7ubuntu15) disco; urgency=medium

  * core: set /run size to 10%, like initramfs-tools does.
    Currently there is a difference between initrd and initrd-less boots,
    w.r.t. size= mount option of /run. This yields different runtime journald caps
    (1% vs 10%), and on dense deployments of containers may result in OOM kills.
    (LP: #1799251)
    File: debian/patches/debian/UBUNTU-core-set-run-size-to-10-like-initramfs-tools-does.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1fac2568fe716dc1a41bada78293dc6327a6df0d

  * resolved: Increase size of TCP stub replies.
    DNS_PACKET_PAYLOAD_SIZE_MAX is limiting the size of the stub replies to
    512 with EDNS off or 4096 with EDNS on, without checking the protocol
    used. This makes TCP replies for clients without EDNS support to be
    limited to 512, making the truncate flag useless if the query result is
    bigger than 512 bytes.
    This commit increases the size of TCP replies to DNS_PACKET_SIZE_MAX
    Fixes: #10816
    (cherry picked from commit e6eed9445956cfa496e1db933bfd3530db23bfce)
    (LP: #1804487)
    Author: Victor Tapia
    File: debian/patches/resolved-Increase-size-of-TCP-stub-replies.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=702a4566174c4d2bd84b70805107cfc1a7c128cc

 -- Dimitri John Ledkov <email address hidden> Mon, 03 Dec 2018 13:49:24 +0000

[Dan Streetman (ddstreet)]

< @vtapia, did you also test if 'ping' and/or 'telnet' DNS resolution was fixed as well?

to follow up - that is a separate bug. this fix does indeed fix the eDNS bug in systemd. I'll open a new bug.

[Bryan Quigley (bryanquigley)]

Looks like the recent systemd security CVE bumped this.

[Łukasz Zemczak (sil2100)]

Hello Victor, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Victor, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Bryan Quigley (bryanquigley)]

Cosmic Verified
ii systemd 239-7ubuntu10.7 amd64 system and service manager

#EDNS query
$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query falling back to TCP
$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query ignoring the truncate flag (stick to UDP only)
$ dig +noall +answer +noedns +ignore testing.irongiantdesign.com @127.0.0.53 | wc -l
29

also fixes 1811471 but I'll put those results on the other bug.

[Bryan Quigley (bryanquigley)]

Bionic verified too
ii systemd 237-3ubuntu10.12 amd64 system and service manager

$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30
$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
30

[Dan Streetman (ddstreet)]

For autopkgtest regression failure analysis/justification, please see bug 1811471 (since this sru contains patches for this and that bug)

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.12

---------------
systemd (237-3ubuntu10.12) bionic; urgency=medium

  * d/p/resolve-enable-EDNS0-towards-the-127.0.0.53-stub-res.patch
    getaddrinfo() failures when fallback to dns tcp queries, so enable
    edns0 in resolv.conf (LP: #1811471)

  [ Victor Tapia ]
  * d/p/resolved-Increase-size-of-TCP-stub-replies.patch
    dns failures with edns0 disabled and truncated response (LP: #1804487)

 -- Dan Streetman <email address hidden> Tue, 29 Jan 2019 14:26:48 -0500

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 239-7ubuntu10.7

---------------
systemd (239-7ubuntu10.7) cosmic; urgency=medium

  * d/p/resolve-enable-EDNS0-towards-the-127.0.0.53-stub-res.patch
    getaddrinfo() failures when fallback to dns tcp queries, so enable
    edns0 in resolv.conf (LP: #1811471)

  [ Victor Tapia ]
  * d/p/resolved-Increase-size-of-TCP-stub-replies.patch
    dns failures with edns0 disabled and truncated response (LP: #1804487)

 -- Dan Streetman <email address hidden> Tue, 29 Jan 2019 14:19:39 -0500