systemd-resolved has issues when the answer is over 512 bytes with EDNS disabled

Bug #1804487 reported by Victor Tapia on 2018-11-21
22
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd
Fix Released
Unknown
systemd (Debian)
Fix Released
Unknown
systemd (Ubuntu)
Status tracked in Disco
Xenial
Undecided
Unassigned
Bionic
High
Victor Tapia
Cosmic
High
Victor Tapia
Disco
High
Victor Tapia

Bug Description

[Impact]

TCP stub is cutting down the payload to 512 bytes when EDNS is disabled. This makes non-EDNS clients (nslookup) receive a "shortened" answer even when UDP returns a truncated reply for a new TCP query. For instance,

- If the client supports EDNS:

$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

- If the client does not support EDNS:

$ dig +noedns +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
29

In the second case, no-EDNS, TCP should provide the complete answer, but it's capped at UDP's size.

[Test Case]

Query systemd-resolved with a domain name that resolves to multiple (lots.. 30+) A records. A client with EDNS support (dig) will receive all of them, a client without support (nslookup or dig +noedns) will have a truncated list. Using the example above:

EDNS: dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
non-EDNS: dig +noedns +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l

[Regression potential]

Minimal. This change only affects TCP requests, and the new size is already used in the code for other requests.

[Other Info]

Upstream bug: https://github.com/systemd/systemd/issues/10816
Fixed upstream with commit: https://github.com/systemd/systemd/commit/e6eed9445956cfa496e1db933bfd3530db23bfce

[Original Description]

Querying a domain name that has >512 bytes in records (e.g. 30+ A records), the number of results depends on the DNS client used:

- If the client supports EDNS:

$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

- If the client does not support EDNS:

$ dig +noedns +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
29

Normally a client that doesn't support EDNS would receive a truncated reply from the initial UDP connection (limited by the spec to 512 bytes) and a second query would be established via TCP to receive the complete results. In this case, the number of results is the same regardless of the protocol used (29).

Upstream bug: https://github.com/systemd/systemd/issues/10816

Related branches

Victor Tapia (vtapia) on 2018-11-29
description: updated
description: updated
Victor Tapia (vtapia) wrote :
Victor Tapia (vtapia) wrote :
Victor Tapia (vtapia) wrote :
Dan Streetman (ddstreet) on 2018-11-29
Changed in systemd (Ubuntu Xenial):
status: New → Invalid
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Victor Tapia (vtapia)
Changed in systemd (Ubuntu Cosmic):
assignee: nobody → Victor Tapia (vtapia)
Changed in systemd (Ubuntu Disco):
assignee: nobody → Victor Tapia (vtapia)
status: New → In Progress
Changed in systemd (Ubuntu Cosmic):
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
Changed in systemd (Ubuntu Disco):
importance: Undecided → High
Changed in systemd (Ubuntu Cosmic):
importance: Undecided → High
Changed in systemd (Ubuntu Bionic):
importance: Undecided → High

The attachment "systemd-TCP-size-bionic.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Changed in systemd:
status: Unknown → Fix Released
Changed in systemd (Debian):
status: Unknown → Fix Committed
Changed in systemd (Debian):
status: Fix Committed → Fix Released
Changed in systemd (Ubuntu Disco):
status: In Progress → Fix Committed
Victor Tapia (vtapia) wrote :

## VERIFICATION DISCO

- Before the fix:

ubuntu@disco:~$ dpkg -l | grep "ii systemd "
ii systemd 239-7ubuntu10 amd64 system and service manager

#EDNS query
ubuntu@disco:~$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query falling back to TCP
ubuntu@disco:~$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
29

#No EDNS query ignoring the truncate flag (stick to UDP only)
ubuntu@disco:~$ dig +noall +answer +noedns +ignore testing.irongiantdesign.com @127.0.0.53 | wc -l
29

- With the fix:

ubuntu@disco:~$ dpkg -l | grep "ii systemd "
ii systemd 239-7ubuntu15 amd64 system and service manager

#EDNS query
ubuntu@disco:~$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query falling back to TCP
ubuntu@disco:~$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query ignoring the truncate flag (stick to UDP only)
ubuntu@disco:~$ dig +noall +answer +noedns +ignore testing.irongiantdesign.com @127.0.0.53 | wc -l
29

Hello Victor, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Bryan Quigley (bryanquigley) wrote :

Cosmic verifing
# Before the fix
ii systemd 239-7ubuntu10 amd64 system and service manager

#EDNS query
$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query falling back to TCP
$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
29

#No EDNS query ignoring the truncate flag (stick to UDP only)
$ dig +noall +answer +noedns +ignore testing.irongiantdesign.com @127.0.0.53 | wc -l
29

- With the fix:
systemd 239-7ubuntu10.5

#EDNS query
$ dig +noall +answer testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query falling back to TCP
$ dig +noall +answer +noedns testing.irongiantdesign.com @127.0.0.53 | wc -l
30

#No EDNS query ignoring the truncate flag (stick to UDP only)
$ dig +noall +answer +noedns +ignore testing.irongiantdesign.com @127.0.0.53 | wc -l
29

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
Dan Streetman (ddstreet) wrote :

@vtapia, did you also test if 'ping' and/or 'telnet' DNS resolution was fixed as well? i.e. currently:

ubuntu@lp1804487:~$ ping testing.irongiantdesign.com
ping: testing.irongiantdesign.com: Temporary failure in name resolution
ubuntu@lp1804487:~$ telnet testing.irongiantdesign.com
telnet: could not resolve testing.irongiantdesign.com/telnet: Temporary failure in name resolution
ubuntu@lp1804487:~$ ssh testing.irongiantdesign.com
ssh: Could not resolve hostname testing.irongiantdesign.com: Temporary failure in name resolution
ubuntu@lp1804487:~$ host testing.irongiantdesign.com
testing.irongiantdesign.com has address 253.0.0.23
testing.irongiantdesign.com has address 253.0.0.6
testing.irongiantdesign.com has address 253.0.0.28
testing.irongiantdesign.com has address 253.0.0.25
testing.irongiantdesign.com has address 253.0.0.22
testing.irongiantdesign.com has address 253.0.0.5
testing.irongiantdesign.com has address 253.0.0.10
testing.irongiantdesign.com has address 253.0.0.4
testing.irongiantdesign.com has address 253.0.0.16
testing.irongiantdesign.com has address 253.0.0.2
testing.irongiantdesign.com has address 253.0.0.7
testing.irongiantdesign.com has address 253.0.0.26
testing.irongiantdesign.com has address 173.236.154.252
testing.irongiantdesign.com has address 253.0.0.21
testing.irongiantdesign.com has address 253.0.0.27
testing.irongiantdesign.com has address 253.0.0.13
testing.irongiantdesign.com has address 253.0.0.3
testing.irongiantdesign.com has address 253.0.0.1
testing.irongiantdesign.com has address 253.0.0.24
testing.irongiantdesign.com has address 253.0.0.12
testing.irongiantdesign.com has address 253.0.0.9
testing.irongiantdesign.com has address 253.0.0.20
testing.irongiantdesign.com has address 253.0.0.8
testing.irongiantdesign.com has address 253.0.0.15
testing.irongiantdesign.com has address 253.0.0.19
testing.irongiantdesign.com has address 253.0.0.17
testing.irongiantdesign.com has address 253.0.0.18
testing.irongiantdesign.com has address 253.0.0.14
testing.irongiantdesign.com has address 253.0.0.11

That doesn't appear fixed with the latest systemd...?

ubuntu@lp1804487:~$ dpkg -l |grep 'ii systemd '
ii systemd 239-7ubuntu10.5 amd64 system and service manager
ubuntu@lp1804487:~$ ping testing.irongiantdesign.com
ping: testing.irongiantdesign.com: Temporary failure in name resolution
ubuntu@lp1804487:~$ telnet testing.irongiantdesign.com
telnet: could not resolve testing.irongiantdesign.com/telnet: Temporary failure in name resolution
ubuntu@lp1804487:~$ ssh testing.irongiantdesign.com
ssh: Could not resolve hostname testing.irongiantdesign.com: Temporary failure in name resolution

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 239-7ubuntu15

---------------
systemd (239-7ubuntu15) disco; urgency=medium

  * core: set /run size to 10%, like initramfs-tools does.
    Currently there is a difference between initrd and initrd-less boots,
    w.r.t. size= mount option of /run. This yields different runtime journald caps
    (1% vs 10%), and on dense deployments of containers may result in OOM kills.
    (LP: #1799251)
    File: debian/patches/debian/UBUNTU-core-set-run-size-to-10-like-initramfs-tools-does.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1fac2568fe716dc1a41bada78293dc6327a6df0d

  * resolved: Increase size of TCP stub replies.
    DNS_PACKET_PAYLOAD_SIZE_MAX is limiting the size of the stub replies to
    512 with EDNS off or 4096 with EDNS on, without checking the protocol
    used. This makes TCP replies for clients without EDNS support to be
    limited to 512, making the truncate flag useless if the query result is
    bigger than 512 bytes.
    This commit increases the size of TCP replies to DNS_PACKET_SIZE_MAX
    Fixes: #10816
    (cherry picked from commit e6eed9445956cfa496e1db933bfd3530db23bfce)
    (LP: #1804487)
    Author: Victor Tapia
    File: debian/patches/resolved-Increase-size-of-TCP-stub-replies.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=702a4566174c4d2bd84b70805107cfc1a7c128cc

 -- Dimitri John Ledkov <email address hidden> Mon, 03 Dec 2018 13:49:24 +0000

Changed in systemd (Ubuntu Disco):
status: Fix Committed → Fix Released
Dan Streetman (ddstreet) wrote :

< @vtapia, did you also test if 'ping' and/or 'telnet' DNS resolution was fixed as well?

to follow up - that is a separate bug. this fix does indeed fix the eDNS bug in systemd. I'll open a new bug.

tags: added: verification-done
removed: verification-needed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.