| 2019-01-11 21:52:09 |
Dan Streetman |
bug |
|
|
added bug |
| 2019-01-11 21:52:19 |
Dan Streetman |
nominated for series |
|
Ubuntu Disco |
|
| 2019-01-11 21:52:19 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Disco) |
|
| 2019-01-11 21:52:19 |
Dan Streetman |
nominated for series |
|
Ubuntu Xenial |
|
| 2019-01-11 21:52:19 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Xenial) |
|
| 2019-01-11 21:52:19 |
Dan Streetman |
nominated for series |
|
Ubuntu Trusty |
|
| 2019-01-11 21:52:19 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Trusty) |
|
| 2019-01-11 21:52:19 |
Dan Streetman |
nominated for series |
|
Ubuntu Cosmic |
|
| 2019-01-11 21:52:19 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Cosmic) |
|
| 2019-01-11 21:52:19 |
Dan Streetman |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-01-11 21:52:19 |
Dan Streetman |
bug task added |
|
systemd (Ubuntu Bionic) |
|
| 2019-01-11 21:52:25 |
Dan Streetman |
systemd (Ubuntu Trusty): status |
New |
In Progress |
|
| 2019-01-11 21:52:26 |
Dan Streetman |
systemd (Ubuntu Xenial): status |
New |
In Progress |
|
| 2019-01-11 21:52:28 |
Dan Streetman |
systemd (Ubuntu Bionic): status |
New |
In Progress |
|
| 2019-01-11 21:52:30 |
Dan Streetman |
systemd (Ubuntu Cosmic): status |
New |
In Progress |
|
| 2019-01-11 21:52:31 |
Dan Streetman |
systemd (Ubuntu Disco): status |
New |
In Progress |
|
| 2019-01-11 21:52:34 |
Dan Streetman |
systemd (Ubuntu Trusty): assignee |
|
Dan Streetman (ddstreet) |
|
| 2019-01-11 21:52:35 |
Dan Streetman |
systemd (Ubuntu Xenial): assignee |
|
Dan Streetman (ddstreet) |
|
| 2019-01-11 21:52:37 |
Dan Streetman |
systemd (Ubuntu Bionic): assignee |
|
Dan Streetman (ddstreet) |
|
| 2019-01-11 21:52:38 |
Dan Streetman |
systemd (Ubuntu Cosmic): assignee |
|
Dan Streetman (ddstreet) |
|
| 2019-01-11 21:52:39 |
Dan Streetman |
systemd (Ubuntu Disco): assignee |
|
Dan Streetman (ddstreet) |
|
| 2019-01-11 21:52:44 |
Dan Streetman |
systemd (Ubuntu Disco): importance |
Undecided |
High |
|
| 2019-01-11 21:52:46 |
Dan Streetman |
systemd (Ubuntu Cosmic): importance |
Undecided |
High |
|
| 2019-01-11 21:52:47 |
Dan Streetman |
systemd (Ubuntu Bionic): importance |
Undecided |
High |
|
| 2019-01-11 21:52:49 |
Dan Streetman |
systemd (Ubuntu Xenial): importance |
Undecided |
High |
|
| 2019-01-11 21:52:50 |
Dan Streetman |
systemd (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2019-01-11 21:58:58 |
Dan Streetman |
bug watch added |
|
https://github.com/systemd/systemd/issues/11332 |
|
| 2019-01-11 21:58:58 |
Dan Streetman |
bug task added |
|
systemd |
|
| 2019-01-11 22:19:03 |
Bug Watch Updater |
systemd: status |
Unknown |
New |
|
| 2019-01-11 23:47:12 |
Donald King |
bug |
|
|
added subscriber Donald King |
| 2019-01-15 12:40:48 |
Kai Kasurinen |
bug |
|
|
added subscriber Kai Kasurinen |
| 2019-01-18 16:47:07 |
Brian Murray |
bug |
|
|
added subscriber Brian Murray |
| 2019-01-23 22:11:32 |
Bryan Quigley |
bug |
|
|
added subscriber Bryan Quigley |
| 2019-01-29 15:11:18 |
Dan Streetman |
attachment added |
|
lp1811471-disco.debdiff https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811471/+attachment/5233683/+files/lp1811471-disco.debdiff |
|
| 2019-01-29 15:13:51 |
Dan Streetman |
attachment removed |
lp1811471-disco.debdiff https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811471/+attachment/5233683/+files/lp1811471-disco.debdiff |
|
|
| 2019-01-29 15:23:49 |
Dan Streetman |
attachment added |
|
lp1811471-disco.debdiff https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811471/+attachment/5233684/+files/lp1811471-disco.debdiff |
|
| 2019-01-29 15:30:07 |
Dan Streetman |
attachment removed |
lp1811471-disco.debdiff https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811471/+attachment/5233684/+files/lp1811471-disco.debdiff |
|
|
| 2019-01-29 19:19:25 |
Dan Streetman |
description |
[Impact]
The systemd local 'stub' resolver handles all local DNS queries (by default configuration used in Ubuntu), and essentially proxies all requests to its configured upstream DNS resolvers.
Most local DNS resolution by applications uses glibc's getaddrinfo() function. This function is configured in various ways by the /etc/resolv.conf file, which tells glibc what nameserver/resolver to contact as well as how to talk to the name server.
By default, glibc performs UDP DNS queries, with a single DNS query per UDP packet. The UDP packet size is limited per DNS spec to 512 bytes. For some DNS lookups, a 512 byte UDP packet is not large enough to contain the entire response - for example, an A record lookup with a large number (e.g. 30) of A record addresses. This number of A record entries is possible in some cases of load balancing. When the DNS UDP response size is larger than 512 bytes, the server puts as much response as it can into the DNS UDP response, and marks the "trunacted" flag. This lets glibc know that the DNS UDP packet did not contain the entire response for all the A records.
When glibc sees a UDP response that is "trunacted", by default it ignores the contents of that response and issues a new DNS query, using TCP instead of UDP. The TCP packet size has a higher size limit (though see bug 1804487 which is a bug in systemd's max-sizing of TCP DNS packets), and so *should* allow glibc to receive the entire DNS response.
However, glibc issues DNS queries for both A and AAAA records. When it uses UDP, those DNS queries are separate (i.e. one UDP DNS packet with a single A query, and one UDP DNS packet with a single AAAA query). When glibc uses TCP, it puts both DNS queries into a single TCP DNS packet - the RFC refers to this as "pipelining" (https://tools.ietf.org/html/rfc7766#section-6.2.1.1) and states that clients SHOULD do this, and that servers MUST expect to receive pipelined queries and SHOULD respond to all of them. (Technically pipelining can be separate DNS queries, one per TCP packet, but both using the same TCP connection - but the clear intention of pipelining is to improve TCP performance, and putting both DNS queries into a single TCP packet is clearly more performant than using separate TCP packets).
Unfortunately, systemd's local stub resolver has only very basic support for TCP DNS, and it handles TCP DNS queries almost identically to UDP DNS queries - it reads the DNS query 2-byte header (containing the length of the query data), reads in the single DNS query data, performs lookup and sends a response to that DNS query, and closes the TCP connection. It does not check for "pipelined" queries in the TCP connection.
That would be bad enough, as glibc is (rightly) expecting a response to both its A and AAAA queries; however what glibc gets is a TCP connection-reset error. That is because the local systemd stub resolver has closed its TCP socket while input data was still pending (i.e. it never even read the second pipelined DNS query). When the kernel sees unread input bytes in a TCP connection that is closed, it sends a TCP RST to the peer (i.e. glibc) and when the kernel sees the RST, it dumps all data in its socket buffer and passes the ECONNRESET error up to the application. So glibc gets nothing besides a connection reset error.
Note also that even if the systemd local stub resolver's socket flushes its input buffer before closing the TCP connection (which will avoid the TCP RST), glibc still expects responses to both its A and AAAA queries before systemd closes the TCP connection, and so a simple change to systemd to flush the input buffer is not enough to fix the bug (and would also not actually fix the bug since glibc would never get the AAAA response).
[Test Case]
This can be reproduced on any system using a local systemd stub resolver, when using an application that uses getaddrinfo() - such as ssh, telnet, ping, etc - or with a simple C program that uses getaddrinfo(). The dns name looked up must have enough A records to overflow the 512 byte maximum for a UDP DNS packet.
Alternately, and trivially, glibc can be forced to always use TCP DNS queries by editing the /etc/resolv.conf file and adding:
options use-vc
With that option, glibc will fail to lookup 100% of DNS names, since all lookups will use TCP to talk to the local systemd stub resolver, which as explained above fails to ever correctly answer glibc's pipelined TCP DNS queries.
Note that in default Ubuntu installs, /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf, which systemd thinks it owns 100% - so any manual changes to the file may be overwritten at any time. There is no way (that I can find) to tell systemd to add any resolv.conf options (like 'use-vc') to its managed stub-resolv.conf file, so this test case requires re-editing the /etc/resolv.conf file intermittently, each time systemd overwrites it.
[Regression Potential]
A (real) fix to this requires adding query TCP pipelining support to the systemd stub resolver. That is not a simple change, and has the potential to cause regressions with the stub resolver's responses to non-TCP queries, or even non-pipelined TCP queries.
[Other Info]
This bug exists upstream.
The specific bug of TCP DNS fallback not working for DNS responses larger than 512 bytes can be worked around by editing the /etc/resolv.conf file to add:
options edns0
The EDNS0 option causes glibc to fall back to attempting UDP EDNS0 query (which has a higher max packet size than the default 512 byte UDP DNS). The systemd stub resolver does support EDNS0. However, this workaround only temporarily works - as explained above, by default /etc/resolv.conf is a symlink to a file that systemd overwrites intermittently, which will remove the EDNS0 option. |
[Impact]
The systemd local 'stub' resolver handles all local DNS queries (by default configuration used in Ubuntu), and essentially proxies all requests to its configured upstream DNS resolvers.
Most local DNS resolution by applications uses glibc's getaddrinfo() function. This function is configured in various ways by the /etc/resolv.conf file, which tells glibc what nameserver/resolver to contact as well as how to talk to the name server.
By default, glibc performs UDP DNS queries, with a single DNS query per UDP packet. The UDP packet size is limited per DNS spec to 512 bytes. For some DNS lookups, a 512 byte UDP packet is not large enough to contain the entire response - for example, an A record lookup with a large number (e.g. 30) of A record addresses. This number of A record entries is possible in some cases of load balancing. When the DNS UDP response size is larger than 512 bytes, the server puts as much response as it can into the DNS UDP response, and marks the "trunacted" flag. This lets glibc know that the DNS UDP packet did not contain the entire response for all the A records.
When glibc sees a UDP response that is "trunacted", by default it ignores the contents of that response and issues a new DNS query, using TCP instead of UDP. The TCP packet size has a higher size limit (though see bug 1804487 which is a bug in systemd's max-sizing of TCP DNS packets), and so *should* allow glibc to receive the entire DNS response.
However, glibc issues DNS queries for both A and AAAA records. When it uses UDP, those DNS queries are separate (i.e. one UDP DNS packet with a single A query, and one UDP DNS packet with a single AAAA query). When glibc uses TCP, it puts both DNS queries into a single TCP DNS packet - the RFC refers to this as "pipelining" (https://tools.ietf.org/html/rfc7766#section-6.2.1.1) and states that clients SHOULD do this, and that servers MUST expect to receive pipelined queries and SHOULD respond to all of them. (Technically pipelining can be separate DNS queries, one per TCP packet, but both using the same TCP connection - but the clear intention of pipelining is to improve TCP performance, and putting both DNS queries into a single TCP packet is clearly more performant than using separate TCP packets).
Unfortunately, systemd's local stub resolver has only very basic support for TCP DNS, and it handles TCP DNS queries almost identically to UDP DNS queries - it reads the DNS query 2-byte header (containing the length of the query data), reads in the single DNS query data, performs lookup and sends a response to that DNS query, and closes the TCP connection. It does not check for "pipelined" queries in the TCP connection.
That would be bad enough, as glibc is (rightly) expecting a response to both its A and AAAA queries; however what glibc gets is a TCP connection-reset error. That is because the local systemd stub resolver has closed its TCP socket while input data was still pending (i.e. it never even read the second pipelined DNS query). When the kernel sees unread input bytes in a TCP connection that is closed, it sends a TCP RST to the peer (i.e. glibc) and when the kernel sees the RST, it dumps all data in its socket buffer and passes the ECONNRESET error up to the application. So glibc gets nothing besides a connection reset error.
Note also that even if the systemd local stub resolver's socket flushes its input buffer before closing the TCP connection (which will avoid the TCP RST), glibc still expects responses to both its A and AAAA queries before systemd closes the TCP connection, and so a simple change to systemd to flush the input buffer is not enough to fix the bug (and would also not actually fix the bug since glibc would never get the AAAA response).
[Test Case]
This can be reproduced on any system using a local systemd stub resolver, when using an application that uses getaddrinfo() - such as ssh, telnet, ping, etc - or with a simple C program that uses getaddrinfo(). The dns name looked up must have enough A records to overflow the 512 byte maximum for a UDP DNS packet.
Alternately, and trivially, glibc can be forced to always use TCP DNS queries by editing the /etc/resolv.conf file and adding:
options use-vc
With that option, glibc will fail to lookup 100% of DNS names, since all lookups will use TCP to talk to the local systemd stub resolver, which as explained above fails to ever correctly answer glibc's pipelined TCP DNS queries.
Note that in default Ubuntu installs, /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf, which systemd thinks it owns 100% - so any manual changes to the file may be overwritten at any time. There is no way (that I can find) to tell systemd to add any resolv.conf options (like 'use-vc') to its managed stub-resolv.conf file, so this test case requires re-editing the /etc/resolv.conf file intermittently, each time systemd overwrites it.
Note also that the patch used to work around this (see Other Info below) will fix the case of lookup failures for very long A records; but the workaround will not help at all with the test case of using 'option use-vc'. That test case will continue to fail for 100% of dns lookups.
[Regression Potential]
To workaround this, the patch enables edns0 in systemd's stub resolver resolv.conf file. This could cause problems for any system code that does not expect the resolv.conf file to include a new line/option, or could introduce problems with edns0 lookups, since glibc was not previously using edns0.
[Other Info]
This bug exists upstream, with proposed patches to add dns tcp pipeline support:
https://github.com/systemd/systemd/pull/11512
The specific bug of TCP DNS fallback not working for DNS responses larger than 512 bytes can be worked around by editing the /etc/resolv.conf file to add:
options edns0
The EDNS0 option causes glibc to fall back to attempting UDP EDNS0 query (which has a higher max packet size than the default 512 byte UDP DNS). The systemd stub resolver does support EDNS0. However, this workaround only temporarily works - as explained above, by default /etc/resolv.conf is a symlink to a file that systemd overwrites intermittently, which will remove the EDNS0 option.
The upstream patch that will be used to work around this bug in exactly that way (i.e. adding option edns0 to resolv.conf) is:
https://github.com/systemd/systemd/commit/93158c77bc69fde7cf5cff733617631c1e566fe8
That patch is already included in Debian and so no Debian bug is required for this bug (since the only fix for this specific bug will be sru'ing the edns0 workaround) |
|
| 2019-01-29 20:00:02 |
Dan Streetman |
systemd (Ubuntu Xenial): status |
In Progress |
Invalid |
|
| 2019-01-29 20:00:06 |
Dan Streetman |
systemd (Ubuntu Trusty): status |
In Progress |
Invalid |
|
| 2019-01-29 20:00:11 |
Dan Streetman |
systemd (Ubuntu Xenial): importance |
High |
Undecided |
|
| 2019-01-29 20:00:13 |
Dan Streetman |
systemd (Ubuntu Trusty): importance |
High |
Undecided |
|
| 2019-01-29 20:00:16 |
Dan Streetman |
systemd (Ubuntu Trusty): assignee |
Dan Streetman (ddstreet) |
|
|
| 2019-01-29 20:00:18 |
Dan Streetman |
systemd (Ubuntu Xenial): assignee |
Dan Streetman (ddstreet) |
|
|
| 2019-01-29 20:00:53 |
Dan Streetman |
description |
[Impact]
The systemd local 'stub' resolver handles all local DNS queries (by default configuration used in Ubuntu), and essentially proxies all requests to its configured upstream DNS resolvers.
Most local DNS resolution by applications uses glibc's getaddrinfo() function. This function is configured in various ways by the /etc/resolv.conf file, which tells glibc what nameserver/resolver to contact as well as how to talk to the name server.
By default, glibc performs UDP DNS queries, with a single DNS query per UDP packet. The UDP packet size is limited per DNS spec to 512 bytes. For some DNS lookups, a 512 byte UDP packet is not large enough to contain the entire response - for example, an A record lookup with a large number (e.g. 30) of A record addresses. This number of A record entries is possible in some cases of load balancing. When the DNS UDP response size is larger than 512 bytes, the server puts as much response as it can into the DNS UDP response, and marks the "trunacted" flag. This lets glibc know that the DNS UDP packet did not contain the entire response for all the A records.
When glibc sees a UDP response that is "trunacted", by default it ignores the contents of that response and issues a new DNS query, using TCP instead of UDP. The TCP packet size has a higher size limit (though see bug 1804487 which is a bug in systemd's max-sizing of TCP DNS packets), and so *should* allow glibc to receive the entire DNS response.
However, glibc issues DNS queries for both A and AAAA records. When it uses UDP, those DNS queries are separate (i.e. one UDP DNS packet with a single A query, and one UDP DNS packet with a single AAAA query). When glibc uses TCP, it puts both DNS queries into a single TCP DNS packet - the RFC refers to this as "pipelining" (https://tools.ietf.org/html/rfc7766#section-6.2.1.1) and states that clients SHOULD do this, and that servers MUST expect to receive pipelined queries and SHOULD respond to all of them. (Technically pipelining can be separate DNS queries, one per TCP packet, but both using the same TCP connection - but the clear intention of pipelining is to improve TCP performance, and putting both DNS queries into a single TCP packet is clearly more performant than using separate TCP packets).
Unfortunately, systemd's local stub resolver has only very basic support for TCP DNS, and it handles TCP DNS queries almost identically to UDP DNS queries - it reads the DNS query 2-byte header (containing the length of the query data), reads in the single DNS query data, performs lookup and sends a response to that DNS query, and closes the TCP connection. It does not check for "pipelined" queries in the TCP connection.
That would be bad enough, as glibc is (rightly) expecting a response to both its A and AAAA queries; however what glibc gets is a TCP connection-reset error. That is because the local systemd stub resolver has closed its TCP socket while input data was still pending (i.e. it never even read the second pipelined DNS query). When the kernel sees unread input bytes in a TCP connection that is closed, it sends a TCP RST to the peer (i.e. glibc) and when the kernel sees the RST, it dumps all data in its socket buffer and passes the ECONNRESET error up to the application. So glibc gets nothing besides a connection reset error.
Note also that even if the systemd local stub resolver's socket flushes its input buffer before closing the TCP connection (which will avoid the TCP RST), glibc still expects responses to both its A and AAAA queries before systemd closes the TCP connection, and so a simple change to systemd to flush the input buffer is not enough to fix the bug (and would also not actually fix the bug since glibc would never get the AAAA response).
[Test Case]
This can be reproduced on any system using a local systemd stub resolver, when using an application that uses getaddrinfo() - such as ssh, telnet, ping, etc - or with a simple C program that uses getaddrinfo(). The dns name looked up must have enough A records to overflow the 512 byte maximum for a UDP DNS packet.
Alternately, and trivially, glibc can be forced to always use TCP DNS queries by editing the /etc/resolv.conf file and adding:
options use-vc
With that option, glibc will fail to lookup 100% of DNS names, since all lookups will use TCP to talk to the local systemd stub resolver, which as explained above fails to ever correctly answer glibc's pipelined TCP DNS queries.
Note that in default Ubuntu installs, /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf, which systemd thinks it owns 100% - so any manual changes to the file may be overwritten at any time. There is no way (that I can find) to tell systemd to add any resolv.conf options (like 'use-vc') to its managed stub-resolv.conf file, so this test case requires re-editing the /etc/resolv.conf file intermittently, each time systemd overwrites it.
Note also that the patch used to work around this (see Other Info below) will fix the case of lookup failures for very long A records; but the workaround will not help at all with the test case of using 'option use-vc'. That test case will continue to fail for 100% of dns lookups.
[Regression Potential]
To workaround this, the patch enables edns0 in systemd's stub resolver resolv.conf file. This could cause problems for any system code that does not expect the resolv.conf file to include a new line/option, or could introduce problems with edns0 lookups, since glibc was not previously using edns0.
[Other Info]
This bug exists upstream, with proposed patches to add dns tcp pipeline support:
https://github.com/systemd/systemd/pull/11512
The specific bug of TCP DNS fallback not working for DNS responses larger than 512 bytes can be worked around by editing the /etc/resolv.conf file to add:
options edns0
The EDNS0 option causes glibc to fall back to attempting UDP EDNS0 query (which has a higher max packet size than the default 512 byte UDP DNS). The systemd stub resolver does support EDNS0. However, this workaround only temporarily works - as explained above, by default /etc/resolv.conf is a symlink to a file that systemd overwrites intermittently, which will remove the EDNS0 option.
The upstream patch that will be used to work around this bug in exactly that way (i.e. adding option edns0 to resolv.conf) is:
https://github.com/systemd/systemd/commit/93158c77bc69fde7cf5cff733617631c1e566fe8
That patch is already included in Debian and so no Debian bug is required for this bug (since the only fix for this specific bug will be sru'ing the edns0 workaround) |
[Impact]
The systemd local 'stub' resolver handles all local DNS queries (by default configuration used in Ubuntu), and essentially proxies all requests to its configured upstream DNS resolvers.
Most local DNS resolution by applications uses glibc's getaddrinfo() function. This function is configured in various ways by the /etc/resolv.conf file, which tells glibc what nameserver/resolver to contact as well as how to talk to the name server.
By default, glibc performs UDP DNS queries, with a single DNS query per UDP packet. The UDP packet size is limited per DNS spec to 512 bytes. For some DNS lookups, a 512 byte UDP packet is not large enough to contain the entire response - for example, an A record lookup with a large number (e.g. 30) of A record addresses. This number of A record entries is possible in some cases of load balancing. When the DNS UDP response size is larger than 512 bytes, the server puts as much response as it can into the DNS UDP response, and marks the "trunacted" flag. This lets glibc know that the DNS UDP packet did not contain the entire response for all the A records.
When glibc sees a UDP response that is "trunacted", by default it ignores the contents of that response and issues a new DNS query, using TCP instead of UDP. The TCP packet size has a higher size limit (though see bug 1804487 which is a bug in systemd's max-sizing of TCP DNS packets), and so *should* allow glibc to receive the entire DNS response.
However, glibc issues DNS queries for both A and AAAA records. When it uses UDP, those DNS queries are separate (i.e. one UDP DNS packet with a single A query, and one UDP DNS packet with a single AAAA query). When glibc uses TCP, it puts both DNS queries into a single TCP DNS packet - the RFC refers to this as "pipelining" (https://tools.ietf.org/html/rfc7766#section-6.2.1.1) and states that clients SHOULD do this, and that servers MUST expect to receive pipelined queries and SHOULD respond to all of them. (Technically pipelining can be separate DNS queries, one per TCP packet, but both using the same TCP connection - but the clear intention of pipelining is to improve TCP performance, and putting both DNS queries into a single TCP packet is clearly more performant than using separate TCP packets).
Unfortunately, systemd's local stub resolver has only very basic support for TCP DNS, and it handles TCP DNS queries almost identically to UDP DNS queries - it reads the DNS query 2-byte header (containing the length of the query data), reads in the single DNS query data, performs lookup and sends a response to that DNS query, and closes the TCP connection. It does not check for "pipelined" queries in the TCP connection.
That would be bad enough, as glibc is (rightly) expecting a response to both its A and AAAA queries; however what glibc gets is a TCP connection-reset error. That is because the local systemd stub resolver has closed its TCP socket while input data was still pending (i.e. it never even read the second pipelined DNS query). When the kernel sees unread input bytes in a TCP connection that is closed, it sends a TCP RST to the peer (i.e. glibc) and when the kernel sees the RST, it dumps all data in its socket buffer and passes the ECONNRESET error up to the application. So glibc gets nothing besides a connection reset error.
Note also that even if the systemd local stub resolver's socket flushes its input buffer before closing the TCP connection (which will avoid the TCP RST), glibc still expects responses to both its A and AAAA queries before systemd closes the TCP connection, and so a simple change to systemd to flush the input buffer is not enough to fix the bug (and would also not actually fix the bug since glibc would never get the AAAA response).
[Test Case]
This can be reproduced on any system using a local systemd stub resolver, when using an application that uses getaddrinfo() - such as ssh, telnet, ping, etc - or with a simple C program that uses getaddrinfo(). The dns name looked up must have enough A records to overflow the 512 byte maximum for a UDP DNS packet.
Alternately, and trivially, glibc can be forced to always use TCP DNS queries by editing the /etc/resolv.conf file and adding:
options use-vc
With that option, glibc will fail to lookup 100% of DNS names, since all lookups will use TCP to talk to the local systemd stub resolver, which as explained above fails to ever correctly answer glibc's pipelined TCP DNS queries.
Note that in default Ubuntu installs, /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf, which systemd thinks it owns 100% - so any manual changes to the file may be overwritten at any time. There is no way (that I can find) to tell systemd to add any resolv.conf options (like 'use-vc') to its managed stub-resolv.conf file, so this test case requires re-editing the /etc/resolv.conf file intermittently, each time systemd overwrites it.
Note also that the patch used to work around this (see Other Info below) will fix the case of lookup failures for very long A records; but the workaround will not help at all with the test case of using 'option use-vc'. That test case will continue to fail for 100% of dns lookups.
[Regression Potential]
To workaround this, the patch enables edns0 in systemd's stub resolver resolv.conf file. This could cause problems for any system code that does not expect the resolv.conf file to include a new line/option, or could introduce problems with edns0 lookups, since glibc was not previously using edns0.
[Other Info]
This bug exists upstream, with proposed patches to add dns tcp pipeline support:
https://github.com/systemd/systemd/pull/11512
The specific bug of TCP DNS fallback not working for DNS responses larger than 512 bytes can be worked around by editing the /etc/resolv.conf file to add:
options edns0
The EDNS0 option causes glibc to fall back to attempting UDP EDNS0 query (which has a higher max packet size than the default 512 byte UDP DNS). The systemd stub resolver does support EDNS0. However, this workaround only temporarily works - as explained above, by default /etc/resolv.conf is a symlink to a file that systemd overwrites intermittently, which will remove the EDNS0 option.
The upstream patch that will be used to work around this bug in exactly that way (i.e. adding option edns0 to resolv.conf) is:
https://github.com/systemd/systemd/commit/93158c77bc69fde7cf5cff733617631c1e566fe8
That patch is already included in Debian and so no Debian bug is required for this bug (since the only fix for this specific bug will be sru'ing the edns0 workaround)
Since Xenial and Trusty do not use the systemd stub resolver (by default) I marked this Invalid for those releases. |
|
| 2019-01-29 20:04:13 |
Dan Streetman |
description |
[Impact]
The systemd local 'stub' resolver handles all local DNS queries (by default configuration used in Ubuntu), and essentially proxies all requests to its configured upstream DNS resolvers.
Most local DNS resolution by applications uses glibc's getaddrinfo() function. This function is configured in various ways by the /etc/resolv.conf file, which tells glibc what nameserver/resolver to contact as well as how to talk to the name server.
By default, glibc performs UDP DNS queries, with a single DNS query per UDP packet. The UDP packet size is limited per DNS spec to 512 bytes. For some DNS lookups, a 512 byte UDP packet is not large enough to contain the entire response - for example, an A record lookup with a large number (e.g. 30) of A record addresses. This number of A record entries is possible in some cases of load balancing. When the DNS UDP response size is larger than 512 bytes, the server puts as much response as it can into the DNS UDP response, and marks the "trunacted" flag. This lets glibc know that the DNS UDP packet did not contain the entire response for all the A records.
When glibc sees a UDP response that is "trunacted", by default it ignores the contents of that response and issues a new DNS query, using TCP instead of UDP. The TCP packet size has a higher size limit (though see bug 1804487 which is a bug in systemd's max-sizing of TCP DNS packets), and so *should* allow glibc to receive the entire DNS response.
However, glibc issues DNS queries for both A and AAAA records. When it uses UDP, those DNS queries are separate (i.e. one UDP DNS packet with a single A query, and one UDP DNS packet with a single AAAA query). When glibc uses TCP, it puts both DNS queries into a single TCP DNS packet - the RFC refers to this as "pipelining" (https://tools.ietf.org/html/rfc7766#section-6.2.1.1) and states that clients SHOULD do this, and that servers MUST expect to receive pipelined queries and SHOULD respond to all of them. (Technically pipelining can be separate DNS queries, one per TCP packet, but both using the same TCP connection - but the clear intention of pipelining is to improve TCP performance, and putting both DNS queries into a single TCP packet is clearly more performant than using separate TCP packets).
Unfortunately, systemd's local stub resolver has only very basic support for TCP DNS, and it handles TCP DNS queries almost identically to UDP DNS queries - it reads the DNS query 2-byte header (containing the length of the query data), reads in the single DNS query data, performs lookup and sends a response to that DNS query, and closes the TCP connection. It does not check for "pipelined" queries in the TCP connection.
That would be bad enough, as glibc is (rightly) expecting a response to both its A and AAAA queries; however what glibc gets is a TCP connection-reset error. That is because the local systemd stub resolver has closed its TCP socket while input data was still pending (i.e. it never even read the second pipelined DNS query). When the kernel sees unread input bytes in a TCP connection that is closed, it sends a TCP RST to the peer (i.e. glibc) and when the kernel sees the RST, it dumps all data in its socket buffer and passes the ECONNRESET error up to the application. So glibc gets nothing besides a connection reset error.
Note also that even if the systemd local stub resolver's socket flushes its input buffer before closing the TCP connection (which will avoid the TCP RST), glibc still expects responses to both its A and AAAA queries before systemd closes the TCP connection, and so a simple change to systemd to flush the input buffer is not enough to fix the bug (and would also not actually fix the bug since glibc would never get the AAAA response).
[Test Case]
This can be reproduced on any system using a local systemd stub resolver, when using an application that uses getaddrinfo() - such as ssh, telnet, ping, etc - or with a simple C program that uses getaddrinfo(). The dns name looked up must have enough A records to overflow the 512 byte maximum for a UDP DNS packet.
Alternately, and trivially, glibc can be forced to always use TCP DNS queries by editing the /etc/resolv.conf file and adding:
options use-vc
With that option, glibc will fail to lookup 100% of DNS names, since all lookups will use TCP to talk to the local systemd stub resolver, which as explained above fails to ever correctly answer glibc's pipelined TCP DNS queries.
Note that in default Ubuntu installs, /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf, which systemd thinks it owns 100% - so any manual changes to the file may be overwritten at any time. There is no way (that I can find) to tell systemd to add any resolv.conf options (like 'use-vc') to its managed stub-resolv.conf file, so this test case requires re-editing the /etc/resolv.conf file intermittently, each time systemd overwrites it.
Note also that the patch used to work around this (see Other Info below) will fix the case of lookup failures for very long A records; but the workaround will not help at all with the test case of using 'option use-vc'. That test case will continue to fail for 100% of dns lookups.
[Regression Potential]
To workaround this, the patch enables edns0 in systemd's stub resolver resolv.conf file. This could cause problems for any system code that does not expect the resolv.conf file to include a new line/option, or could introduce problems with edns0 lookups, since glibc was not previously using edns0.
[Other Info]
This bug exists upstream, with proposed patches to add dns tcp pipeline support:
https://github.com/systemd/systemd/pull/11512
The specific bug of TCP DNS fallback not working for DNS responses larger than 512 bytes can be worked around by editing the /etc/resolv.conf file to add:
options edns0
The EDNS0 option causes glibc to fall back to attempting UDP EDNS0 query (which has a higher max packet size than the default 512 byte UDP DNS). The systemd stub resolver does support EDNS0. However, this workaround only temporarily works - as explained above, by default /etc/resolv.conf is a symlink to a file that systemd overwrites intermittently, which will remove the EDNS0 option.
The upstream patch that will be used to work around this bug in exactly that way (i.e. adding option edns0 to resolv.conf) is:
https://github.com/systemd/systemd/commit/93158c77bc69fde7cf5cff733617631c1e566fe8
That patch is already included in Debian and so no Debian bug is required for this bug (since the only fix for this specific bug will be sru'ing the edns0 workaround)
Since Xenial and Trusty do not use the systemd stub resolver (by default) I marked this Invalid for those releases. |
[Impact]
The systemd local 'stub' resolver handles all local DNS queries (by default configuration used in Ubuntu), and essentially proxies all requests to its configured upstream DNS resolvers.
Most local DNS resolution by applications uses glibc's getaddrinfo() function. This function is configured in various ways by the /etc/resolv.conf file, which tells glibc what nameserver/resolver to contact as well as how to talk to the name server.
By default, glibc performs UDP DNS queries, with a single DNS query per UDP packet. The UDP packet size is limited per DNS spec to 512 bytes. For some DNS lookups, a 512 byte UDP packet is not large enough to contain the entire response - for example, an A record lookup with a large number (e.g. 30) of A record addresses. This number of A record entries is possible in some cases of load balancing. When the DNS UDP response size is larger than 512 bytes, the server puts as much response as it can into the DNS UDP response, and marks the "trunacted" flag. This lets glibc know that the DNS UDP packet did not contain the entire response for all the A records.
When glibc sees a UDP response that is "trunacted", by default it ignores the contents of that response and issues a new DNS query, using TCP instead of UDP. The TCP packet size has a higher size limit (though see bug 1804487 which is a bug in systemd's max-sizing of TCP DNS packets), and so *should* allow glibc to receive the entire DNS response.
However, glibc issues DNS queries for both A and AAAA records. When it uses UDP, those DNS queries are separate (i.e. one UDP DNS packet with a single A query, and one UDP DNS packet with a single AAAA query). When glibc uses TCP, it puts both DNS queries into a single TCP DNS packet - the RFC refers to this as "pipelining" (https://tools.ietf.org/html/rfc7766#section-6.2.1.1) and states that clients SHOULD do this, and that servers MUST expect to receive pipelined queries and SHOULD respond to all of them. (Technically pipelining can be separate DNS queries, one per TCP packet, but both using the same TCP connection - but the clear intention of pipelining is to improve TCP performance, and putting both DNS queries into a single TCP packet is clearly more performant than using separate TCP packets).
Unfortunately, systemd's local stub resolver has only very basic support for TCP DNS, and it handles TCP DNS queries almost identically to UDP DNS queries - it reads the DNS query 2-byte header (containing the length of the query data), reads in the single DNS query data, performs lookup and sends a response to that DNS query, and closes the TCP connection. It does not check for "pipelined" queries in the TCP connection.
That would be bad enough, as glibc is (rightly) expecting a response to both its A and AAAA queries; however what glibc gets is a TCP connection-reset error. That is because the local systemd stub resolver has closed its TCP socket while input data was still pending (i.e. it never even read the second pipelined DNS query). When the kernel sees unread input bytes in a TCP connection that is closed, it sends a TCP RST to the peer (i.e. glibc) and when the kernel sees the RST, it dumps all data in its socket buffer and passes the ECONNRESET error up to the application. So glibc gets nothing besides a connection reset error.
Note also that even if the systemd local stub resolver's socket flushes its input buffer before closing the TCP connection (which will avoid the TCP RST), glibc still expects responses to both its A and AAAA queries before systemd closes the TCP connection, and so a simple change to systemd to flush the input buffer is not enough to fix the bug (and would also not actually fix the bug since glibc would never get the AAAA response).
[Test Case]
This can be reproduced on any system using a local systemd stub resolver, when using an application that uses getaddrinfo() - such as ssh, telnet, ping, etc - or with a simple C program that uses getaddrinfo(). The dns name looked up must have enough A records to overflow the 512 byte maximum for a UDP DNS packet; e.g.:
$ ping testing.irongiantdesign.com
ping: testing.irongiantdesign.com: Temporary failure in name resolution
Alternately, and trivially, glibc can be forced to always use TCP DNS queries by editing the /etc/resolv.conf file and adding:
options use-vc
With that option, glibc will fail to lookup 100% of DNS names, since all lookups will use TCP to talk to the local systemd stub resolver, which as explained above fails to ever correctly answer glibc's pipelined TCP DNS queries.
Note that in default Ubuntu installs, /etc/resolv.conf is a symlink to ../run/systemd/resolve/stub-resolv.conf, which systemd thinks it owns 100% - so any manual changes to the file may be overwritten at any time. There is no way (that I can find) to tell systemd to add any resolv.conf options (like 'use-vc') to its managed stub-resolv.conf file, so this test case requires re-editing the /etc/resolv.conf file intermittently, each time systemd overwrites it.
Note also that the patch used to work around this (see Other Info below) will fix the case of lookup failures for very long A records; but the workaround will not help at all with the test case of using 'option use-vc'. That test case will continue to fail for 100% of dns lookups.
[Regression Potential]
To workaround this, the patch enables edns0 in systemd's stub resolver resolv.conf file. This could cause problems for any system code that does not expect the resolv.conf file to include a new line/option, or could introduce problems with edns0 lookups, since glibc was not previously using edns0.
[Other Info]
This bug exists upstream, with proposed patches to add dns tcp pipeline support:
https://github.com/systemd/systemd/pull/11512
The specific bug of TCP DNS fallback not working for DNS responses larger than 512 bytes can be worked around by editing the /etc/resolv.conf file to add:
options edns0
The EDNS0 option causes glibc to fall back to attempting UDP EDNS0 query (which has a higher max packet size than the default 512 byte UDP DNS). The systemd stub resolver does support EDNS0. However, this workaround only temporarily works - as explained above, by default /etc/resolv.conf is a symlink to a file that systemd overwrites intermittently, which will remove the EDNS0 option.
The upstream patch that will be used to work around this bug in exactly that way (i.e. adding option edns0 to resolv.conf) is:
https://github.com/systemd/systemd/commit/93158c77bc69fde7cf5cff733617631c1e566fe8
That patch is already included in Debian and so no Debian bug is required for this bug (since the only fix for this specific bug will be sru'ing the edns0 workaround)
Since Xenial and Trusty do not use the systemd stub resolver (by default) I marked this Invalid for those releases. |
|
| 2019-01-30 11:09:32 |
Dimitri John Ledkov |
systemd (Ubuntu Disco): status |
In Progress |
Fix Committed |
|
| 2019-01-30 11:09:37 |
Dimitri John Ledkov |
systemd (Ubuntu Disco): assignee |
Dan Streetman (ddstreet) |
Dimitri John Ledkov (xnox) |
|
| 2019-01-31 15:21:02 |
Łukasz Zemczak |
systemd (Ubuntu Cosmic): status |
In Progress |
Fix Committed |
|
| 2019-01-31 15:21:04 |
Łukasz Zemczak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-01-31 15:21:06 |
Łukasz Zemczak |
bug |
|
|
added subscriber SRU Verification |
| 2019-01-31 15:21:11 |
Łukasz Zemczak |
tags |
|
verification-needed verification-needed-cosmic |
|
| 2019-01-31 15:27:17 |
Łukasz Zemczak |
systemd (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2019-01-31 15:27:22 |
Łukasz Zemczak |
tags |
verification-needed verification-needed-cosmic |
verification-needed verification-needed-bionic verification-needed-cosmic |
|
| 2019-01-31 21:02:16 |
Bryan Quigley |
tags |
verification-needed verification-needed-bionic verification-needed-cosmic |
verification-done-cosmic verification-needed verification-needed-bionic |
|
| 2019-01-31 21:14:27 |
Bryan Quigley |
tags |
verification-done-cosmic verification-needed verification-needed-bionic |
verification-done verification-done-bionic verification-done-cosmic |
|
| 2019-02-03 22:58:43 |
Launchpad Janitor |
systemd (Ubuntu Disco): status |
Fix Committed |
Fix Released |
|
| 2019-02-05 22:32:10 |
Launchpad Janitor |
systemd (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2019-02-05 22:32:30 |
Adam Conrad |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-02-05 22:32:56 |
Launchpad Janitor |
systemd (Ubuntu Cosmic): status |
Fix Committed |
Fix Released |
|
| 2019-03-02 07:53:21 |
Bug Watch Updater |
systemd: status |
New |
Fix Released |
|
| 2019-03-30 02:39:44 |
Mathew Hodson |
bug task deleted |
systemd (Ubuntu Trusty) |
|
|
| 2019-03-30 02:39:52 |
Mathew Hodson |
bug task deleted |
systemd (Ubuntu Xenial) |
|
|
| 2019-04-07 01:26:54 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~ddstreet/ubuntu/+source/systemd/+git/systemd/+merge/365627 |
|
| 2019-05-17 07:13:18 |
Steve Langasek |
bug task added |
|
resolvconf (Ubuntu) |
|
| 2019-05-17 07:14:18 |
Steve Langasek |
resolvconf (Ubuntu Bionic): status |
New |
Triaged |
|
| 2019-05-17 07:14:21 |
Steve Langasek |
resolvconf (Ubuntu Cosmic): status |
New |
Triaged |
|
| 2019-05-17 07:14:23 |
Steve Langasek |
resolvconf (Ubuntu Disco): status |
New |
Triaged |
|
| 2019-05-17 07:14:25 |
Steve Langasek |
resolvconf (Ubuntu): status |
New |
Triaged |
|
| 2019-05-17 12:25:01 |
Francis Ginther |
tags |
verification-done verification-done-bionic verification-done-cosmic |
id-5cde5f8331588344774efccb verification-done verification-done-bionic verification-done-cosmic |
|
| 2019-06-17 07:23:18 |
sordna |
bug |
|
|
added subscriber sordna |
| 2019-06-17 16:47:43 |
Nick B. |
bug |
|
|
added subscriber Nick B. |
| 2020-04-07 10:48:17 |
Marin Nedea |
bug watch added |
|
https://github.com/Azure/WALinuxAgent/issues/1673 |
|
| 2020-07-02 20:00:08 |
Steve Langasek |
resolvconf (Ubuntu Disco): status |
Triaged |
Won't Fix |
|
| 2024-07-26 16:57:47 |
Brian Murray |
resolvconf (Ubuntu Cosmic): status |
Triaged |
Won't Fix |
|
| 2024-07-30 20:57:20 |
Dan Streetman |
removed subscriber Dan Streetman |
|
|
|
