systemd-resolved can't resolve Comcast mail server addresses

[Tomdkat]

1) Ubuntu release: 18.10
2) systemd-resolved version: (Default latest version that comes with Ubuntu 18.10)
3) Expected behavior: Comcast's POP3 mail server addresses to be resolved to IP addresses
4) Actual behavior: Comcast's POP3 mail server addresses can't be resolved to IP addresses

Starting on Monday, November 19, 2018, Comcast made a DNS change related to its POP3 mail servers (mail.comcast.net and pop3.comcast.net) that prevent resolved from being able to resolve those domains into IP addresses. When I try to ping either host (mail.comcast.net or pop2.comcast.net), I get this error:

tom@deathstar:~$ ping mail.comcast.net
ping: mail.comcast.net: Name or service not known
tom@deathstar:~$

When I manually lookup up the domain, I get these results:

tom@deathstar:~$ nslookup mail.comcast.net
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
mail.comcast.net canonical name = imap.ge.xfinity.com.
Name: imap.ge.xfinity.com
Address: 96.118.242.209
Name: imap.ge.xfinity.com
Address: 96.118.242.197
Name: imap.ge.xfinity.com
Address: 96.118.242.233
Name: imap.ge.xfinity.com
Address: 96.118.242.225
Name: imap.ge.xfinity.com
Address: 96.118.242.226
Name: imap.ge.xfinity.com
Address: 96.118.242.217
Name: imap.ge.xfinity.com
Address: 96.118.242.208
Name: imap.ge.xfinity.com
Address: 96.118.242.230
Name: imap.ge.xfinity.com
Address: 96.118.242.232
Name: imap.ge.xfinity.com
Address: 96.118.242.218
Name: imap.ge.xfinity.com
Address: 96.118.242.211
Name: imap.ge.xfinity.com
Address: 96.118.242.242
Name: imap.ge.xfinity.com
Address: 96.118.242.221
Name: imap.ge.xfinity.com
Address: 96.118.242.196
Name: imap.ge.xfinity.com
Address: 96.118.208.40
Name: imap.ge.xfinity.com
Address: 96.118.208.99
Name: imap.ge.xfinity.com
Address: 2001:558:fc11:9:f816:3eff:fee8:4f07
Name: imap.ge.xfinity.com
Address: 2001:558:fc11:9:f816:3eff:fe7d:1b0c
Name: imap.ge.xfinity.com
Address: 2001:558:fc11:9:f816:3eff:fe25:5ae5
Name: imap.ge.xfinity.com
Address: 2001:558:fc11:9:f816:3eff:fef6:babc
Name: imap.ge.xfinity.com
Address: 2001:558:fc11:9:f816:3eff:fe87:c172
Name: imap.ge.xfinity.com
Address: 2001:558:fc11:9:f816:3eff:fee6:7a57
Name: imap.ge.xfinity.com
Address: 2001:558:fc11:9:f816:3eff:fe0f:a4a
Name: imap.ge.xfinity.com
Address: 2001:558:fc11:2:f816:3eff:fec7:cb93
Name: imap.ge.xfinity.com
Address: 2001:558:fee2:1000:f816:3eff:fe42:4f14
Name: imap.ge.xfinity.com
Address: 2001:558:fc18:0:f816:3eff:fe33:9aaa
Name: imap.ge.xfinity.com
Address: 2001:558:fc18:0:f816:3eff:feb2:8c0d
Name: imap.ge.xfinity.com
Address: 2001:558:fc18:0:f816:3eff:fef1:25a5
Name: imap.ge.xfinity.com
Address: 2001:558:fc18:0:f816:3eff:febd:320a
Name: imap.ge.xfinity.com
Address: 2001:558:fc18:0:f816:3eff:fe36:aba3
Name: imap.ge.xfinity.com
Address: 2001:558:fc18:0:f816:3eff:fe3f:76f2
Name: imap.ge.xfinity.com
Address: 2001:558:fc18:0:f816:3eff:fe45:1d1e

tom@deathstar:~$ dig mail.comcast.net

; <<>> DiG 9.11.4-3ubuntu5-Ubuntu <<>> mail.comcast.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15037
;; flags: qr rd ra; QUERY: 1, ANSWER: 17, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;mail.comcast.net. IN A

;; ANSWER SECTION:
mail.comcast.net. 15 IN CNAME imap.ge.xfinity.com.
imap.ge.xfinity.com. 12 IN A 96.117.3.119
imap.ge.xfinity.com. 12 IN A 96.117.3.96
imap.ge.xfinity.com. 12 IN A 96.117.3.143
imap.ge.xfinity.com. 12 IN A 96.117.3.145
imap.ge.xfinity.com. 12 IN A 96.117.3.129
imap.ge.xfinity.com. 12 IN A 96.117.3.148
imap.ge.xfinity.com. 12 IN A 96.117.3.201
imap.ge.xfinity.com. 12 IN A 96.117.3.136
imap.ge.xfinity.com. 12 IN A 96.118.133.238
imap.ge.xfinity.com. 12 IN A 96.117.3.128
imap.ge.xfinity.com. 12 IN A 96.117.3.144
imap.ge.xfinity.com. 12 IN A 96.117.2.238
imap.ge.xfinity.com. 12 IN A 96.117.3.110
imap.ge.xfinity.com. 12 IN A 96.117.3.140
imap.ge.xfinity.com. 12 IN A 96.117.3.154
imap.ge.xfinity.com. 12 IN A 96.117.3.132

;; Query time: 13 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Nov 20 06:58:31 PST 2018
;; MSG SIZE rcvd: 334

tom@deathstar:~$

Before November 19, 2018, everything was working fine. The initial indication there was a problem was Mozilla Thunderbird failed to connect to Comcast's POP3 mail server. Once I did some investigation, I found the information above.

I have since disabled system-resolved and have installed "Unbound". After installing "Unbound", I'm able to ping "mail.comcast.net" without issue and Mozilla Thunderbird can contact successfully Comcast's POP3 mail servers.

I started a thread about this on Ubuntuforums.org:

https://ubuntuforums.org/showthread.php?t=2406399

and others report being impacted by this as well.

Let me know if more information is needed! Thanks!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Trev Peterson (trev-advanced-reality)]

This certainly seems like an ubuntu 18.04 bug. 3 machines here all affected. dig imap.comcast.net resolves to imap.ge.xfinity.com. Ping, telnet, wget are all unable to resolve imap.comcast.net.

[Brian Murray (brian-murray)]

fetchmail is also affected e.g.

 $ fetchmail -p imap imap.comcast.net
Enter password for <email address hidden>:
fetchmail: couldn't find canonical DNS name of imap.comcast.net (imap.comcast.net): Temporary failure in name resolution
fetchmail: Query status=11 (DNS)

Even if you aren't using comcast for email this is a good way to test it, just use some made up password.

[Brian Murray (brian-murray)]

There is a systemd SRU for Cosmic in progress, bug 1804487, which may be the same issue as this bug. If someone affected could install the version of systemd from cosmic-proposed and report back that would be helpful. Thanks!

[Mathieu Trudel-Lapierre (cyphermox)]

Setting to Triaged: it's easily reproduced by developers, and we have all the information we need to debug it -- nothing is secret or hidden, just needs someone to look at the packets and what resolved does with them.

[Brian Murray (brian-murray)]

I copied the patch debian/patches/resolved-Increase-size-of-TCP-stub-replies.patch (cherry picked from commit e6eed9445956cfa496e1db933bfd3530db23bfce) from the upload of systemd that was in cosmic-proposed and added to a package I built for 18.04 on top of systemd version 237-3ubuntu10.11. I installed the following packages:

ii libnss-systemd:amd64 237-3ubuntu10.11~ppa1 amd64 nss module providing dynamic user and group name resolution
ii libpam-systemd:amd64 237-3ubuntu10.11~ppa1 amd64 system and service manager - PAM module
ii libsystemd0:amd64 237-3ubuntu10.11~ppa1 amd64 systemd utility library
ii systemd 237-3ubuntu10.11~ppa1 amd64 system and service manager
ii systemd-sysv 237-3ubuntu10.11~ppa1 amd64 system and service manager - SysV links

and the issue was not resolved.

 $ wget imap.comcast.net
--2019-01-17 09:15:18-- http://imap.comcast.net/
Resolving imap.comcast.net (imap.comcast.net)... failed: Name or service not known.
wget: unable to resolve host address ‘imap.comcast.net’

[Brian Murray (brian-murray)]

The workaround, of adding "options edns0" to my /etc/resolv.conf, in bug 1811471 resolved the issue for me. That's only temporary though as /etc/resolv.conf will get overwritten.

[Brian Murray (brian-murray)]

I installed the version of systemd from -updates for Ubuntu 18.04 and Ubuntu 18.10, with those versions of systemd I was no longer able to recreate the problem regarding resolving imap.comcast.net. As such I am marking this bug as a duplicate of bug 1811471.

[Markus Alexander Kuppe (c-launchpad-net-lemmster-de)]

The same symptons appear for "emea.mail.microsoft.com" even with systemd 237-3ubuntu10.21 from -updates.