Constant dns traffic for daisy.ubuntu.com

[Daniel Holth]

Watching GNetworkMonitor's network-changed signal causes constant DNS traffic.

Andy Whitcroft points out that the NETLINK_ROUTE socket set up by GNetworkMonitor will fire events every time an ARP entry appears or disappears.

Unfortunately, we currently need an additional layer of connectivity checking because checking NetworkManager's state for CONNECTED_GLOBAL is not enough to know whether we're really online. Ubuntu does not yet use the NetworkManager connectivity check [1].

The likely solution to this bug is a replacement for GNetworkMonitor in whoopsie.

1: http://bazaar.launchpad.net/~daisy-pluckers/whoopsie/trunk/view/head:/src/connectivity.c#L326

Original report follows:

Every few seconds, I see a dns query for daisy.ubuntu.com. After removing whoopsie, the traffic goes away.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stefan Lasiewski (stefanlasiewski)]

Here is the output from tcpdump showing the requests:

$ sudo tcpdump -i br0 -n | egrep "91.189.95.54|91.189.95.55"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes

12:05:18.811705 IP 192.168.1.1.53 > 192.168.1.2.59038: 29089 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:05:19.814040 IP 192.168.1.1.53 > 192.168.1.2.39628: 50822 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:05:49.160363 IP 192.168.1.1.53 > 192.168.1.2.41610: 46776 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:05:50.150565 IP 192.168.1.1.53 > 192.168.1.2.48341: 34805 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:06:02.574822 IP 192.168.1.1.53 > 192.168.1.2.34793: 27746 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:06:31.643716 IP 192.168.1.1.53 > 192.168.1.2.42548: 4290 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:06:32.646046 IP 192.168.1.1.53 > 192.168.1.2.33775: 41350 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:06:55.174117 IP 192.168.1.1.53 > 192.168.1.2.41563: 58810 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:07:24.251676 IP 192.168.1.1.53 > 192.168.1.2.55275: 16142 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:07:25.253938 IP 192.168.1.1.53 > 192.168.1.2.53597: 49367 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:08:08.040542 IP 192.168.1.1.53 > 192.168.1.2.46786: 63000 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:08:37.083645 IP 192.168.1.1.53 > 192.168.1.2.58403: 42910 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:08:38.085865 IP 192.168.1.1.53 > 192.168.1.2.54067: 19063 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:09:00.614035 IP 192.168.1.1.53 > 192.168.1.2.40992: 60850 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:09:29.691718 IP 192.168.1.1.53 > 192.168.1.2.55677: 52805 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:09:30.693931 IP 192.168.1.1.53 > 192.168.1.2.49417: 14672 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:10:13.986847 IP 192.168.1.1.53 > 192.168.1.2.35143: 41079 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:10:43.035720 IP 192.168.1.1.53 > 192.168.1.2.50395: 57540 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:10:44.038024 IP 192.168.1.1.53 > 192.168.1.2.46235: 47800 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:11:06.054218 IP 192.168.1.1.53 > 192.168.1.2.39691: 47657 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:11:35.131772 IP 192.168.1.1.53 > 192.168.1.2.48769: 19738 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:11:36.133743 IP 192.168.1.1.53 > 192.168.1.2.41880: 6446 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:12:19.345661 IP 192.168.1.1.53 > 192.168.1.2.38669: 20570 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:12:48.347665 IP 192.168.1.1.53 > 192.168.1.2.57886: 20847 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:12:49.349822 IP 192.168.1.1.53 > 192.168.1.2.49550: 5360 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:13:11.493990 IP 192.168.1.1.53 > 192.168.1.2.35797: 60789 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:13:28.540213 IP 192.168.1.1.53 > 192.168.1.2.57875: 6206 2/0/0 A 91.189.95.54, A 91.189.95.55 (66)
12:13:40.571853 IP 192.168.1.1.53 > 192.168.1.2.37527: 57599 2/0/0 A 91.189.95.55, A 91.189.95.54 (66)
12:13:41.574065 IP 192.168.1.1.53 > 192.168.1.2.44370: 3...

[Stéphane Graber (stgraber)]

A friend of mine reported the same symptoms.

When starting manually from a root shell with:
export CRASH_DB_URL=https://daisy.ubuntu.com
whoopsie -f

He's getting:
Could not get the Network Manager state: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.NetworkManager was not provided by any .service files.

Could it be that whoopsie is somehow trying to reconnect/re-resolve the host every few seconds on systems without NetworkManager or is the DBUS error unrelated?

[Evan (ev)]

I'm pretty sure this is GNetworkMonitor being rubbish.

[Scott Kitterman (kitterman)]

I manually installed whoopsie on Kubuntu 12.10 and I see this as well.

[Florian La Roche (florian-laroche)]

This also affects the current 12.04 LTS release and is really annoying:

# grep -ri daisy.ubuntu.com /etc/
/etc/init/whoopsie.conf:env CRASH_DB_URL=https://daisy.ubuntu.com

best regards,

Florian La Roche

[Florian La Roche (florian-laroche)]

Btw: This also affects the default Cloud images for Ubuntu like e.g.:
http://cloud-images.ubuntu.com/releases/precise/release/ubuntu-12.04-server-cloudimg-amd64-disk1.img or
http://cloud-images.ubuntu.com/releases/precise/release/ubuntu-12.04-server-cloudimg-amd64.tar.gz

best regards,

Florian La Roche

[Kasper Dupont (ubuntu-launchpad-feb)]

This is really problematic when debugging networking issues. It is hard to find relevant information, when all my packet dumps are being cluttered with needless DNS lookups.

[Vivien GUEANT (vivienfr)]

This also affects Ubuntu 13.04

[swmike (ubuntu-s-plass)]

I see this problem in 13.04 as well. I "dpkg --remove whoopsie" and regular (every 10 seconds or so) questions for daisy.ubuntu.com stopped.

[hackerseraph (hackerseraph-gmail)]

im seeing it every few minutes or so via wireshark

ubuntu 13.04

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Evan (ev)]

[11:52:24] <apw> ok there are a continuious stream of updates from the kernel
[11:52:37] <apw> becasue link level mappings are in the routing table, and they appear and dissapear
[11:52:39] <apw> all the time
[11:53:04] <apw> try sudo ip monitor
[15:12:24] <ev> so basically this approach is bogus then?
[15:13:46] <apw> well it needs consideration whether it is being selective enough
[15:14:14] <apw> as as a naive user i look at the raw stream it changes whenever an arp entry appears or dissappears
[15:14:30] <apw> now they mey be doing it 'righter' by trying to select specific subsets of
[15:14:42] <apw> the messages, or not, one wuold need to emit the ones they keep/drop to see
[15:14:52] <apw> it is possible they have not adapted to the new world order
[15:16:41] <ev> new world order?
[15:19:05] <apw> when the link level addresses got promoted from another separate cache into the
[15:19:13] <apw> routing table, which sped things up
[15:19:53] <apw> so ipv6 neibour discovery results, and ipv4 arp results get put in as a real 'host route' now
[15:20:01] <apw> and then removed when they time out
[15:20:09] <apw> which might mean things looking for route changes would see both
[15:20:22] <apw> of those, and think networking changed, but you would need to shove a print in there
[15:20:24] <apw> to be sure me thinks

[jean-christophe manciot (manciot-jeanchristophe)]

Uninstalled - One DNS request for daisy.ubuntu.com should be enough.

[Anders Kaseorg (andersk)]

If I might ask a stupid question, why does whoopsie think it needs to constantly monitor network connectivity when it doesn’t even have any reports to upload?

[Matthew Hall (mhall-9)]

This bug has been open two years and it's still occurring. I've noticed it during development of an open-source network security sensor I'm planning to publish in the next few weeks:

{ "source": "dns", "port_id": 0, "direction": 1, "self": 0, "length": 76, "eth_type": 2048, "smac": "50:e5:49:36:0a:db", "dmac": "78:96:84:71:ea:c0", "sip": "192.168.1.5", "dip": "192.168.1.254", "ip_protocol": 17, "ttl": 0, "l4_length": 42, "icmp_type": 255, "icmp_code": 255, "sport": 37657, "dport": 53, "dns_name": "daisy.ubuntu.com." }

It's a little unbelievable it's been open this long with no concrete plan taking shape.

[John Pyper (jpyper)]

This is a bit redonkulous. I just setup a new box using the utopic 14.10 amd64 desktop dvd image. I upgraded it from utopic to devel, which points to vivid 15.04 currently.

I didn't notice this traffic before on my LAN's DNS cache because normally I would configure the static IP info from /etc/networking/interfaces, but this time I decided to do it from within the Unity GUI with NetworkManager. Then I started seeing lots of queries for daisy on my DNS cache server.

IPv4 through NAT works, as does IPv6 using Hurricane Electric's TunnelBroker. I have blanked out any identifying IP info from the output below.

john@venus:~$ sudo tail -f /var/log/dnsmasq.log | grep -i daisy
[sudo] password for john:
Nov 25 12:54:01 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:01 dnsmasq[12968]: query[A] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:01 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:02 dnsmasq[12968]: query[A] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:02 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:04 dnsmasq[12968]: query[A] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:04 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:05 dnsmasq[12968]: query[A] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:05 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:06 dnsmasq[12968]: query[A] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:06 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:16 dnsmasq[12968]: query[A] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:16 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:17 dnsmasq[12968]: query[A] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:17 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 2001:470:-:---::--
Nov 25 12:54:32 dnsmasq[12968]: query[A] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:32 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:32 dnsmasq[12968]: query[A] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:32 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:33 dnsmasq[12968]: query[A] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:33 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:35 dnsmasq[12968]: query[A] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:35 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:36 dnsmasq[12968]: query[A] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:36 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:42 dnsmasq[12968]: query[A] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:42 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:42 dnsmasq[12968]: query[A] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:42 dnsmasq[12968]: query[AAAA] daisy.ubuntu.com from 192.168.1.--
Nov 25 12:54:43 dnsmasq[12968]: query[A] daisy.ubuntu.com from 192.168.1.--

[John Pyper (jpyper)]

Added info...

The icmp6_send events seems to coincide with the DNS queries to daisy.ubuntu.com. I'm not quite sure why the kernel is complaining about IPv6 stuff, as it is working just fine. Connecting to https://www.v6.facebook.com/ and https://ipv6.google.com/ come up with no problem using the IPv6 tunnel I mentioned in my previous post.

john@minime:~$ uname -a
Linux minime 3.16.0-25-generic #33-Ubuntu SMP Tue Nov 4 12:06:54 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

john@minime:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Vivid Vervet (development branch)
Release: 15.04
Codename: vivid

john@minime:~$ sudo tail -n 500 /var/log/syslog | grep -i icmp6
Nov 25 17:40:52 minime kernel: [43678.127112] icmp6_send: no reply to icmp error
Nov 25 17:41:03 minime kernel: [43688.373483] icmp6_send: no reply to icmp error
Nov 25 17:41:11 minime kernel: [43696.621244] icmp6_send: no reply to icmp error
Nov 25 17:41:17 minime kernel: [43703.162287] icmp6_send: no reply to icmp error
Nov 25 17:41:17 minime kernel: [43703.162310] icmp6_send: no reply to icmp error
Nov 25 17:41:33 minime kernel: [43718.375637] icmp6_send: no reply to icmp error
Nov 25 17:41:33 minime kernel: [43718.375662] icmp6_send: no reply to icmp error
Nov 25 17:41:38 minime kernel: [43723.458797] icmp6_send: no reply to icmp error
Nov 25 17:41:48 minime kernel: [43733.693169] icmp6_send: no reply to icmp error
Nov 25 17:41:48 minime kernel: [43733.693192] icmp6_send: no reply to icmp error
Nov 25 17:41:52 minime kernel: [43738.215536] icmp6_send: no reply to icmp error
Nov 25 17:42:00 minime kernel: [43745.682069] icmp6_send: no reply to icmp error
Nov 25 17:42:05 minime kernel: [43750.729201] icmp6_send: no reply to icmp error
Nov 25 17:42:05 minime kernel: [43750.729213] icmp6_send: no reply to icmp error
Nov 25 17:42:23 minime kernel: [43768.478054] icmp6_send: no reply to icmp error
Nov 25 17:42:33 minime kernel: [43778.824591] icmp6_send: no reply to icmp error

[Klaus Bielke (k-bielke)]

I did some experiments on an Ubuntu GNOME 14.04 LTS desktop standard installation. There are 4 (!) components involved in this annoying traffic:

whoopsie
network manager
dnsmasq (started by network manager)
IPv6

The repeated DNS-requests for daisy.ubuntu.com stop when de-activating anyone of these! You (as root) may use these recipes as workarounds:
Workaround #1: De-activate whoopsie:
#initctl stop whoopsie
#echo 'manual' > /etc/init/whoopsie.override

Or workaround #2: De-activate Network Manager:
#initctl stop network-manager
#echo 'manual' > /etc/init/network-manager.override
- Bring up network interface through config lines in file /etc/network/interfaces, e.g.
  iface eth0 inet dhcp
  auto eth0
#ifup eth0

Or workaround #3: Disable dnsmasq
- Edit file /etc/NetworkManager/NetworkManager.conf and change line with
  dns=dnsmasq
to
  #dns=dnsmasq
- Restart Network Manager:
# initctl restart network-manager

Or workaround #4: Disable IPv6:
#sysctl -w net/ipv6/conf/all/disable_ipv6=1
#echo 'net/ipv6/conf/all/disable_ipv6=1' >/etc/sysctl.d/10-ipv6-off

[Klaus Bielke (k-bielke)]

More experiments, results and proposed solution:

1) On a system with

  whoopsie running
  network-manager NOT running, but network interface initialized via /etc/network/interfaces
  dnsmask running with network-managers configuration for dnsmasq
  IPv6 aktivated

I see the annoying repetive DNS-traffic for daisy.ubuntu.com. One or more packets per minute. This proves network-manager itself is not the bad guy!

2) Network-manager provides a configuration to dnsmask which disabled cache (--cache-size=0). Removing this option from dnsmasq configuration reduces traffic rate to 1 packet per 10 minutes. This corresponds to the lifetime for the DNS record of daisy.ubuntu.com which is set to 600 seconds.

3) Running network-manager again and dnsmasq started from network-manager, but with the modified configuration shows same result as #2.

4) HOWTO modify configuration for dnsmasq: Network-manager provides the option --cache-size=0 as argument on command line. This cannot be overriden by config-files, but you can use a wrapper script which is called by network-manager instead of real dnsmasq. The fake dnsmask purges the option --cache-size=0 and calls the real dnsmasq.

I provide this wrapper script as attachment. Place it in /usr/local/sbin (this directory is searched before /usr/sbin), name it dnsmasq, set owner and group to root and set execute bits. Use at own risk.

Enjoy!

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "wrapper script for dnsmasq, purges option --cache-size=0" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Seth Arnold (seth-arnold)]

If you're going to bodge this with a wrapper script, a better choice would address the constant network monitoring performed in whoopsie by using the assume-online command line option.

I don't know the whoopsie codebase but based on a quick reading it looks like the only real downside to this change would be that error reports generated when offline wouldn't be sent up as soon as possible, but would wait until the next time the whoopsie daemon is restarted *and* there is network connectivity.

It feels like if someone cared enough to devote two hours to this, a better middle-ground could be found that only polled the network once an error report had been generated. It might require even more abuse of global variables, but it shouldn't be too hideous.

Thanks

[Klaus Bielke (k-bielke)]

@#22: Seth Arnold,

I agree that there must be better solution. The easiest may be just changing in network-manager the configuration for dnsmasq by omitting the option --cache-size=0.
But: I am not package maintainer or developer for whoopsie, network-manager or dnsmasq and I do not fully understand your technical hints. I am just an Ubuntu user affected by this annoying bug open for more than 2 1/2 years. I do not know why whoppsie must look at DNS every minute and why network-manager starts a caching DNS forwarder without cache. I just want get rid of this dammed traffic. As far as I can see, my wrapper script helps. I will use it until the package maintainers and developers for whoopsie, network-manager or dnsmasq provide a better solution of their choice.

[Bryan Quigley (bryanquigley)]

@mathieu-tl
You assigned this issue to yourself for network-manager. Do you see a fix coming from there?

[Mathieu Trudel-Lapierre (cyphermox)]

Yes. I've discussed this with Evan Dandrea a few times in the past, the right way to fix this is by enabling connectivity checking in NetworkManager itself, which can be done with a simple configuration file change in /etc/NetworkManager/NetworkManager.conf (see manual for NetworkManager).

The idea is to point it to a file under start.ubuntu.com which can be polled regularly by NM when connected (every five minutes by default, but this is configurable).

In the past, there has been some pushback on enabling this due to privacy concerns and concerns that the connectivity checking wasn't configurable/reliable enough. It will need to be investigated again, and discussed again on ubuntu-devel@ or somewhere else suitable for broader discussion.

I'm not planning on touching this again in the very near future, so unassigning -- please, anyone want to shepherd this to be completed, feel free to assign it to yourself and start up the dicussion again to see if we can enable connectivity checking. This may be useful for Ubuntu Touch as well.

[Jeremy Bicha (jbicha)]

The network-manager connectivity config bug is bug 997200 and I restarted the ubuntu-devel discussion yesterday.

Therefore, I'm closing the network-manager task here.