DNS hijacked in default installation by OpenDNS

[Michael Farrell]

On installing Linux Mint 14.1 Cinnamon, I discovered that my DNS configuration has been hijacked by default to use the provider "OpenDNS", in addition to my DHCP-configured nameservers.

OpenDNS by default hijacks NXDOMAIN responses, and if a hostname does not resolve, it will resolve to it's own servers:

    www.googgle.com.au has address 67.215.65.132
    132.65.215.67.in-addr.arpa domain name pointer hit-nxdomain.opendns.com.

On connecting to this IP with a web browser, the OpenDNS servers will hijack the request in order to present their own web page, displaying advertising and offering to direct users through a third-party search engine.

This also adds the potential to leak internal DNS names to the internet, and shares browsing information with several third-party providers.

In addition, OpenDNS allows any user on the sharing your public IP address (such as behind a NAT) to "register" it and adjust several settings including producing reports about DNS activity from the IP, and enabling various types of content filtering.

Because of the information disclosure and blocking functionality accessible to any user on the same network as you, I consider this to be a security issue.

Please remove OpenDNS from the default installation of Linux Mint. This is a disaster on all fronts. Half the reason I am running Linux Mint is because Canonical are introducing privacy-invading functionality in Ubuntu.

[Michael Farrell (micolous)]

I've made this a public security issue as these issues are common to all users of OpenDNS, and widely known.

[David Ulevitch (david-opendns)]

I'm the Founder and CEO of OpenDNS. I'd have to agree that this shouldn't be the default unless users are informed. We always want people to pick OpenDNS. This is one of the many reasons why we've never done deals with ISPs where it would be defaulted to "on."

Would love for it to be an option. Most of the concerns listed above are the same benefits our consumer and enterprise customers choose to use us for, which makes sense when you consider an IT person choosing to use and manage it in a concerted and deliberate way.

Thanks,
David

[brownian.walker (brownian-walker)]

This behavior is unacceptable. I never expect Linux Mint will do something like this. If the development team decided to insert this kind of code, there is no way we can be sure the system is clean and secure. I guess my search for a better distro has to continue.

[jbo5112 (jbo5112)]

OpenDNS is a terrible default for ruining a proper not found response and replacing it with spam. Imagine every Mint user thinking their computer is infected by spamware and that Linux runs of spamware. Sometimes programs are configured to respond to a DNS not found. There should at least be a choice to opt-out or preferably an opt-in during the install, without having to manually edit my config files to unmangle them.

On a related note, I don't particularly want my computer running its own nameserver. Some of the computers I'm installing Mint on are already resource starved, and I have a nice router to do all my DNS caching for me. I thought that was fairly normal for routers, and therefore fairly useless for home desktops. However, I have more pressing issues to pursue.

[Michael Farrell (micolous)]

Hi David,

Thanks very much for being on-side with this issue -- you have earned a lot of respect from me with this kind of sensible approach. My number one concern here is that a decision that has such huge implications like this needs to be in the hands of the user, and most importantly, opt-in rather than opt-out.

In situations where an enterprise deploying OpenDNS on their network, it is typically done by changing the forwarders for their LAN DNS server, and they have static IP addresses allowing them to authenticate themselves strongly with OpenDNS.

Unfortunately for Mint users, they won't have this kind of luxury. As soon as they roam onto any network with other users where there is no such agreement with OpenDNS, they will be vulnerable to these sorts of issues. When opting in to OpenDNS, users typically do it will full knowledge of these repercussions and take steps to address it by setting up an account.

I think it's very important to make sure the view that OpenDNS does not like people engaging in behaviour like Mint has is articulated somewhere on the OpenDNS website, as I fear this is not an isolated incident, or will be the last!

I am however less impressed with the lack of response from Linux Mint's developers. This is a grave privacy and security issue because of the potential for other networks users (not even network administrators) to impose restrictions on users of OpenDNS without authorisation. In addition, NXDOMAIN hijacking I would be less upset about if I knowingly opted in to the OpenDNS service, but Mint has made that choice for me.

If I were naïve to the workings of DNS, I'd suspect foul play on the part of Mint, such as a revenue sharing agreement. It wouldn't be the first time that a Mint [0] or another distributor [1] has engaged in such conduct!

Sadly this has left me with a bad experience from Mint.

Michael.

References:
[0]: https://lwn.net/Articles/471484/
[1]: https://lwn.net/Articles/428196/

[Dagni McPhee (dagnusmaximus)]

It seems like this hijacking is occuring everytime I connect to my vpn?! Please remove(or change) the default behavior. I have my dns options on my dhcp server set to 8.8.8.8. and every other machine on my network can access it fine. Mint, on the other hand, decides that it can't reach 8.8.8.8 and reduces my productivity by ignoring all visible options purposely set by me to configure my domain not found behavior. At least make this a configurabe option in the network manager rather than have it negatively affect an already configured machine.

[Thomas (t.c)]

I also dont like this as default!

But you can change it by a

sudo rm /etc/resolvconf/resolv.conf.d/tail

and reconnect to your network, thats your resolv.conf file get updatet.

[Shaun Kruger (shaun-kruger)]

I stumbled across this after installing Linux Mint 15 MATE. I had setup this system at work and suddenly found myself unable to reach hosts in the local search path. The fact that I was using this for days and then it took over the DNS settings in a way that it replaced what DHCP was handing out is entirely unacceptable.

I have taken the approach suggested in the previous comment to provide a hopefully permanent solution to this problem.

[Clement Lefebvre (clementlefebvre)]

Hi,

- We can still change things for Mint 16, so I'm eager to get to the bottom of this and find the best solution quickly to improve the next release.

- There are no commercial aspects here. We're configuring the "tail" of resolvconf in an efforts to bring an out of the box DNS fallback to all those who wouldn't have one.

- I still think we need to have a DNS fallback out of the box. Maybe using tail isn't the best way. I'd love to hear more feedback on that.

- I thought OpenDNS was the best alternative at the time. I respect David's comment and understand he would prefer us not to use the service by default, so we're likely to honor that. And I also understand that the lack of proper error codes is problematic for networking applications.

So with this in mind, I'm interested in learning more about this.

- Should we continue to use a fallback? I think we should... but please try to convince us if it's a bad idea.
- Should we switch to another service than OpenDNS? Yes, but then which one? (Note that we will not use Google here).
- Is using resolvconf's tail the best option? Tell us... (note that the UI, at least outside of Cinnamon, isn't maintained by us).

I look forward to your feedback. A discussion is also taking place on the forums at http://forums.linuxmint.com/viewtopic.php?uid=2&f=90&t=128529&start=20

As you can imagine this is a VERY touchy subject. People assume bad intentions, fud is spread, and it makes it harder to lead a constructive design session... not to mention Mint 16 is coming fast.

Hopefully we can talk calmly and learn from each others in an efforts to improve things in time for Mint 16.

[Michael Farrell (micolous)]

Why do you even need to fall back? No other operating system does this, and
any fallback will break local DNS. And some networks will block third party
DNS to deal with software that hijacks it in the way Mint does.

Just take it out, don't try to fix a problem that doesn't need fixing.

[Clement Lefebvre (clementlefebvre)]

It's a common problem. Many people seek help saying they can't connect to the Internet and when we troubleshoot we find out they're already connected, they're routing works and all they need is DNS resolution.

If we used a fallback which properly failed on wrong DNS, that wouldn't be a problem right?

As far as I know, the tail is used only if the local DNS doesn't manage to resolve the domain name, so it shouldn't break anything.

I might be wrong, please elaborate on this if I am.

[Aloysius (aloysius-w)]

@clem:
are there so many instances where the resolver doesn't work but everything else does?

[Michael Farrell (micolous)]

Clement, can you provide a statistic for those with broken DNS that went
away after pushing OpenDNS by default?

The problems people experience with their resolver would occur on **any
other operating system they use**. Is there something about Mint's
resolver configuration that is causing these problems "just for Mint"? Are
there timeout values that are much shorter?

If you look at the comment above from David @ OpenDNS, he says specifically
that he only wants OpenDNS to be used with user consent, and that they've
deliberately not done deals with ISPs to use their DNS. This should be
reason enough to cease pushing out OpenDNS.

Not to the mention you're leaking out DNS queries to a non-authoritative
third-party without telling the user about it. Big privacy oops if you're
trying to clean up after the Canonical-Amazon mess.

In some configurations of resolv.conf, the order does not matter, and it
will cycle through or randomise the name servers in use. This would break
local DNS. There is also a limit of three resolvers in use, and the
behaviour if there are more is undefined.

I'd rather Mint not break the resolver configuration of those of us with
working resolvers in order to fix those with broken ones. And really, if
they're playing with installing another operating system on their computer,
they should be able to fix their router's DNS forwarder if it's broken.

On 19 October 2013 05:27, Clement Lefebvre <email address hidden> wrote:

> It's a common problem. Many people seek help saying they can't connect
> to the Internet and when we troubleshoot we find out they're already
> connected, they're routing works and all they need is DNS resolution.
>
> If we used a fallback which properly failed on wrong DNS, that wouldn't
> be a problem right?
>
> As far as I know, the tail is used only if the local DNS doesn't manage
> to resolve the domain name, so it shouldn't break anything.
>
> I might be wrong, please elaborate on this if I am.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1133777
>
> Title:
> DNS hijacked in default installation by OpenDNS
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linuxmint/+bug/1133777/+subscriptions
>

[Clement Lefebvre (clementlefebvre)]

"If you look at the comment above from David @ OpenDNS, he says specifically
that he only wants OpenDNS to be used with user consent, and that they've
deliberately not done deals with ISPs to use their DNS. This should be
reason enough to cease pushing out OpenDNS."

--> Yes, that's reason enough. That and the fact that it doesn't fail but redirects on incorrect domain names.

"Not to the mention you're leaking out DNS queries to a non-authoritative
third-party without telling the user about it."

--> If DNS fallback is a good thing and resolvconf is reliable, I'm ok with using a 3rd party we can trust. At the time OpenDNS seemed trustworthy when it came to privacy (and that might still be true I don't know). Right now I see OpenNIC is openly guaranteeing to protect your privacy.

"In some configurations of resolv.conf, the order does not matter, and it will cycle through or randomise the name servers in use. This would break local DNS. There is also a limit of three resolvers in use, and the behaviour if there are more is undefined."

--> That's key info right there. That's the tech feedback I wanted to hear. If the order isn't always respected the whole concept fails. I'll need to verify/test that. If it's confirmed we'll no longer preset the tail for resolvconf.

"The problems people experience with their resolver would occur on **any other operating system they use**. Is there something about Mint's resolver configuration that is causing these problems "just for Mint"? Are there timeout values that are much shorter?"

--> We discussed that on the forums. It's an interesting question and lack of DNS resolution was indeed widespread at some stage. We did notice two upstream changes though around the same time... first, the introduction of resolvconf in Ubuntu, second changes in the filesystem paths for /run and /var/run. It is possible Ubuntu broke something or didn't have things properly in sync between NetworkManager and resolvconf for a couple of releases... and so it's very possible DNS fallback isn't as much an issue as it once was. The best way to tackle that would be to remove DNS fallback in Mint 16 RC and keep an eye out for feedback to see if people are still hit with the problem.

The preferred approach so far seems to be:

- Remove DNS Fallback in Mint 16 RC
- If RC feedback doesn't bring it as an issue, remove it entirely
- Otherwise review the technical implementation (order needs to be guaranteed and fallback should never be queried before the user's DNS) and if a working solution is found, look for an alternative DNS service (so far openNIC seems to be the best).

[Clement Lefebvre (clementlefebvre)]

Hi,

The decision was taken to remove DNS fallback from Mint.

- OpenDNS was identified as not being the right choice for DNS Fallback (mostly due to the way they handle errors).
- It was demonstrated that resolvconf's tail technique was not adequate technically to implement the concept of DNS Fallback.
- The need for DNS Fallback was questioned and although it did fix an issue from upstream Ubuntu at the time, it's arguable whether we do still need it nowadays.
- Privacy concerns are peripheral to this. Everything we do is to improve Mint, including adding DNS Fallback at the time, and removing it right now. ISPs know much more than OpenDNS since they don't just know what you're visiting but also your real name, your address, your phone number, your credit card details and so on. Again, I've no interest in getting into the privacy argument. I'm very well aware of the problems with PRISM and all at the moment and I understand why people are so concerned over this. The best I can say is that we're trying our best out here to make a great OS, and that means something that works great for you, not against you.

DNS Fallback isn't the first time Mint ventures outside the boundaries of what other distros do and solve something on its own. We were proud of it, I'm still proud of the way we tried and took the initiative, but I'm happy we identified the drawbacks here as well and so it's time to remove it.

Mint 16 will be the first release to come with no DNS Fallback. In prior release you can remove it with the command:

sudo rm /etc/resolvconf/resolv.conf.d/tail

I'd like to thank everyone here and on Launchpad for the feedback they gave us.

[Michael Farrell (micolous)]

Clement,

On 13/11/13 03:14, Clement Lefebvre wrote:
> - Privacy concerns are peripheral to this. Everything we do is to improve Mint, including adding DNS Fallback at the time, and removing it right now. ISPs know much more than OpenDNS since they don't just know what you're visiting but also your real name, your address, your phone number, your credit card details and so on.

Yes, but my ISP is bound by a few laws which OpenDNS are not:

- Privacy Act 1988
- Telecommunications Act 1997

I also have an avenue to chase them down with, called the
Telecommunications Industry Ombudsman. They levy fines on ISPs for even
receiving a complaint. They levy more fines against ISPs if the dispute
is not resolved in a timely manner.

OpenDNS I have no such recourse with.

My own ISP also actually respect user privacy by:

- Actively fighting against lobby groups who aimed to have access to
customer information, even when it cost them million of dollars to
defend against it.
- Only intercepting and interrupting communications when required to by
law, and not being party to back-room deals.

This is a choice I make, and why I'm happy to continue being their customer.

Mint sabotaged this by not disclosing when it was sharing information
from my computer with third parties. If Mint was made by an Australian
company, it would have violated the law.

> DNS Fallback isn't the first time Mint ventures outside the boundaries
> of what other distros do and solve something on its own. We were proud
> of it, I'm still proud of the way we tried and took the initiative

It's nice that you take initiative to do these things, but they have to
be well thought out, and consider privacy, and get user feedback. Users
need to be informed when you're sharing information from their computer
with Mint or with third parties. There was no such disclosure.

It would be why I'd also be annoyed if Mint decided to overclock my CPU
without asking. Or upload all of my photos to a web photo sharing service.

It's why the implementation of the shopping lens in Ubuntu is so terrible.

Michael

[Clement Lefebvre (clementlefebvre)]

- "sabotaged"
- "not disclosing"
- "sharing information from my computer with third parties"
- "violated the law"

You can make your points without depicting us as criminals. I'm happy to explain decisions and consider changes as long as the discussion remains constructive.

[Michael Farrell (micolous)]

Clement,

On 14/11/13 03:01, Clement Lefebvre wrote:
> You can make your points without depicting us as criminals. I'm happy to
> explain decisions and consider changes as long as the discussion remains
> constructive.

At no point did I describe you personally, or other Mint contributors,
as criminals. Civil law applies to corporations, and the specific
example where I said Mint would be breaking the law was in relation to a
__hypothetical Mint corporate entity__.

I have a problem that you decided to wilfully ignore the privacy issues
associated with this problem.

I would like in future for these things to be carefully considered, and
evaluate Mint's existing functionality in light of this issue.

As an example, VLC has functionality allowing it to download metadata
and cover art related to music files, and provide usage statistics to
the project. When you first run VLC, it asks you if you wish to to
enable these functions, and allows you to disable them entirely.

Debian's installer has a function allowing you to participate in a
package popularity survey (popcon), which on installation, you are
presented with the choice of opting in to this. It also has an
installation report which you have the option of sending to Debian's
developers for statistical purposes.

In these two examples, the user is asked first, and acknowledgement is
required to proceed, and the option to opt-out is presented.

I also have a problem with the argument of "your ISP has more on you",
because there are legal barriers that apply to my ISP but not OpenDNS
(or anyone else for that matter).

Michael