[SRU] dnsmasq doesn't start on boot because its interface isn't up yet

[Simon Déziel]

[Impact]
dnsmasq will fail to respond on network devices that weren't up when its
service started, thus not binding as expected.

[Test Case]
TBD

[Regression Potential]
The fix is just configuring the order of service startup, so is unlikely to create any regressions. Things to watch would be service related misbehaviors and general availability of the dnsmasq functionality.

[Fix]
Straightforward packaging fix to the service to make it delay startup
until after the network is online.

https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=774970;filename=774970-network-online.debdiff;msg=22

[Discussion]

[Original Report]
My dnsmasq instance uses "interface=br-vz0" and the interface br-vz0 is managed manually in /etc/network/interfaces.

During boot, dnsmasq is started before br-vz0 is created and this causes dnsmasq to exit:

Jan 5 08:56:16 simon-laptop dnsmasq[1008]: dnsmasq: unknown interface br-vz0
Jan 5 08:56:16 simon-laptop dnsmasq[1008]: unknown interface br-vz0
Jan 5 08:56:16 simon-laptop dnsmasq[1008]: FAILED to start up
Jan 5 08:56:17 simon-laptop NetworkManager[937]: <info> NetworkManager (version 1.0.4) is starting...
...
Jan 5 08:56:18 simon-laptop NetworkManager[937]: <info> interface-parser: parsing file /etc/network/interfaces
...
Jan 5 08:56:18 simon-laptop NetworkManager[937]: <info> found bridge ports none for br-vz0
Jan 5 08:56:18 simon-laptop NetworkManager[937]: <info> adding bridge port none to eni_ifaces
Jan 5 08:56:18 simon-laptop NetworkManager[937]: <info> management mode: unmanaged

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: dnsmasq 2.75-1
ProcVersionSignature: Ubuntu 4.3.0-5.16-generic 4.3.3
Uname: Linux 4.3.0-5-generic x86_64
ApportVersion: 2.19.3-0ubuntu2
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Jan 5 09:53:30 2016
PackageArchitecture: all
SourcePackage: dnsmasq
UpgradeStatus: No upgrade log present (probably fresh install)

[Simon Déziel (sdeziel)]

Adding the following to the [Unit] section of dnsmasq.service fixes the problem:

 After=network-online.target
 Wants=network-online.target

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Immatix (immatix)]

I believe I was having the same problem, I use the --bind-interfaces option as well as a few --interface options for different interfaces, and dnsmasq would regularly fail to start during boot because one of the interfaces wasn't present yet. Logs show dnsmasq was being started before network.target was reached.

Apr 30 22:23:19 sirius systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...
Apr 30 22:23:20 sirius dnsmasq[1152]: dnsmasq: syntax check OK.
Apr 30 22:23:20 sirius dnsmasq[1296]: dnsmasq: unknown interface trunk.3
Apr 30 22:23:20 sirius systemd[1]: dnsmasq.service: Control process exited, code=exited status=2
Apr 30 22:23:20 sirius systemd[1]: Failed to start dnsmasq - A lightweight DHCP and caching DNS server.
Apr 30 22:23:20 sirius systemd[1]: dnsmasq.service: Unit entered failed state.
Apr 30 22:23:20 sirius systemd[1]: dnsmasq.service: Failed with result 'exit-code'.
---
Apr 30 22:23:27 sirius systemd[1]: Reached target Network.

I've since added

 After=network-online.target
 Wants=network-online.target

to the [Unit] section of /lib/systemd/system/dnsmasq.service and restarted once and the problem didn't occur, but this hasn't been extensively tested.

[Max Chen (maxchen)]

yeah, Immatix! Thanks so much
added

 After=network-online.target
 Wants=network-online.target

to the [Unit] section of /lib/systemd/system/dnsmasq.service

waiting for the fixed to be release

[Robie Basak (racb)]

04:47 <rbasak> stgraber: do you have an opinion on the proposed fix for bug 1531184 please?

04:49 <stgraber> rbasak: it's probably fine. Note that anyone who actually has the dnsmasq package installed on their system is likely to run into other problems, so not a big priority for us.

04:49 <stgraber> rbasak: (dnsmasq-base is what we care about in the distro and that doesn't ship the init script)

[Sarraceno (jfhtomas)]

I have openvswitch-switch, which did required from me to apply some changes to systemd service file that seems not to be aware of virtual devices up.

So, bottom line, /etc/network/interface files are taking action in a way that smashes the rush to boot up, so dnsmasq also become a victim, and for me the solution above was not compatible.

So I had to add to dnsmasq.service the follow lines:

     Requires=sys-subsystem-net-devices-<needed device>.device
     After=sys-subsystem-net-devices-<needed device>.device
     Wants=sys-subsystem-net-devices-<needed device>.device

Got the above info from

     systemctl --full | grep subsystem

Best regards!

[John Radley (jradxl)]

I have found I needed to add
    After=network-online.target
to dnsmasq.service
to ensure my ethernet port, ens18, is available, on boot.
Using Ubuntu 16.04.2
In my case, I am using dnsmasq as per the PiHole installation where the IP is static and dhcpclient is turned off. But this is clearly not a PiHole problem as it just installs Dnsmasq.
Rather surprised I need to add this report, and that this Bug report isn't a higher Importance and not resolved in 3 months.
Surely loads of users will have this issue?
Kind Regards

[Robie Basak (racb)]

> Rather surprised I need to add this report, and that this Bug report isn't a higher Importance and not resolved in 3 months.
Surely loads of users will have this issue?

dnsmasq (as opposed to dnsmasq-base) is not in main in Ubuntu, so relies entirely on community support. It is not installed by default on Ubuntu, and is an "unusual end-user configuration" as far as I can tell. And a workaround appears available and is trivial.

If somebody wants to drive getting a fix landed, please do.

[David Britton (dpb)]

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774970

[David Britton (dpb)]

Submitted patch upstream, will revisit in a couple weeks before taking on an extra deviation in Ubuntu (since currently we do not have any delta).

[Alecz20 (alexguzu)]

I also see this issue after installing pihole which uses dnsmasq.

As a solution until we have a fix available, would it not be better to run `sudo systemctl edit dnsmasq` and add the three lines:
[Unit]
After=network-online.target
Wants=network-online.target

rather than edit /lib/systemd/system/dnsmasq.service directly?

[Bean Kylla (beankylla)]

The solution proposed by Alecz20 (alexguzu) works for me :)

[Christian Ehrhardt  (paelzer)]

Added bug 1777094 as a dup which is about the same issue (and essentially the same suggested solution).

Also related on askubuntu:
https://askubuntu.com/questions/536787/dnsmasq-starting-before-eth0-is-ready-and-therefore-not-binding

[Terrence Houlahan (f1linux)]

The reference paelzer references is related to Ubuntu 14.04. In Ubuntu 18.04, "/etc/netplan/01-network-manager-all.yaml" defaults all network device management to NetworkManager. Therefore any solution will have to be to NetworkManager's systemd service file.

My solution which I've tested and consistently raises dnsmasq up only after the interfaces are up in 18.04 is as follows:

sed -i 's/After=network.target/After=NetworkManager-wait-online.service/' /lib/systemd/system/dnsmasq.service

The 18.04 askubuntu related link is:
https://askubuntu.com/questions/1124530/systemd-starts-dnsmasq-before-starting-network-manager/1154727#1154727

[Benjamin Bach (benjaoming)]

The fix by Terrence Houlahan works for me on 18.04. Thanks!

However, I'm wondering if adding this "After" clause is a problem since the package doesn't currently depend on Network Manager? Does "NetworkManager-wait-online.service" satisfy a more generic SystemD target?

From the dnsmasq package:

Depends: netbase, dnsmasq-base,
         init-system-helpers (>= 1.18~), lsb-base (>= 3.0-6)

[Christian Ehrhardt  (paelzer)]

@Robie - I subscribed you as I wondered if this is another one for your list of bugs related to "binding to IPs available late".

[Robie Basak (racb)]

Thanks Christian. This bug was already on my list.

More details here: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

It is not obvious to me that adding After/Wants=network-online.target to the packaging for _all_ users, or anything similar, is the correct fix here. Will this delay boot for laptop users who are offline, for example?

[Benjamin Bach (benjaoming)]

Adding to my previous message: After=NetworkManager-wait-online.service will not work, as it doesn't respect for instance setups where networkd handles connections.

@Robie Basak

> Will this delay boot for laptop users who are offline, for example?

If someone has installed the dnsmasq package with the dnsmasq.service file, they already intend to run this as a network-dependent service? It's not part of a normal user's setup - I think they have dnsmasq-base, which doesn't include the service.

dnsmasq-base:

> This package contains the dnsmasq executable and documentation, but not the infrastructure required to run it as a system daemon. For that, install the dnsmasq package.

[Robie Basak (racb)]

On Tue, Sep 24, 2019 at 04:57:59PM -0000, Benjamin Bach wrote:
> @Robie Basak
>
> > Will this delay boot for laptop users who are offline, for example?
>
> If someone has installed the dnsmasq package with the dnsmasq.service
> file, they already intend to run this as a network-dependent service?
> It's not part of a normal user's setup - I think they have dnsmasq-base,
> which doesn't include the service.

They already intend to run this as a service, yes. But why do you
qualify that as "network-dependent"? dnsmasq is still useful on a laptop
only occasionally connected to the Internet, is it not? What if a user
wants it inward-facing, for example for VMs?

[Zhang Youfu (zhangyoufu)]

For my installation, I found that dnsmasq is running with -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new arguments.

/etc/dnsmasq.d/libvirt-daemon specified bind-interfaces, causing dnsmasq trying to bind to each interface on startup, which was unfortunately not available yet.

Comment out the CONFIG_DIR= directive in /etc/default/dnsmasq works for me.

[Dave Musicant (musicant)]

Terrence Houlahan's fix also works for me on 18.04 Many thanks!

[Lei Zhao (leeijaw)]

For my case, I simply cannot update the systemd to solve this problem since the address I want to bind belongs to virtual tunnel device which only gets added when there is a new tunnel connection. Oops!

[Lucas Kanashiro (lucaskanashiro)]

The Debian maintainer suggested to also add a WantedBy= entry in the Install section:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774970#32

[Sergio Durigan Junior (sergiodj)]

Xenial has reached end of standard support.