dnsmasq started too early, not getting all interfaces

[Hadmut Danisch]

Hi,

I'm still struggling with 18.04 and the move from ifupdown to netplan.

I am running a local virtual linux bridge as a network for several virtual machines and containers, which is to be serviced with dhcp and dns by dnsmasq.

Conforming to latest designs from Ubuntu the bridge is now started by netplan and configured in /etc/netplan/60-vlan0.yaml straightforward.

Since there is no ifupdown-scripts anymore, I've configured the default dnsmasq daemon in /etc/dnsmasq.d/vlan0 to offer dhcp for that bridge.

This works only when started manually.

When booting ubuntu the normal way, the bridge is correctly generated, and dnsmasq is running, but it does *not* offer DNS for the bridge and does not occupy its port 53. But just a manual systemctl restart dnsmasq.service make it run as expected, then everything is fine.

So my guess is that my configuration is correct, but dnsmasq simply started to early, i.e. before the bridge is created, and thus does not see the bridge when starting up.

There's something wrong in the dependencies of the startup configuration.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: dnsmasq 2.79-1
ProcVersionSignature: Ubuntu 4.15.0-23.25-generic 4.15.18
Uname: Linux 4.15.0-23-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: LXDE
Date: Fri Jun 15 10:59:43 2018
InstallationDate: Installed on 2018-04-30 (45 days ago)
InstallationMedia: Lubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
PackageArchitecture: all
SourcePackage: dnsmasq
UpgradeStatus: No upgrade log present (probably fresh install)

[Ryan Harper (raharper)]

In your netplan example config, keyword 'optional' means that systemd-networkd won't wait until this interface is up before declaring to systemd that the network is online. THis means dnsmasq is starting before the vlan0 interface is created.

If you remove the optional value that should ensure it is up and configured before the dnsmasq service starts.

[Hadmut Danisch (hadmut)]

So it is not possible to have an interface as optional (for other reasons beyond that example) and still have a dnsmasq running for it?

In former versions of ubuntu I had my bridges (some containing usb ethernet interfaces to bridge virtual with physical machines) configured in /etc/network/if-up.d, and started separate dnsmasq instances for every single one of these bridges.

Now there is no regular way to start a daemon like dnsmasq through ifup-scripts, since netplan does not support them. Some people try to workaround that problem with udev clauses, starting systemd services, but then it's overcomplicated and error prone.

So what is the offical way to have services like dnsmasq run for optional interfaces under 18.04?

"Don't use optional" is a workaround, not a solution to the problem. If there is no ifup-script anymore, then how are services to be started, once the interface is up (late or just sometimes)?

[Ryan Harper (raharper)]

https://netplan.io/faq#example-for-an-ifupdown-legacy-hook-for-post-uppost-down-states

You can use a networkd-dispatcherd to hook on vlan0 becoming routeable
and then start/restart the dnsmasq service;

On Fri, Jun 15, 2018 at 12:36 PM, Hadmut Danisch <email address hidden> wrote:
> So it is not possible to have an interface as optional (for other
> reasons beyond that example) and still have a dnsmasq running for it?
>
> In former versions of ubuntu I had my bridges (some containing usb
> ethernet interfaces to bridge virtual with physical machines) configured
> in /etc/network/if-up.d, and started separate dnsmasq instances for
> every single one of these bridges.
>
> Now there is no regular way to start a daemon like dnsmasq through ifup-
> scripts, since netplan does not support them. Some people try to
> workaround that problem with udev clauses, starting systemd services,
> but then it's overcomplicated and error prone.
>
> So what is the offical way to have services like dnsmasq run for
> optional interfaces under 18.04?
>
>
> "Don't use optional" is a workaround, not a solution to the problem. If there is no ifup-script anymore, then how are services to be started, once the interface is up (late or just sometimes)?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1777094
>
> Title:
> dnsmasq started too early, not getting all interfaces
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1777094/+subscriptions

[Hadmut Danisch (hadmut)]

setting optional: false does not solve the problem, a systemctl restart dnsmasq is still required.

[Ryan Harper (raharper)]

I see, interesting. The dnsmasq.service does not wait for
network-online.target;

This appears to be a duplicate of:

https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1531184

In addition to the netplan config to ensure it waits for your bridge
to come up, you need to delay dnsmasq.service until after the network
is online. You can do that with something like this:

% mkdir -p /etc/systemd/system/dnsmasq.service.d
% echo -e "[Unit]\nAfter=network-online.target\nWants=network-online.target"
| sudo tee /etc/systemd/system/dnsmasq.service.d/network-online.conf
% reboot

After rebooting, you should see that networkd runs, brings up the
interfaces, then network-online.target is reached, and lastly dnsmasq
is then started.

% journalctl -b -o short-monotonic | egrep "(Reboot|networkd|Network
is Online|Starting dnsmasq)"
[1835835.825873] b1 systemd-networkd[173]: vlan0: IPv6 successfully enabled
[1835835.827054] b1 systemd-networkd[173]: vlan0: netdev ready
[1835835.837274] b1 systemd-networkd[173]: Enumeration completed
[1835835.864747] b1 systemd-networkd[173]: eth1: Lost carrier
[1835835.871748] b1 systemd-networkd[173]: eth0: DHCPv4 address
10.8.107.77/24 via 10.8.107.1
[1835835.879184] b1 systemd-networkd[173]: eth1: IPv6 successfully disabled
[1835835.908530] b1 systemd-networkd[173]: eth1: Gained carrier
[1835835.908903] b1 systemd-networkd[173]: eth1: Configured
[1835835.909878] b1 systemd-networkd[173]: vlan0: Gained carrier
[1835835.913807] b1 systemd-networkd-wait-online[175]: managing: eth1
[1835836.288648] b1 systemd-networkd[173]: eth0: Gained IPv6LL
[1835836.289022] b1 systemd-networkd[173]: eth0: Configured
[1835836.290405] b1 systemd-networkd-wait-online[175]: managing: eth1
[1835836.290834] b1 systemd-networkd-wait-online[175]: managing: eth0
[1835836.291077] b1 systemd-networkd-wait-online[175]: ignoring: lo
[1835837.760738] b1 systemd-networkd[173]: vlan0: Gained IPv6LL
[1835837.761713] b1 systemd-networkd[173]: vlan0: Configured
[1835837.762522] b1 systemd-networkd-wait-online[175]: managing: eth1
[1835837.763413] b1 systemd-networkd-wait-online[175]: managing: eth0
[1835837.763705] b1 systemd-networkd-wait-online[175]: ignoring: lo
[1835837.764057] b1 systemd-networkd-wait-online[175]: managing: vlan0
[1835838.445945] b1 systemd[1]: Reached target Network is Online.
[1835838.496394] b1 dbus-daemon[195]: [system] Activating via systemd:
service name='org.freedesktop.hostname1'
unit='dbus-org.freedesktop.hostname1.service' requested by ':1.0'
(uid=100 pid=173 comm="/lib/systemd/systemd-networkd "
label="unconfined")
[1835838.516683] b1 systemd[1]: Starting dnsmasq - A lightweight DHCP
and caching DNS server...
On Tue, Jun 19, 2018 at 3:51 AM Hadmut Danisch <email address hidden> wrote:
>
> setting optional: false does not solve the problem, a systemctl
> restart dnsmasq is still required.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1777094
>
> Title:
> dnsmasq started too early, not getting all interfaces
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1777094/+subscriptions

[Hadmut Danisch (hadmut)]

dnsmasq _after_ networking?

Isn't it part of networking and should be running and ready for other services depending on networking?

(my guess is that Ubuntu has never defined the targets very precisely. it should distinguish between low level /interface level configuration and networking services.)

[Ryan Harper (raharper)]

On Tue, Jun 19, 2018 at 9:41 AM Hadmut Danisch <email address hidden> wrote:
>
> dnsmasq _after_ networking?
>
> Isn't it part of networking and should be running and ready for other
> services depending on networking?

Yes, but it's complicated; see below.

>
>
> (my guess is that Ubuntu has never defined the targets very precisely. it should distinguish between low level /interface level configuration and networking services.)
>

This is rather precisely defined; not by Ubuntu but by the init
system. In systemd there are many targets that indicate various state
of the system during boot[1][2]
For networking, the ones that matter are:

network-pre.target (before systemd has attempted to bring the network up)
network.target (systemd has started to bring network up; for
Artful/Bionic, this means systemd-networkd.service has started)
network-online.target ("required" networking interfaces are up,
configured, routable)

Dnsmasq itself is a networking service, so arguably, it is fine to run
at After=network.target; but network.target does not guarantee that
all of networking has been configured. Dnsmasq has two choices. 1)
it can run later, at network-online.target which means systemd is
done with creating and configuring interfaces. 2) start at
After=network.target but listen to dbus or netlink layer to react to
when a
configured interface is present and ready for it to use.

Since dnsmasq does not watch dbus or netlink for new interfaces and
when they're configured, then the only solution is to run after such
a time that the configured interfaces are ready for use by dnsmasq.

1. https://www.freedesktop.org/software/systemd/man/systemd.special.html
2. https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1777094
>
> Title:
> dnsmasq started too early, not getting all interfaces
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1777094/+subscriptions

[Launchpad Janitor (janitor)]

[Expired for netplan.io (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for dnsmasq (Ubuntu) because there has been no activity for 60 days.]

[Terrence Houlahan (f1linux)]

The problem lives in the [Unit] section in the After= directive which when modified as follows raises dnsmasq.service unbroken on boot:

sed -i 's/After=network.target/After=NetworkManager-wait-online.service/' /etc/systemd/system/multi-user.target.wants/dnsmasq.service

That's a fairly tidy, and *reproducible* fix which works on successive reboots.

[Terrence Houlahan (f1linux)]

In respect to my above post, the correct target of the sed expression should be as follows:

sed -i 's/After=network.target/After=NetworkManager-wait-online.service/' /lib/systemd/system/dnsmasq.service

Please note the fix remains nonetheless valid and ensures that dnsmasq rises up correctly on boot-

[Christian Ehrhardt  (paelzer)]

Hi Terrence, yeah and while yours is still NetworkManager specific it is still essentially the same as bug 1531184 - making this a dup to have all on one.