Snapd starts network-enabled snaps before networkd/netplan say the network is ready

[Pavel Stratil]

After hitting a problem with the lxd snap (see https://github.com/lxc/lxd/issues/4631) I was advised by Stéphane Graber to report a bug here. The gist of the bug is the following:

- lxd snap uses internally dnsmasq
- the 18.04 server install defaults to using systemd-resolved
- upon (re)boot, the lxd snap fails to start because of systemd-resolved not resolving shortly after boot, snapd doesn't wait for networkd/netplan to get ready.

To keep this bug report short, please kindly refer to https://github.com/lxc/lxd/issues/4631 for a complete set of logs and a "how to replicate" guide.

Thanks for looking into this,
cheers, Pavel

[Zygmunt Krynicki (zyga)]

I had a look at a simple service snap: xkcd-webserver

This is a subset of the .service unit we generate:

zyga@x240:~$ cat /etc/systemd/system/snap.xkcd-webserver.xkcd-webserver.service

[Unit]
Wants=network.target
After=snap-xkcd\x2dwebserver-16.mount network.target

As you can see we are waiting for "network.target" but not for "network-online.target". If this is about being able to wait for "network-online.target" then I think this is a new feature request that we need to discuss and design.

[Zygmunt Krynicki (zyga)]

Dear reporter, can you please clarify this aspect: Is this bug about ability to wait for network-online.target?

[Pavel Stratil (kirru)]

Hey Zygmunt, I'm not really sure, its been more then a years since I reported the bug so I don't really recall details. The gist of the bug is a chicken-and-egg problem between systemd-networkd (which expects upon start to receive a response from the 240.102.0.1 nameserver of lxd) and lxd which needs network provided by systemd-networkd. In detail, this is written out in the last 3 comments of the relevand lxd issue, please see https://github.com/lxc/lxd/issues/4631#issuecomment-397438786 and the two comments below. Basically, I'm forced to use a workaround that goes like this

kirru@az01:/etc/systemd/system$ cat snap.lxd.daemon.service.d/*.conf
[Service]
ExecStartPre=/bin/sleep 20
ExecStartPost=/usr/local/bin/lxdhostdns_start.sh
ExecStopPre=/usr/local/bin/lxdhostdns_stop.sh

kirru@az01:/usr/local/bin$ cat lxdhostdns_start.sh
#!/bin/sh
sleep 15
LXDINTERFACE=lxdfan0
LXDDOMAIN=lxd
LXDDNSIP=`ip addr show lxdfan0 | grep -Po 'inet \K[\d.]+'`

/usr/bin/systemd-resolve --interface ${LXDINTERFACE} \
                         --set-dns ${LXDDNSIP} \
                         --set-domain ${LXDDOMAIN}

kirru@az01:/usr/local/bin$ cat lxdhostdns_stop.sh
#!/bin/sh
LXDINTERFACE=lxdfan0
/usr/bin/systemd-resolve --interface ${LXDINTERFACE} --revert

This makes the whole thing work, but is a really ugly hack.