dnsmasq failed to create listening socket. Address already in use.

[José Pekkarinen]

Hi,

In a fresh installation of xenial with maas, juju and lxd, I'm trying to bootstrap juju on a lxd container. However, it returns the following error:

juju bootstrap lxd-controller lxd
ERROR cannot find network interface "lxdbr0": route ip+net: no such network interface
ERROR invalid config: route ip+net: no such network interface

lxdbr0 doesn't exists.

systemctl status lxd
● lxd.service - LXD - main daemon
   Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2016-08-04 14:05:19 BST; 12min ago
     Docs: man:lxd(1)
  Process: 19477 ExecStartPost=/usr/bin/lxd waitready --timeout=600 (code=exited, status=0/SUCCESS)
  Process: 19447 ExecStartPost=/usr/lib/lxd/profile-config (code=exited, status=0/SUCCESS)
  Process: 19434 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
 Main PID: 19446 (lxd)
    Tasks: 8
   Memory: 6.6M
      CPU: 198ms
   CGroup: /system.slice/lxd.service
           └─19446 /usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log

Aug 04 14:05:19 xenial-maas systemd[1]: Starting LXD - main daemon...
Aug 04 14:05:19 xenial-maas lxd[19446]: t=2016-08-04T14:05:19+0100 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
Aug 04 14:05:19 xenial-maas systemd[1]: Started LXD - main daemon.

From journalctl I extract the following:

-- Unit lxd-bridge.service has begun starting up.
Aug 04 14:05:18 xenial-maas named[2382]: listening on IPv4 interface lxdbr0, 10.76.187.1#53
Aug 04 14:05:18 xenial-maas systemd-udevd[19361]: Could not generate persistent MAC address for lxdbr0: No such file or directory
Aug 04 14:05:18 xenial-maas lxd-bridge.start[19326]: dnsmasq: failed to create listening socket for 10.76.187.1: Address already in use
Aug 04 14:05:18 xenial-maas dnsmasq[19391]: failed to create listening socket for 10.76.187.1: Address already in use
Aug 04 14:05:18 xenial-maas dnsmasq[19391]: FAILED to start up
Aug 04 14:05:18 xenial-maas lxd-bridge.start[19326]: Failed to setup lxd-bridge.
Aug 04 14:05:18 xenial-maas named[2382]: no longer listening on 10.76.187.1#53
Aug 04 14:05:19 xenial-maas systemd[1]: Started LXD - network bridge.
-- Subject: Unit lxd-bridge.service has finished start-up

sudo lsof -i | grep 10.76.187.1 provides empty input. dpkg-reconfigure -p medium lxd has been followed in dumb mode, all to yes, and accepting default suggestions for ipv4 and ipv6 ranges.

Version of lxd package is 2.0.3-0ubuntu1~ubuntu16.04.2.

Thanks!

José.

[Stéphane Graber (stgraber)]

This typically happens when you already have a DNS server running on your machine.

Do you have bind9 (named) installed by any chance?

If so, either remove it (if you don't use it) or configure it to only bind to the IP addresses that you want it to respond on, otherwise its default configuration conflicts with dnsmasq's DNS server running on lxdbr0 leading to the error you received.

[José Pekkarinen (koalinux)]

Thanks for the quick reply. As it seems maas is using bind9, I configured it to listen on the physical network using the following line in /etc/bind/maas/named.conf.options.inside.maas:

listen-on { 192.168.0.28; 192.168.122.1; };

And listen-on-v6 { none; }; in /etc/bind/named.conf.options

Errors in journalctl are gone, but dpkg-reconfigure -p medium lxd, still renders in a dead service and no bridge is turning up.

[Stéphane Graber (stgraber)]

What happens if you run "systemctl start lxd-bridge"?

And then the "systemctl status lxd-bridge" output.

The content of /etc/default/lxd-bridge may also be useful.

[José Pekkarinen (koalinux)]

here it goes:

● lxd-bridge.service - LXD - network bridge
   Loaded: loaded (/lib/systemd/system/lxd-bridge.service; static; vendor preset: enabled)
   Active: active (exited) since Fri 2016-08-05 11:07:08 BST; 10s ago
     Docs: man:lxd(1)
  Process: 14983 ExecStop=/usr/lib/lxd/lxd-bridge stop (code=exited, status=1/FAILURE)
  Process: 14994 ExecStart=/usr/lib/lxd/lxd-bridge.start (code=exited, status=0/SUCCESS)
 Main PID: 14994 (code=exited, status=0/SUCCESS)

Aug 05 11:07:08 xenial-maas systemd[1]: Starting LXD - network bridge...
Aug 05 11:07:08 xenial-maas systemd[1]: Started LXD - network bridge.

# WARNING: This file is generated by a debconf template!
# It is recommended to update it by using "dpkg-reconfigure -p medium lxd"

# Whether to setup a new bridge or use an existing one
USE_LXD_BRIDGE="false"

# Bridge name
# This is still used even if USE_LXD_BRIDGE is set to false
# set to an empty value to fully disable
LXD_BRIDGE="lxdbr0"

# Update the "default" LXD profile
UPDATE_PROFILE="true"

# Path to an extra dnsmasq configuration file
LXD_CONFILE=""

# DNS domain for the bridge
LXD_DOMAIN="lxd"

# IPv4
## IPv4 address (e.g. 10.0.8.1)
LXD_IPV4_ADDR="10.76.187.1"

## IPv4 netmask (e.g. 255.255.255.0)
LXD_IPV4_NETMASK="255.255.255.0"

## IPv4 network (e.g. 10.0.8.0/24)
LXD_IPV4_NETWORK="10.76.187.1/24"

## IPv4 DHCP range (e.g. 10.0.8.2,10.0.8.254)
LXD_IPV4_DHCP_RANGE="10.76.187.2,10.76.187.254"

## IPv4 DHCP number of hosts (e.g. 250)
LXD_IPV4_DHCP_MAX="252"

## NAT IPv4 traffic
LXD_IPV4_NAT="true"

# IPv6
## IPv6 address (e.g. 2001:470:b368:4242::1)
LXD_IPV6_ADDR="fdc:9a63:4482:30ca::1"

## IPv6 CIDR mask (e.g. 64)
LXD_IPV6_MASK="64"

## IPv6 network (e.g. 2001:470:b368:4242::/64)
LXD_IPV6_NETWORK="fdc:9a63:4482:30ca::1/64"

## NAT IPv6 traffic
LXD_IPV6_NAT="true"

# Run a minimal HTTP PROXY server
LXD_IPV6_PROXY="false"

[José Pekkarinen (koalinux)]

Agh... Misregard that config, it was mistaken as I was trying to use an existing bridge, that is not existing. When I passed the dpkg-reconfigure -p medium lxd, and asked for using a new bridge it went ahead, and now the bridge is visible.

7: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 06:31:7b:9d:d7:85 brd ff:ff:ff:ff:ff:ff
    inet 10.76.187.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fdc:9a63:4482:30ca::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::431:7bff:fe9d:d785/64 scope link
       valid_lft forever preferred_lft forever

juju bootstrap lxd-controller localhost
Creating Juju controller "lxd-controller" on localhost/localhost
Bootstrapping model "controller"
Starting new instance for initial controller
Launching instance
 - juju-1f3ce3-0d
Installing Juju agent on bootstrap instance
Preparing for Juju GUI 2.1.8 release installation
Waiting for address
Attempting to connect to fdc:9a63:4482:30ca:216:3eff:fe93:9ecd:22
Attempting to connect to 10.76.187.143:22
sudo: unable to resolve host juju-1f3ce3-0
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: cloud-utils
Installing package: cloud-image-utils
Installing package: tmux
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.0-beta12/juju-2.0-beta12-xenial-amd64.tgz]>
Bootstrapping Juju machine agent
Waiting for API to become available: upgrade in progress (upgrade in progress)
Waiting for API to become available: upgrade in progress (upgrade in progress)
Waiting for API to become available: upgrade in progress (upgrade in progress)
Waiting for API to become available: upgrade in progress (upgrade in progress)
Bootstrap complete, lxd-controller now available.

Thanks for your help Stephan!

José.

[Joel Johnston (joeldjohnston)]

I'd like to add that I've replicated this issue twice. The resolution of adding specific listeners to maas's bind works temporarily, but we need a way to have maas specify the listeners in bind by network address/network by default. Restarting maas resets the bind settings and the issue with lxd-bridge crops right back up. Please let me know if I'm missing something here, but this constitutes a real blocker when deploying maas and juju as far as I can see.

[Daniel Cordey (dc-b)]

Well, again a fresh install which fails to run 'lxd init'. This time, it's the dnsmasq which seems to be involved. In fact, dnsmasq enters in conflict with named (bind9). I removed the bind9 which doesn't seem to have dependencies, but it was installed by... something (who knows what).

We should probably fix these conflicts (addres binding) once and for all, since it's really bothering and stupid (this kind of problem is several years old now). Can't we examine ahead what addresses and ports are bound before attempting to create the socket ? Is there anything I can do to help work on this issue ?

dc

[José Pekkarinen (koalinux)]

It's maas itself pulling bind9 as a dependency for the dns infra it offers.