sudo service lxc-net restart does not reload dnsmasq when there is a container running

[Adam Ryczkowski]

I know, that I can assign static IP manually, using /etc/network/interfaces.

I also know, that I can read the MAC address of the LXC container (e.g. by looking for lxc.network.hwaddr entry in /var/lib/lxc/<container-name>/config and assign the IP based using entries dhcp-host=<mac-addr>,10.0.3.3 in /etc/dnsmasq.d/<some file>.

In the file /etc/default/lxc-net I read

# Uncomment the next line if you'd like to use a conf-file for the lxcbr0
# dnsmasq. For instance, you can use 'dhcp-host=mail1,10.0.3.100' to have
# container 'mail1' always get ip address 10.0.3.100.
#LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf

That would suit my needs; unfortunately doing so has no effect - at least not until the next computer reboot.

I would expect that `sudo service lxc-net restart` forces the dnsmasq to reload - unfortunately it never works.

To reproduce:
On Ubuntu Trusty 14.04 64bit,
1. install package lxc,
2. create a container (e.g. sudo lxc-create -n mycontainer -t ubuntu -- -r trusty)
3. start it (sudo lxc-start -d -n mycontainer)
4. create another container (e.g. sudo lxc-create -n my2ndcontainer -t ubuntu -- -r trusty)
5. edit /etc/default/lxc-net to uncomment the LXC_DHCP_CONFILE
6. edit /etc/lxc/dnsmasq.conf to contain a line like `dhcp-host=my2ndcontainer,10.0.3.142`
7. sudo service lxc-net restart
8. start the 2nd container (sudo lxc-start -d -n my2ndcontainer)

The steps 2 and 3 are optional.

The 2nd container never gets the ip 10.0.3.142, but it keeps the assigned dynamic IP

Walkaround 1:
Turn off the computer and test again tomorrow.

Walkaround 2 (more serious, but works only if steps 2 and 3 are skipped):

name=my2ndcontainer
sudo lxc-stop -n $name >/dev/null
sudo service lxc-net stop >/dev/null
if [ -d /sys/class/net/$internalif ]; then
   sudo brctl delbr $internalif >/dev/null #Why? See below.
fi
sudo rm /var/lib/misc/dnsmasq.$internalif.leases
sudo service lxc-net start >/dev/null
sudo lxc-start -d -n $name >/dev/null
sleep 5

Unfortunately, there is a bug (feature?) in the /etc/init/lxc-net.conf in Ubuntu 14.04 that prevents reloading the dnsmasq unless the bridge device is down for the host.

[Adam Ryczkowski (adam-ryczkowski)]

After consulting with Serge Hallyn, the original author of the upstart script that governs creation of the lxc-net bridge, I came up with the following solution:

1. Turn off all lxc containers and the lxc-net (sudo service lxc-net stop)
2. Remove (or move away) the file /etc/init/lxc-net.conf
3. Create the file /etc/init/lxc-net.conf with the following contents:

description "lxc network"
author "Serge Hallyn <email address hidden>"

start on starting lxc
stop on stopped lxc

env USE_LXC_BRIDGE="true"
env LXC_BRIDGE="lxcbr0"
env LXC_ADDR="10.0.3.1"
env LXC_NETMASK="255.255.255.0"
env LXC_NETWORK="10.0.3.0/24"
env varrun="/run/lxc"
env LXC_DOMAIN=""

pre-start script
    [ -f /etc/default/lxc ] && . /etc/default/lxc

    [ "x$USE_LXC_BRIDGE" = "xtrue" ] || { stop; exit 0; }

    use_iptables_lock="-w"
    iptables -w -L -n > /dev/null 2>&1 || use_iptables_lock=""
    cleanup() {
        # dnsmasq failed to start, clean up the bridge
        iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 67 -j ACCEPT
        iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 67 -j ACCEPT
        iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 53 -j ACCEPT
        iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 53 -j ACCEPT
        iptables $use_iptables_lock -D FORWARD -i ${LXC_BRIDGE} -j ACCEPT
        iptables $use_iptables_lock -D FORWARD -o ${LXC_BRIDGE} -j ACCEPT
        iptables $use_iptables_lock -t nat -D POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE || true
        iptables $use_iptables_lock -t mangle -D POSTROUTING -o ${LXC_BRIDGE} -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
        ifconfig ${LXC_BRIDGE} down || true
        brctl delbr ${LXC_BRIDGE} || true
    }
    if [ -d /sys/class/net/${LXC_BRIDGE} ]; then
        if [ ! -f ${varrun}/network_up ]; then
            # bridge exists, but we didn't start it
            stop;
        fi
        exit 0;
    fi

    # set up the lxc network
    brctl addbr ${LXC_BRIDGE} || { echo "Missing bridge support in kernel"; stop; exit 0; }
    echo 1 > /proc/sys/net/ipv4/ip_forward
    mkdir -p ${varrun}
    ifconfig ${LXC_BRIDGE} ${LXC_ADDR} netmask ${LXC_NETMASK} up
    iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p udp --dport 67 -j ACCEPT
    iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p tcp --dport 67 -j ACCEPT
    iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p udp --dport 53 -j ACCEPT
    iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p tcp --dport 53 -j ACCEPT
    iptables $use_iptables_lock -I FORWARD -i ${LXC_BRIDGE} -j ACCEPT
    iptables $use_iptables_lock -I FORWARD -o ${LXC_BRIDGE} -j ACCEPT
    iptables $use_iptables_lock -t nat -A POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE
    iptables $use_iptables_lock -t mangle -A POSTROUTING -o ${LXC_BRIDGE} -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill

    touch ${varrun}/network_up
end script

post-stop script
    [ -f /etc/default/lxc ] && . /etc/default/lxc
    [ -f "${varrun}/network_up" ] || exit 0;
    # if $LXC_BRIDGE has attached interfaces, don't shut it down
    ls ...

[Adam Ryczkowski (adam-ryczkowski)]

There is also a little more advanced solution, that claims to be compatible with systemd as well: https://github.com/CameronNemo/lxc-net

[Serge Hallyn (serge-hallyn)]

Thanks - the solution in your comment #1 would be good, if it was accompanied by a /etc/init/lxc-dnsmasq.conf which had 'start on started lxc-net; stop on stopping lxc-net; instance $BRIDGE"

[Adam Ryczkowski (adam-ryczkowski)]

Why would you need other instances of the service lxc-dnsmasq? The lxc-net doesn't support instances - there could be only max. one bridge at any given time, and I believe setting two dnsmasq servers on the same network interface would cause conflict.

[Serge Hallyn (serge-hallyn)]

Well, lxc-net doesn't support instances - yet :)

But you're right we dont' need that at this point. My point was that your
example didn't seem to have dnsmasq at all.

[Stéphane Graber (stgraber)]

root@dakara:~# ps aux | grep lxc-dnsm
lxc-dns+ 1551 0.0 0.0 28204 1096 ? S Oct31 0:18 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative --dhcp-range=2607:f2c0:f00f:2760::1,ra-only --listen-address 2607:f2c0:f00f:2760::1
root 1697815 0.0 0.0 11780 940 pts/33 S+ 13:14 0:00 grep --color=auto lxc-dnsm
root@dakara:~# brctl show
bridge name bridge id STP enabled interfaces
lxcbr0 8000.fe443b4e3bcb no veth3O4O1Q
       vethKVIT3V
       vethXEK7VF
root@dakara:~# restart lxc-net
lxc-net start/running
root@dakara:~# ps aux | grep lxc-dnsm
lxc-dns+ 1697873 0.0 0.0 28204 976 ? S 13:15 0:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative --dhcp-range=2607:f2c0:f00f:2760::1,ra-only --listen-address 2607:f2c0:f00f:2760::1
root 1697877 0.0 0.0 11776 940 pts/33 R+ 13:15 0:00 grep --color=auto lxc-dnsm