unprivileged lxc containers fails with custom bridge

[god]

Using 15.04 x86_64 with all the updates installed.
I'd like 2 containers to communicate with each other via bridge interface. For that I've created interface as follows:
/etc/systemd/network/internalbridge1.netdev:
[NetDev]
Name=ibr1
Kind=bridge

/etc/lxc/lxc-usernet:
# USERNAME TYPE BRIDGE COUNT
x veth ibr1 8

The 'x' is my username, it's first and only user configured in the system so default id map should work fine.

The container network is configured as follows:
lxc.network.type = veth
lxc.network.link = ibr1
lxc.network.flags = up
lxc.network.name = internal
lxc.network.ipv4 = 10.1.8.2/24
lxc.network.ipv4.gateway = 10.1.8.1

I've enabled the service and brought the bridge up - showing ok via 'ip a' and 'brctl show'. Also works fine with priviledged containers. However with unpriviledged containers it fails:

lxc-start -n asterisk -l debug -F --logfile lxc-user.log

[Serge Hallyn (serge-hallyn)]

Does it work if you remove the lxc.network.ipv4 and ipv4.gateway lines?

 status: incomplete

[god (humper)]

Commenting those 2 lines change nothing - same error. Is there way to figure out what exactly went wrong while creating network? The log seems way too brief about it even with debug enabled.

[Serge Hallyn (serge-hallyn)]

Indeed, on every error path lxc_user_nic should print an error message to stderr, but that doesn't end up in the log. If you simply do

lxc-start -n container_name -F

What do you see? On one failure case I see "Quota reached", for instance.

When I copy/paste your network config excerpt, it does work for me.

Is it possible that x does not have an allocation in /etc/lxc/lxc-usernet for lxcbr0, but the configuration file still has 'lxc.include = /etc/lxc/default.conf' ?

Please show the full container config and the full /etc/lxc/lxc-usernet (obfuscated if need be, but then please with annotations so we can be certain).

[god (humper)]

 cat /etc/lxc/lxc-usernet
# USERNAME TYPE BRIDGE COUNT
x veth ibr1 8
x veth xbr1 8
x veth ubr1 8

cat .local/share/lxc/asterisk/config
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: -d ubuntu -r vivid -a amd64
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.include = /usr/share/lxc/config/ubuntu.userns.conf
lxc.arch = x86_64

# Container specific configuration
lxc.include = /etc/lxc/default.conf
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
lxc.rootfs = /home/x/.local/share/lxc/asterisk/rootfs
lxc.utsname = asterisk

# Network configuration
lxc.network.type = veth
lxc.network.link = ubr1
lxc.network.flags = up
lxc.network.name = internal
#lxc.network.ipv4 = 10.1.8.2/24
#lxc.network.ipv4.gateway = 10.1.8.1

[god (humper)]

lxc-start -n asterisk -F
Quota reached.
lxc-start: start.c: lxc_spawn: 1000 failed to create configured network
lxc-start: start.c: __lxc_start: 1164 failed to spawn 'asterisk'
lxc-start: start.c: main: 344 The container failed to start.
lxc-start: start.c: main 348 Additional information can be obtained by setting the --logfile and --logpriority options.

[god (humper)]

Btw, is there a way to make logging to actually work and log everything?
Maybe add --logging-indeed-log or smth like that in addition to existing -l and --logfile?

[god (humper)]

Also, what does "Quota reached." mean? What kind of quota is that?

[Serge Hallyn (serge-hallyn)]

Quoting god (<email address hidden>):
> Also, what does "Quota reached." mean? What kind of quota is that?

The last number in /etc/lxc/lxc-usernic gives the # nics which the user
may connect to the specified bridge. The active connections are listed
in /run/lxc/nics. If Quota Reached is seen then the user already has
as many nics as allowed connected to the bridge.

[god (humper)]

Thanks for clarification! Would be kinda helpful if lxc-start could print actual numbers (X quota configured for bridge123, Y is in use). What's the upper limit on those quota numbers? Can I have 100 bridges with 4000000 interfaces in each?

[Serge Hallyn (serge-hallyn)]

Quoting god (<email address hidden>):
> Thanks for clarification! Would be kinda helpful if lxc-start could
> print actual numbers (X quota configured for bridge123, Y is in use).

Agreed, this would be a huge improvement. Would you like to re-title
this bug to turn it into a feature request for that? An 'issue' on the
github tracker at github.com/lxc/lxc may be more likely to yield fruit.

If you're able to submit your own patch that would be even better.

> What's the upper limit on those quota numbers? Can I have 100 bridges
> with 4000000 interfaces in each?

The limits would be the practical ones, for instance if you are using
a /24 you can only have 254 addresses on the bridge.

[Stéphane Graber (stgraber)]

So was this bug just about you reaching the quota and LXC failing then?

Just checking whether there's an actual bug we need to fix ASAP or if it's just about better error handling.

[Launchpad Janitor (janitor)]

[Expired for lxc (Ubuntu) because there has been no activity for 60 days.]