libvirt inside lxd container cannot start virbr0 (Unable to set bridge virbr0 forward_delay: Permission denied)

Bug #1802906 reported by Brian Candler on 2018-11-12
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned

Bug Description

Ubuntu 18.04 lxd container, running on Ubuntu 18.04 host (kernel 4.15.0-38-generic)

Inside the container, I installed libvirt-bin. However it fails to start the predefined 'default' network:

root@bionic:/etc# virsh net-start default
error: Failed to start network default
error: Unable to set bridge virbr0 forward_delay: Permission denied

root@bionic:/etc# echo $?
1

root@bionic:/etc# virsh net-list
 Name State Autostart Persistent
----------------------------------------------------------

root@bionic:/etc# virsh net-list --all
 Name State Autostart Persistent
----------------------------------------------------------
 default inactive yes yes

Here is the config:

root@bionic:/etc# cat /etc/libvirt/qemu/networks/default.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh net-edit default
or other application using the libvirt API.
-->

<network>
  <name>default</name>
  <uuid>0c431cb9-7348-48df-b692-8eece268b0a0</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:11:cc:e6'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

Using "virsh net-edit default" to remove "delay='0'" does not make a difference; it gets reinserted and the same error occurs.

/var/log/syslog shows:

Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 4 seen, reloading interface list
Nov 12 11:59:11 bionic systemd-networkd[178]: virbr0-nic: Gained carrier
Nov 12 11:59:11 bionic libvirtd[225]: 2018-11-12 11:59:11.257+0000: 251: error : virNetDevBridgeSet:140 : Unable to set bridge virbr0 forward_delay: Permission denied
Nov 12 11:59:11 bionic systemd-networkd[178]: virbr0-nic: Lost carrier
Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 5 seen, reloading interface list
Nov 12 11:59:11 bionic networkd-dispatcher[212]: ERROR:Unknown interface index 5 seen even after reload
Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 5 seen, reloading interface list
Nov 12 11:59:11 bionic networkd-dispatcher[212]: ERROR:Unknown interface index 5 seen even after reload
Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 5 seen, reloading interface list
Nov 12 11:59:11 bionic networkd-dispatcher[212]: ERROR:Unknown interface index 5 seen even after reload
Nov 12 11:59:11 bionic networkd-dispatcher[212]: WARNING:Unknown index 5 seen, reloading interface list
Nov 12 11:59:11 bionic networkd-dispatcher[212]: ERROR:Unknown interface index 5 seen even after reload

Attaching strace to libvirtd, this is what I see:

...
[pid 225] <... recvmsg resumed> {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000001}, msg_namelen=12, msg_iov=[{iov_base="add@/devices/virtual/net/virbr0-nic/queues/tx-0\0ACTION=add\0DEVPATH=/devices/virtual/net/virbr0-nic/queues/tx-0\0SUBSYSTEM=queues\0"..., iov_len=16384}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_PEEK|MSG_TRUNC) = 141
[pid 250] ioctl(23, SIOCGIFINDEX, {ifr_name="virbr0-nic" <unfinished ...>
[pid 225] recvmsg(13, <unfinished ...>
[pid 250] <... ioctl resumed> , }) = 0
[pid 225] <... recvmsg resumed> {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=0x000001}, msg_namelen=12, msg_iov=[{iov_base="add@/devices/virtual/net/virbr0-nic/queues/tx-0\0ACTION=add\0DEVPATH=/devices/virtual/net/virbr0-nic/queues/tx-0\0SUBSYSTEM=queues\0"..., iov_len=16384}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 141
[pid 250] close(23 <unfinished ...>
[pid 225] poll([{fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=9, events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=17, events=POLLIN}, {fd=18, events=POLLIN}, {fd=20, events=POLLIN}], 11, 4996 <unfinished ...>
[pid 250] <... close resumed> ) = 0
[pid 250] ioctl(22, SIOCBRADDIF) = 0
[pid 250] close(22) = 0
[pid 250] socket(AF_UNIX, SOCK_DGRAM, 0) = 22
[pid 250] fcntl(22, F_GETFD) = 0
[pid 250] fcntl(22, F_SETFD, FD_CLOEXEC) = 0
[pid 250] ioctl(22, SIOCGIFFLAGS, {ifr_name="virbr0-nic", ifr_flags=IFF_BROADCAST|IFF_MULTICAST}) = 0
[pid 250] ioctl(22, SIOCSIFFLAGS, {ifr_name="virbr0-nic", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_MULTICAST}) = 0
[pid 250] close(22) = 0
[pid 250] access("/var/lib/libvirt/dnsmasq/virbr0.macs", F_OK) = -1 ENOENT (No such file or directory)
[pid 250] socket(AF_UNIX, SOCK_DGRAM, 0) = 22
[pid 250] fcntl(22, F_GETFD) = 0
[pid 250] fcntl(22, F_SETFD, FD_CLOEXEC) = 0
[pid 250] access("/sys/class/net/virbr0/bridge/forward_delay", F_OK) = 0
[pid 250] openat(AT_FDCWD, "/sys/class/net/virbr0/bridge/forward_delay", O_WRONLY|O_TRUNC) = -1 EACCES (Permission denied)
[pid 250] gettid() = 250
[pid 250] write(2, "2018-11-12 12:02:07.815+0000: 250: error : virNetDevBridgeSet:140 : Unable to set bridge virbr0 forward_delay: Permission denied"..., 129) = 129
...

WORKAROUND: "lxc config set bionic security.privileged yes && lxc restart bionic"

However, I don't think that privileged mode should be necessary. If I turn off privileged mode, I can still create and edit bridges by hand, including setting the forwarding delay:

root@bionic:~# brctl show
bridge name bridge id STP enabled interfaces
root@bionic:~# brctl addbr testbr0
root@bionic:~# brctl show
bridge name bridge id STP enabled interfaces
testbr0 8000.000000000000 no
root@bionic:~# brctl setfd testbr0 0
root@bionic:~# brctl showstp testbr0 | grep "forward delay"
 forward delay 0.00 bridge forward delay 0.00
root@bionic:~# cat /sys/class/net/testbr0/bridge/forward_delay
0
root@bionic:~# brctl setfd testbr0 1
root@bionic:~# brctl showstp testbr0 | grep "forward delay"
 forward delay 1.00 bridge forward delay 1.00
root@bionic:~# cat /sys/class/net/testbr0/bridge/forward_delay
100

However, writing to the /sys filesystem directly does not work:

root@bionic:~# echo 0 > /sys/class/net/testbr0/bridge/forward_delay
bash: /sys/class/net/testbr0/bridge/forward_delay: Permission denied
root@bionic:~#

In fact, it looks like "brctl setfd" is failing silently to access the /sys entry, as shown by strace, but is falling back to using an ioctl which succeeds.

root@bionic:~# strace -f brctl setfd testbr0 1
...
socket(AF_UNIX, SOCK_STREAM, 0) = 4
brk(NULL) = 0x55e926464000
brk(0x55e926485000) = 0x55e926485000
openat(AT_FDCWD, "/sys/class/net/testbr0/bridge/forward_delay", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EACCES (Permission denied)
ioctl(4, SIOCDEVPRIVATE, 0x7fff63a06da0) = 0
exit_group(0) = ?
+++ exited with 0 +++
root@bionic:~# echo $?
0

This suggests that the proper solution is for libvirt to do something similar.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt0:amd64 4.0.0-1ubuntu8.5
ProcVersionSignature: Ubuntu 4.15.0-38.41-generic 4.15.18
Uname: Linux 4.15.0-38-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.4
Architecture: amd64
Date: Mon Nov 12 11:44:59 2018
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)

Brian Candler (b-candler) wrote :

Hi,
to me it seems this is not a bug, but an issue with the default config.
To run virtualization in a LXD container - which by default is unprivileged for security you have to make some changes.
We will not change LXD/Libvirt defaults for that afaik, but the following is my recommendation as a container profile addition to get KVM+Libvirt running fine in a container.

config:
  boot.autostart: "true"
  linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tables,kvm
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  eth0:
    mtu: "9000"
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  kvm:
    path: /dev/kvm
    type: unix-char
  mem:
    path: /dev/mem
    type: unix-char
  tun:
    path: /dev/net/tun
    type: unix-char
name: kvm
used_by: []

You can create that with "lxc profile new kvm" and then launch those guests that need it with default+kvm profile, while leaving the other secure and unprivileged.
  $ lxc launch ubuntu-daily:c/amd64 c --profile default --profile kvm

I hope that helps to understand, but IMHO it is not a bug.

I'm marking it incomplete assuming it is not a bug as explained and hope my explanations helped you.
Let me know if you think it is actually a bug and set it back to new then.

Changed in libvirt (Ubuntu):
status: New → Incomplete
Brian Candler (b-candler) wrote :

Sorry, but I still think it's a bug.

The reason is that "brctl setfd ..." works fine in an unprivileged container, but libvirt-daemon fails. In other words: brctl shows that it *is* possible to create and manage bridges in an unprivileged container, but libvirt-daemon isn't doing it correctly.

I am currently only using libvirt to create virbr0 (for GNS3) - I'm not running kvm.

It's quite possible that anyone who wants to run kvm would require privileged mode for other reasons, but that would be a separate point.

Thanks for the clarification Brian.
I'll take a deeper look then, but FYI I'm on a trip this week - so it might take a few days more - sorry.

Changed in libvirt (Ubuntu):
status: Incomplete → New
Download full text (4.8 KiB)

Hi Brian, I now had more time to look at it - thanks for your patience and your detailed report!
The around that currently is this in libvirt doing that is like this:

 110 #if defined(HAVE_STRUCT_IFREQ) && defined(__linux__)
 111 /*
 112 * Bridge parameters can be set via sysfs on newish kernels,
 113 * or by ioctl on older kernels. Perhaps we could just use
 114 * ioctl for every kernel, but its not clear what the long
 115 * term lifespan of the ioctl interface is...
 116 */
 117 static int virNetDevBridgeSet(const char *brname,
 118 const char *paramname, /* sysfs param name */
 119 unsigned long value, /* new value */
 120 int fd, /* control socket */
 121 struct ifreq *ifr) /* pre-filled bridge name */
 122 {
 123 VIR_AUTOFREE(char *) path = NULL;
 124
 125 if (virAsprintf(&path, SYSFS_NET_DIR "%s/bridge/%s", brname, paramname) < 0)
 126 return -1;
 127
 128 if (virFileExists(path)) {
 129 char valuestr[INT_BUFSIZE_BOUND(value)];
 130 snprintf(valuestr, sizeof(valuestr), "%lu", value);
 131 if (virFileWriteStr(path, valuestr, 0) < 0) {
 132 virReportSystemError(errno,
 133 _("Unable to set bridge %s %s"), brname, paramname);
 134 return -1;
 135 }
 136 } else {
 137 unsigned long paramid;
 138 if (STREQ(paramname, "stp_state")) {
 139 paramid = BRCTL_SET_BRIDGE_STP_STATE;
 140 } else if (STREQ(paramname, "forward_delay")) {
 141 paramid = BRCTL_SET_BRIDGE_FORWARD_DELAY;
 142 } else {
 143 virReportSystemError(EINVAL,
 144 _("Unable to set bridge %s %s")...

Read more...

Brian Candler (b-candler) wrote :

Yes, it worked - as soon as the new packages were installed. Thank you!

root@bionic:~# ls
libvirt-bin_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt-clients_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt-daemon-driver-storage-rbd_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt-daemon-system_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt-daemon_4.0.0-1ubuntu8.7~ppa2_amd64.deb
libvirt0_4.0.0-1ubuntu8.7~ppa2_amd64.deb
root@bionic:~# dpkg -i *.deb
(Reading database ... 30717 files and directories currently installed.)
...
root@bionic:~# ifconfig virbr0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
        inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
        ether 52:54:00:11:cc:e6 txqueuelen 1000 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

root@bionic:~# virsh net-destroy default
Network default destroyed

root@bionic:~# virsh net-start default
Network default started

I submitted it upstream for review there, if you want pass me your full email so I can add you as "Reported-by" on the patch. You can find my email here on launchpad on my users page in case you don't want to add it to the bug :-)

Ref: https://www.redhat.com/archives/libvir-list/2018-November/msg00747.html

Note: upstream had made slight changes to the File in August, but the logical approach still was the same. Lets see what the feedback will be there.

Changed in libvirt (Ubuntu):
status: New → Confirmed

Upstreaming complete:
https://libvirt.org/git/?p=libvirt.git;a=commit;h=6aa75b94627c9ad3bf6a836cc821750979a2fe05

I already have a working backport version from our PPA Tests on Bionic before.

To hit this with virsh you'd see in an unprivileged contianer after install that it failed to start

$ apt install uvtool-libvirt
$ virsh net-list --all
 Name State Autostart Persistent
----------------------------------------------------------
 default inactive yes yes

If you try to do so you'll see your error:
$ virsh net-start default
error: Failed to start network default
error: Unable to set bridge virbr0 forward_delay: Permission denied

Right after the upgrade to the new version (or if you would have installed the new version to begin with) the fix works:

$ virsh net-list --all
 Name State Autostart Persistent
----------------------------------------------------------
 default active yes yes

If one wants he can now stop/start networks in unprivileged guest
root@b:~# virsh net-destroy default
Network default destroyed
root@b:~# virsh net-start default
Network default started

Of course without some privileges you will not be able to run full-virt KVM.
But qemu emulation based VMs work fine.

# Get an ISO
$ wget http://releases.ubuntu.com/18.04.1/ubuntu-18.04.1-live-server-amd64.iso
# Use virt manager to start a guest with that ISO
# It will auto-select Qemu-TCG mode as it can't run KVM without some privileges on the container

=> Works fine with the PPA.

I still think while nice this isn't an SRU case IMHO.
People that really need to run KVM in containers back in existing releases can just tweak the container privileges a bit - and they most likely want to to get full virtualization instead of "just" TCG-emulation.

But for upcoming releases this surely is a nice little feature to grow.

Changed in libvirt (Ubuntu Bionic):
status: New → Won't Fix
Changed in libvirt (Ubuntu Cosmic):
status: New → Won't Fix
Changed in libvirt (Ubuntu):
status: Confirmed → In Progress

To get it ahead of the merge of a newer libvirt which will take a while I'll push the change to the current Ubuntu Development release (19.04 Disco) soon.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 4.6.0-2ubuntu4

---------------
libvirt (4.6.0-2ubuntu4) disco; urgency=medium

  * debian/patches/ubuntu/lp1787405-*: Support guest dedicated Crypto
    Adapters on s390x (LP: #1787405)
  * d/p/ubuntu/lp-1802727-netdevbridge-fall-back-to-ioctl-from-sysfs.patch:
    fix libvirt bridge handling in unprivileged containers (LP: #1802906)

 -- Christian Ehrhardt <email address hidden> Fri, 09 Nov 2018 07:42:01 +0100

Changed in libvirt (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers