No external network access for VMs

Bug #1724441 reported by Michael Gratton on 2017-10-18
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
gnome-boxes (Ubuntu)
Undecided
Unassigned
libvirt (Ubuntu)
Undecided
Unassigned

Bug Description

VMs under Boxes no longer have external network access - there seems to be a private internal network that is functioning fine, but when the host is connected to a network it is not possible to access this external network from VMs. Eg. It is possible to resolve DNS names, but it is not possible to ping external hosts from a VM when the host connected to the Internet.

This is a regression from zesty.

Looking at virt-manager, qemu:///system has a NAT network defined using virbr0, but qemu:///session (which Boxes uses) has no network, and it is not possible to create one - attempting to do so results in a "Operation not permitted" error:

> Error creating virtual network: error creating bridge interface virbr1: Operation not permitted
>
> Traceback (most recent call last):
> File "/usr/share/virt-manager/virtManager/asyncjob.py", line 88, in cb_wrapper
> callback(asyncjob, *args, **kwargs)
> File "/usr/share/virt-manager/virtManager/createnet.py", line 746, in _async_net_create
> net.install()
> File "/usr/share/virt-manager/virtinst/network.py", line 262, in install
> net.create()
> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2892, in create
> if ret == -1: raise libvirtError ('virNetworkCreate() failed', net=self)
> libvirtError: error creating bridge interface virbr1: Operation not permitted

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: gnome-boxes 3.26.1-1
ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
Uname: Linux 4.13.0-16-generic x86_64
ApportVersion: 2.20.7-0ubuntu3
Architecture: amd64
CurrentDesktop: GNOME
Date: Wed Oct 18 15:01:42 2017
InstallationDate: Installed on 2015-07-22 (819 days ago)
InstallationMedia: Ubuntu-GNOME 15.04 "Vivid Vervet" - Release amd64 (20150422)
SourcePackage: gnome-boxes
UpgradeStatus: No upgrade log present (probably fresh install)

Michael Gratton (mjog) wrote :
Michael Gratton (mjog) wrote :

Added libvirt since I can't add a NAT network to the qemu:///session using virt-manager, either.

Hi,
thanks Michael for the report.
Looking into this from the libvirt POV.

This is - on the virtualization side - a known limitation of qemu:///session
Here an example to show the basics.

I first started with a test to confirm/debug the issue

test xml describing a most basic nat network:

# cat testnet.xml
<network>
  <name>testnet</name>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='testnetbr0' stp='on' delay='0'/>
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.123.2' end='192.168.123.254'/>
    </dhcp>
  </ip>
</network>

That works fine in a normal lifecycle in qemu:///system.
 $ virsh -c qemu:///system net-destroy testnet
 $ virsh -c qemu:///system net-start testnet
 $ virsh -c qemu:///system net-destroy testnet
 $ virsh -c qemu:///system net-undefine testnet

It also works fine if running on qemu:///session as root, but if being a normal user that fails on the "net-start" action. The reason for this is a lack of permissions of the normal user to e.g. define the bridge. In detail you see like:
 $ virsh -c qemu:///session net-start testnet
 error: Failed to start network testnet
 error: error creating bridge interface testnetbr0: Operation not permitted

Now there is a way to resolve that usually in the form of "/usr/lib/qemu/qemu-bridge-helper".
This is a special tool provided, but not further enabled/preconfigured mostly for:
a) being an uncommon case
b) security concerns
So an admin (or another program using it) has to it set up as needed

I wonder if gnome-boxes did something about/with it before which now doesn't work - but if so I don't know what - sorry.

If this is an issue in libvirt I'd need more details what worked before and failed now.
Maybe you or some gnome people can uncover that?

Some references on the general topic:
https://wiki.qemu.org/Features/HelperNetworking
http://jonaspfannschmidt.com/libvirt_session.html
http://isonprojects.com/qemu-bridge-network-in-ubuntu-14-04/

Changed in libvirt (Ubuntu):
status: New → Incomplete
Michael Gratton (mjog) wrote :

Hey, thanks for the pointers, I've been looking into this a bit and performed the following:

- Set `bridge_helper` correctly in `/etc/libvirt/qemu.conf` - it was commented out and pointing to the wrong binary
- Made qemu-bridge-helper set-uid
- Created an appropriate /etc/qemu/bridge.conf
- Restarted all the services

I'm getting a different error message now, at least.

As far as I can tell, gnome-boxes is just a libvirt client, so it just expects a working installation.

Michael Gratton (mjog) wrote :

I managed to get this working in the end. Turns out that despite using the libvirt/qemu user session for the VMs, gnome-boxes actually uses the *system* libvirt network named "default" for networking so it works "out of the box". In reality, I had to delete that network and re-create it using virt-manager before it actually started bridging traffic to my physical connection.

See this blog post and the comments: http://blog.wikichoon.com/2016/01/qemusystem-vs-qemusession.html for more information.

So to fix this, at a minimum gnome-boxes should depend on libvirt-daemon-system so that the "default" network is running and attendent virbr0 interface is created at startup, and whichever libvirt package is responsible for creating that network needs to ensure it bridges to all physical devices by default.

Changed in libvirt (Ubuntu):
status: Incomplete → New

Hi,
the dependency to libvirt atm is only
-> libvirt-daemon (Depend) -> libvirt-daemon-system (suggest)
And if that would be all that would be needed I'd expect that it could be changed in gnome-boxes.

But I wonder about "whichever libvirt package is responsible for creating that network needs to ensure it bridges to all physical devices by default.".
The default configuration is to have the default network as "nat". Which means the guests can reach out and since the host has a device to the guest network it can reach the guests.
It will give you external network access - if that is all you need, fine - no change in libvirt will be needed.
If you actually meant real "bridged to all physical devices" that is unlikely to be the default as it is a too invasive change to the networking on a package install.
For now I assume you "just" need external network which you will have - with that the libvirt part of this bug is "invalid" as no change is needed - Let me know if I misunderstood this part of the request.

Changed in libvirt (Ubuntu):
status: New → Invalid
Michael Gratton (mjog) on 2018-04-23
Changed in libvirt (Ubuntu):
status: Invalid → Confirmed
Michael Gratton (mjog) wrote :

Apologies to re-open this for libvirt again, but I just had to go through the same song and dance again after upgrading to 18.04 and fixing the problem this required making some changes to the apparmour profile for libvirt. Will attach a diff of the changes in a moment.

Michael Gratton (mjog) wrote :
Michael Gratton (mjog) wrote :

The changes to the apparmour profile aren't minimal, they could probably be tightened up, but I'm not very familiar with it.

I also needed to "sudo chmod u+s /usr/lib/qemu/qemu-bridge-helper" again, as well.

The attachment "Changes needed to /etc/qemu.conf" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch

Hi Michael,
thank you for the discussion, no need to apologize - I can see your troubles and want to help.
But lets dissect the different aspects of it.

TL;DR:
- path in qemu.conf should not matter
- need to set-uid manually is wanted for security reasons
- apparmor fixes exist, please chime in on bug 1754871 [2] if you think 18.04 should get that

1. apparmor to be able to use ubuntu-bridge-helper. I pushed a change upstream [3] for bug 1754871 already, that is similar to yours but more limited to what you actually need to get it working.
Although this was done at a low prio and my intention was to only pick this up with Ubuntu 18.10 and I was not yet planning to backport the fix.
The reason is that it is a config file change everybody can do and the number of use cases depending on it was considered very low.
Although I think I could be convinced this is used more often than I thought to push the fix to the apparmor profile at least to all >= Ubuntu 18.04.

I'd ask you to post on bug 1754871 if you really think we should make the apparmor change generally available. Since 18.04 is an LTS I think that would be fair.

2. set-uid
qemu-bridge-helper is a bit of an ugly case, qemu.bridge-helper is considered "unwanted but sometimes needed" at best I feel. In bug 1754871 we outlined and referred to a bunch of reasons why it won'd be delivered as set-uid (not doing again here).
Therefore you'd still need to add the setuid as it was considered to insecure to deliver it with setuid.

3. path in qemu.conf
I think changing the path for qemu-bridge-helper in qemu.conf to be correct should be "ok", but I actually think it is not needed.
At build time libvirt detects where qemu-bridge-helper is on a system (as it might differ per distribution) and it does:
...
checking for sys/inotify.h... yes
checking for qemu-bridge-helper... /usr/lib/qemu/qemu-bridge-helper
checking for xen_vm_start in -lxenserver... no
...
So if nothing is set in the config it should use this path.
The path you see commented out in the config file is just an example and that way because of [1] is what it is upstream.

Summary:
- I think #1 is covered by the bug I referred.
- #2 is intentionally staying the way it is.
- And #3 should not be an issue - it would be great if you can double-check that.

If you agree we can set this bug here back to invalid and you can chime in on [2] to push this to Bionic as well.

[1]: https://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/qemu/qemu.conf;hb=HEAD
[2]: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1754871
[3]: https://libvirt.org/git/?p=libvirt.git;a=commit;h=6a9bdf3f25fb3941d587b3f2877b36e4412d6762

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in gnome-boxes (Ubuntu):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers