No external network access for VMs

[Michael Gratton]

VMs under Boxes no longer have external network access - there seems to be a private internal network that is functioning fine, but when the host is connected to a network it is not possible to access this external network from VMs. Eg. It is possible to resolve DNS names, but it is not possible to ping external hosts from a VM when the host connected to the Internet.

This is a regression from zesty.

Looking at virt-manager, qemu:///system has a NAT network defined using virbr0, but qemu:///session (which Boxes uses) has no network, and it is not possible to create one - attempting to do so results in a "Operation not permitted" error:

> Error creating virtual network: error creating bridge interface virbr1: Operation not permitted
>
> Traceback (most recent call last):
> File "/usr/share/virt-manager/virtManager/asyncjob.py", line 88, in cb_wrapper
> callback(asyncjob, *args, **kwargs)
> File "/usr/share/virt-manager/virtManager/createnet.py", line 746, in _async_net_create
> net.install()
> File "/usr/share/virt-manager/virtinst/network.py", line 262, in install
> net.create()
> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2892, in create
> if ret == -1: raise libvirtError ('virNetworkCreate() failed', net=self)
> libvirtError: error creating bridge interface virbr1: Operation not permitted

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: gnome-boxes 3.26.1-1
ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
Uname: Linux 4.13.0-16-generic x86_64
ApportVersion: 2.20.7-0ubuntu3
Architecture: amd64
CurrentDesktop: GNOME
Date: Wed Oct 18 15:01:42 2017
InstallationDate: Installed on 2015-07-22 (819 days ago)
InstallationMedia: Ubuntu-GNOME 15.04 "Vivid Vervet" - Release amd64 (20150422)
SourcePackage: gnome-boxes
UpgradeStatus: No upgrade log present (probably fresh install)

[Michael Gratton (mjog)]

Added libvirt since I can't add a NAT network to the qemu:///session using virt-manager, either.

[Christian Ehrhardt  (paelzer)]

Hi,
thanks Michael for the report.
Looking into this from the libvirt POV.

This is - on the virtualization side - a known limitation of qemu:///session
Here an example to show the basics.

I first started with a test to confirm/debug the issue

test xml describing a most basic nat network:

# cat testnet.xml
<network>
  <name>testnet</name>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='testnetbr0' stp='on' delay='0'/>
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.123.2' end='192.168.123.254'/>
    </dhcp>
  </ip>
</network>

That works fine in a normal lifecycle in qemu:///system.
 $ virsh -c qemu:///system net-destroy testnet
 $ virsh -c qemu:///system net-start testnet
 $ virsh -c qemu:///system net-destroy testnet
 $ virsh -c qemu:///system net-undefine testnet

It also works fine if running on qemu:///session as root, but if being a normal user that fails on the "net-start" action. The reason for this is a lack of permissions of the normal user to e.g. define the bridge. In detail you see like:
 $ virsh -c qemu:///session net-start testnet
 error: Failed to start network testnet
 error: error creating bridge interface testnetbr0: Operation not permitted

Now there is a way to resolve that usually in the form of "/usr/lib/qemu/qemu-bridge-helper".
This is a special tool provided, but not further enabled/preconfigured mostly for:
a) being an uncommon case
b) security concerns
So an admin (or another program using it) has to it set up as needed

I wonder if gnome-boxes did something about/with it before which now doesn't work - but if so I don't know what - sorry.

If this is an issue in libvirt I'd need more details what worked before and failed now.
Maybe you or some gnome people can uncover that?

Some references on the general topic:
https://wiki.qemu.org/Features/HelperNetworking
http://jonaspfannschmidt.com/libvirt_session.html
http://isonprojects.com/qemu-bridge-network-in-ubuntu-14-04/

[Michael Gratton (mjog)]

Hey, thanks for the pointers, I've been looking into this a bit and performed the following:

- Set `bridge_helper` correctly in `/etc/libvirt/qemu.conf` - it was commented out and pointing to the wrong binary
- Made qemu-bridge-helper set-uid
- Created an appropriate /etc/qemu/bridge.conf
- Restarted all the services

I'm getting a different error message now, at least.

As far as I can tell, gnome-boxes is just a libvirt client, so it just expects a working installation.

[Michael Gratton (mjog)]

I managed to get this working in the end. Turns out that despite using the libvirt/qemu user session for the VMs, gnome-boxes actually uses the *system* libvirt network named "default" for networking so it works "out of the box". In reality, I had to delete that network and re-create it using virt-manager before it actually started bridging traffic to my physical connection.

See this blog post and the comments: http://blog.wikichoon.com/2016/01/qemusystem-vs-qemusession.html for more information.

So to fix this, at a minimum gnome-boxes should depend on libvirt-daemon-system so that the "default" network is running and attendent virbr0 interface is created at startup, and whichever libvirt package is responsible for creating that network needs to ensure it bridges to all physical devices by default.

[Christian Ehrhardt  (paelzer)]

Hi,
the dependency to libvirt atm is only
-> libvirt-daemon (Depend) -> libvirt-daemon-system (suggest)
And if that would be all that would be needed I'd expect that it could be changed in gnome-boxes.

But I wonder about "whichever libvirt package is responsible for creating that network needs to ensure it bridges to all physical devices by default.".
The default configuration is to have the default network as "nat". Which means the guests can reach out and since the host has a device to the guest network it can reach the guests.
It will give you external network access - if that is all you need, fine - no change in libvirt will be needed.
If you actually meant real "bridged to all physical devices" that is unlikely to be the default as it is a too invasive change to the networking on a package install.
For now I assume you "just" need external network which you will have - with that the libvirt part of this bug is "invalid" as no change is needed - Let me know if I misunderstood this part of the request.

[Michael Gratton (mjog)]

Apologies to re-open this for libvirt again, but I just had to go through the same song and dance again after upgrading to 18.04 and fixing the problem this required making some changes to the apparmour profile for libvirt. Will attach a diff of the changes in a moment.

[Michael Gratton (mjog)]

The changes to the apparmour profile aren't minimal, they could probably be tightened up, but I'm not very familiar with it.

I also needed to "sudo chmod u+s /usr/lib/qemu/qemu-bridge-helper" again, as well.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Changes needed to /etc/qemu.conf" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Christian Ehrhardt  (paelzer)]

Hi Michael,
thank you for the discussion, no need to apologize - I can see your troubles and want to help.
But lets dissect the different aspects of it.

TL;DR:
- path in qemu.conf should not matter
- need to set-uid manually is wanted for security reasons
- apparmor fixes exist, please chime in on bug 1754871 [2] if you think 18.04 should get that

1. apparmor to be able to use ubuntu-bridge-helper. I pushed a change upstream [3] for bug 1754871 already, that is similar to yours but more limited to what you actually need to get it working.
Although this was done at a low prio and my intention was to only pick this up with Ubuntu 18.10 and I was not yet planning to backport the fix.
The reason is that it is a config file change everybody can do and the number of use cases depending on it was considered very low.
Although I think I could be convinced this is used more often than I thought to push the fix to the apparmor profile at least to all >= Ubuntu 18.04.

I'd ask you to post on bug 1754871 if you really think we should make the apparmor change generally available. Since 18.04 is an LTS I think that would be fair.

2. set-uid
qemu-bridge-helper is a bit of an ugly case, qemu.bridge-helper is considered "unwanted but sometimes needed" at best I feel. In bug 1754871 we outlined and referred to a bunch of reasons why it won'd be delivered as set-uid (not doing again here).
Therefore you'd still need to add the setuid as it was considered to insecure to deliver it with setuid.

3. path in qemu.conf
I think changing the path for qemu-bridge-helper in qemu.conf to be correct should be "ok", but I actually think it is not needed.
At build time libvirt detects where qemu-bridge-helper is on a system (as it might differ per distribution) and it does:
...
checking for sys/inotify.h... yes
checking for qemu-bridge-helper... /usr/lib/qemu/qemu-bridge-helper
checking for xen_vm_start in -lxenserver... no
...
So if nothing is set in the config it should use this path.
The path you see commented out in the config file is just an example and that way because of [1] is what it is upstream.

Summary:
- I think #1 is covered by the bug I referred.
- #2 is intentionally staying the way it is.
- And #3 should not be an issue - it would be great if you can double-check that.

If you agree we can set this bug here back to invalid and you can chime in on [2] to push this to Bionic as well.

[1]: https://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/qemu/qemu.conf;hb=HEAD
[2]: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1754871
[3]: https://libvirt.org/git/?p=libvirt.git;a=commit;h=6a9bdf3f25fb3941d587b3f2877b36e4412d6762

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Thomas Schweikle (tps)]

Same problem again: guests can resolve internet addresses, but are unable to access them:

# host google.com
google.com has address 172.217.20.238
google.com has IPv6 address 2a00:1450:4016:801::200e
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 10 aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
google.com mail is handled by 40 alt3.aspmx.l.google.com.

# ping google.com
PING google.com (172.217.20.238): 56 data bytes
^C
--- google.com ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

iptables is set as expected:
# iptables-save
# Generated by iptables-save v1.8.5 on Fri Mar 26 13:03:26 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:LIBVIRT_INP - [0:0]
:LIBVIRT_OUT - [0:0]
:LIBVIRT_FWO - [0:0]
:LIBVIRT_FWI - [0:0]
:LIBVIRT_FWX - [0:0]
-A INPUT -j LIBVIRT_INP
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_INP -i virbr8 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr8 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr8 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr8 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr1 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr1 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr1 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr8 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr8 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr8 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr8 -p tcp -m tcp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr1 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr1 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr1 -p tcp -m tcp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT
-A LIBVIRT_FWO -s 172.19.18.0/24 -i virbr8 -j ACCEPT
-A LIBVIRT_FWO -i virbr8 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -i virbr1 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -s 172.19.10.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWI -d 172.19.18.0/24 -o virbr8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr8 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWI -o virbr1 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWI -d 172.19.10.0/24 -o virbr0 -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX...

[Paride Legovini (paride)]

Speaking specifically of the libvirt task: in comment #12 Christian dug into this issue and concluded that this is not a libvirt bug, proposing to set it to Invalid and focus on LP: #1754871, which is now Fix Released.

I see no elements here pointing to a libvirt bug, but as I can't fully rule it out I'm setting the libvirt task status to Incomplete for the moment. If you think we do have a libvirt bug here please share your findings/reasoning, change the task status back to New and we'll look at it again. Thanks!