virsh Fails at Hot Plugging Network Device with model virtio

[Launchy Man]

Hi,

I am on Ubuntu 16.04.2 LTS with libvirt-bin v1.3.1-1ubuntu10.15

When trying to hot add a NIC using virsh normally, it works.

However, if I try to add a --model virtio, I get the following error:

error: Failed to attach interface
error: internal error: unable to execute QEMU command 'getfd': No file descriptor supplied via SCM_RIGHTS

So this works:

virsh attach-interface --domain mydomain --type network --source mytestnetwork --mac "02:20:c5:8b:22:12" --config --live

but this fails:

virsh attach-interface --domain mydomain --type network --source mytestnetwork --model virtio --mac "02:20:c5:8b:22:13" --config --live

This is the exact same bug that was reported here - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1322568

It was fixed in 14.04 but it looks like the bug fix never made it to Xenial

I tested the bug fix mentioned in the link above (updating the AppArmor profile) and it works to resolve it here as well but I thought I'd open this bug report so others can benefit and the the bug can be fixed at the package level.

[Christian Ehrhardt  (paelzer)]

Hi Launchy Man,
and thanks for your report.

So to clarify you get an error like:
  apparmor="DENIED" operation="file_receive" profile="libvirt-..." name="/dev/vhost-net" comm="qemu-system-x86" requested_mask="rw" denied_mask="rw"

The fix for this actually made it upstream in [1].
Since then the individual guest profile gets the rule "if needed".
That is even done with the default devices these days, so my guests always have this:

grep vhost /etc/apparmor.d/libvirt/$(virsh dominfo xenial-test2 | awk '/^Security label:/ {print $3}').files
  /dev/vhost-net rw,

My test system had
    <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
    </interface>

I wonder if this could be a timing issue - for example:
1. start a guest without any vhost needing device
   (rule is not added intentionally)
2. later hot-add such a device
   (error would be that it is missed to add the rule now)

To test that I started my guest without any network.
Now the guest profile has no vhost-net rule (as expected)

Then I added the userspace network like you (working, no rule change) and removed it again.
$ virsh attach-interface --domain xenial-test2 --type network --source default --mac "de:ad:be:ef:de:ad" --config --live
$ virsh detach-interface --domain xenial-test2 --type network --mac "de:ad:be:ef:de:ad" --config --live

Note that on each of those actions as I expected I saw a trigger to reload the profile on hot-add.
So we got:
[...] apparmor="STATUS" operation="profile_replace" [...] name="libvirt-2359c3ff-6003-429a-b56e-bf55ec84203b//qemu_bridge_helper" pid=7493 comm="apparmor_parser"

Note: the env obviously will have to be vhost-net capable or it will silently fall back to non-vhost.

Next I attached the mode virtio device like you did (destroyed and re-spawned the guest before that to be clean):
$ virsh attach-interface --domain xenial-test2 --type network --source default --model virtio --mac "de:ad:be:ef:de:ad" --config --live
error: Failed to attach interface
error: internal error: unable to execute QEMU command 'getfd': No file descriptor supplied via SCM_RIGHTS

So far confirmed, but great Idea where to intercept at the right time - I'd have expected that the profile reload would load a new profile with the needed rule added.
OTOH as soon as you have any vhost-net device on the normal start (most common) it will work.
So severity is low.

[1]: https://libvirt.org/git/?p=libvirt.git;a=commit;h=c7abe7448c746cf0e3a6b7fab80e083afba5d5ae

[Christian Ehrhardt  (paelzer)]

Note: I now checked the stack based on newer libvirt, this is occurring up to the latest version (what we have in Artful libvirt=3.6).

[Christian Ehrhardt  (paelzer)]

The add should be done by "get_files" which is called on create and reload mode.

A call in such a case is like:
-p 0 -r -u libvirt-ebd1d390-05b4-4078-b7a3-a6322142b516 -F /dev/net/tun

If using user net I see the net-dev appearing in the XML eventually, but on this call to virt-aa-helper it seems not to be present (yet?).

I'd need to sort out which other way virt-aa-helper could know at that point in time to add the rule correctly, but won't get to it very soon.

[Christian Ehrhardt  (paelzer)]

Note: checked against latest libvirt plus all my incoming libvirt improvements on security domain callbacks - still affected.

The generated profile on the replace triggers:
  operation="profile_replace" info="same as current profile, skipping"

Note: workaround for now if you are affected
Add
/dev/vhost-net rw,
to
/etc/apparmor.d/abstractions/libvirt-qemu
(Sad that we have local overrides for all but the guest templates, might want to add that later)

[Christian Ehrhardt  (paelzer)]

I discussed with a few more people, to get this resolved is a major dev effort on the dynamic apparmor for libvirt/qemu - and we only have unproven ideas how to do it for now.

I made this "a sibling" of bug 1677398 which I tracked for just this already.

Sorry I can't help immediately, but thinking about it also made me think we might want a per-guest local override - I think I open a bug for that and see if I get to it sooner (as it is simpler).

[Christian Ehrhardt  (paelzer)]

FYI - bug for better local overrides which would allow e.g. more granular workarounds is bug 1745114