virsh Fails at Hot Plugging Network Device with model virtio

Bug #1739674 reported by Launchy Man
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Confirmed
Low
Unassigned

Bug Description

Hi,

I am on Ubuntu 16.04.2 LTS with libvirt-bin v1.3.1-1ubuntu10.15

When trying to hot add a NIC using virsh normally, it works.

However, if I try to add a --model virtio, I get the following error:

error: Failed to attach interface
error: internal error: unable to execute QEMU command 'getfd': No file descriptor supplied via SCM_RIGHTS

So this works:

virsh attach-interface --domain mydomain --type network --source mytestnetwork --mac "02:20:c5:8b:22:12" --config --live

but this fails:

virsh attach-interface --domain mydomain --type network --source mytestnetwork --model virtio --mac "02:20:c5:8b:22:13" --config --live

This is the exact same bug that was reported here - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1322568

It was fixed in 14.04 but it looks like the bug fix never made it to Xenial

I tested the bug fix mentioned in the link above (updating the AppArmor profile) and it works to resolve it here as well but I thought I'd open this bug report so others can benefit and the the bug can be fixed at the package level.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Launchy Man,
and thanks for your report.

So to clarify you get an error like:
  apparmor="DENIED" operation="file_receive" profile="libvirt-..." name="/dev/vhost-net" comm="qemu-system-x86" requested_mask="rw" denied_mask="rw"

The fix for this actually made it upstream in [1].
Since then the individual guest profile gets the rule "if needed".
That is even done with the default devices these days, so my guests always have this:

grep vhost /etc/apparmor.d/libvirt/$(virsh dominfo xenial-test2 | awk '/^Security label:/ {print $3}').files
  /dev/vhost-net rw,

My test system had
    <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
    </interface>

I wonder if this could be a timing issue - for example:
1. start a guest without any vhost needing device
   (rule is not added intentionally)
2. later hot-add such a device
   (error would be that it is missed to add the rule now)

To test that I started my guest without any network.
Now the guest profile has no vhost-net rule (as expected)

Then I added the userspace network like you (working, no rule change) and removed it again.
$ virsh attach-interface --domain xenial-test2 --type network --source default --mac "de:ad:be:ef:de:ad" --config --live
$ virsh detach-interface --domain xenial-test2 --type network --mac "de:ad:be:ef:de:ad" --config --live

Note that on each of those actions as I expected I saw a trigger to reload the profile on hot-add.
So we got:
[...] apparmor="STATUS" operation="profile_replace" [...] name="libvirt-2359c3ff-6003-429a-b56e-bf55ec84203b//qemu_bridge_helper" pid=7493 comm="apparmor_parser"

Note: the env obviously will have to be vhost-net capable or it will silently fall back to non-vhost.

Next I attached the mode virtio device like you did (destroyed and re-spawned the guest before that to be clean):
$ virsh attach-interface --domain xenial-test2 --type network --source default --model virtio --mac "de:ad:be:ef:de:ad" --config --live
error: Failed to attach interface
error: internal error: unable to execute QEMU command 'getfd': No file descriptor supplied via SCM_RIGHTS

So far confirmed, but great Idea where to intercept at the right time - I'd have expected that the profile reload would load a new profile with the needed rule added.
OTOH as soon as you have any vhost-net device on the normal start (most common) it will work.
So severity is low.

[1]: https://libvirt.org/git/?p=libvirt.git;a=commit;h=c7abe7448c746cf0e3a6b7fab80e083afba5d5ae

Changed in libvirt (Ubuntu):
status: New → Confirmed
importance: Undecided → Low
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Note: I now checked the stack based on newer libvirt, this is occurring up to the latest version (what we have in Artful libvirt=3.6).

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The add should be done by "get_files" which is called on create and reload mode.

A call in such a case is like:
-p 0 -r -u libvirt-ebd1d390-05b4-4078-b7a3-a6322142b516 -F /dev/net/tun

If using user net I see the net-dev appearing in the XML eventually, but on this call to virt-aa-helper it seems not to be present (yet?).

I'd need to sort out which other way virt-aa-helper could know at that point in time to add the rule correctly, but won't get to it very soon.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Note: checked against latest libvirt plus all my incoming libvirt improvements on security domain callbacks - still affected.

The generated profile on the replace triggers:
  operation="profile_replace" info="same as current profile, skipping"

Note: workaround for now if you are affected
Add
/dev/vhost-net rw,
to
/etc/apparmor.d/abstractions/libvirt-qemu
(Sad that we have local overrides for all but the guest templates, might want to add that later)

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I discussed with a few more people, to get this resolved is a major dev effort on the dynamic apparmor for libvirt/qemu - and we only have unproven ideas how to do it for now.

I made this "a sibling" of bug 1677398 which I tracked for just this already.

Sorry I can't help immediately, but thinking about it also made me think we might want a per-guest local override - I think I open a bug for that and see if I get to it sooner (as it is simpler).

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI - bug for better local overrides which would allow e.g. more granular workarounds is bug 1745114

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.