vif_type='tap' fails with permission error on /dev/net/tun

[Kevin Benton]

*On master branch.*

I'm working on switching the Linux Bridge plugin in Neutron to return vif_type='tap' to Nova so we can avoid the race condition of Nova and the Neutron agent trying to create the network bridge and conditional logic guessing whether or not the agent should try to add a given port to a bridge.[1]

However, Nova can't seem to boot instances with that vif_type with errors like the following:

libvirtError: internal error: process exited while connecting to monitor: 2017-03-22T16:37:48.246587Z qemu-system-x86_64: -netdev tap,ifname=tap2b1add98-31,script=,id=hostnet0: could not open /dev/net/tun: Operation not permitted

Here is the full gate run for that error above: http://logs.openstack.org/50/447150/5/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/9153647/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-03-22_16_37_48_708

I see https://review.openstack.org/#/c/448203/ was supposed to fix it, but it didn't seem to work even though 'script=' is visible in the qemu call.

1. https://review.openstack.org/#/c/447150/

[Stephen Finucane (stephenfinucane)]

What version of libvirt are you using?

[Stephen Gordon (sgordon)]

On face value this doesn't seem related to the use of script='' or not, it seems more like an ACL issue when accessing /dev/net/tun (it's not part of the cgroup_device_acl in Libvirt's qemu.conf on my Fedora machine, not sure how this works on Ubuntu). If I'm understanding correctly this is only hit now because you are introducing the use of tap devices here where they weren't used before which in turn is the reason it is trying to access /dev/net/tun?

[James Page (james-page)]

Confirmed /dev/net/tun is not part of the cgroup_device_acl on Ubuntu either.

I think most deployment tooling is actually writing out a revised qemu.conf with the right values -for example:

  https://github.com/openstack/charm-nova-compute/blob/master/templates/qemu.conf

[Matt Riedemann (mriedem)]

Sounds like this is not a nova issue, but a config issue on the host, so I've invalidated for nova and added devstack to this bug.

[Kevin Benton (kevinbenton)]

Maybe I'm missing something, but it seems like the vif_type='tap' should be a subset of the operations performed for vif_type='bridge'. The latter is currently in use and works fine.

vif_type='bridge' results in a tap device being created and plugged into a bridge. I just want the creation of the tap device without the plugging part. Why does the tap alone require more permissions?

[Nell Jerram (neil-jerram)]

Yes, I'm afraid this is expected. Please see the Calico install instructions say about this, at http://docs.projectcalico.org/master/getting-started/openstack/installation/ubuntu#compute-node-install

If vif_type 'tap' is starting to be more widely used than just Calico, it might be worthwhile to seek a better upstream solution with libvirt, or at least to (review and) document exactly what we think is needed as part of OpenStack install instructions, instead of in a Calico-specific scope.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/465405

[Kevin Benton (kevinbenton)]

Adding openstack-manuals so we can document the qemu.conf permissions required.

[Kevin Benton (kevinbenton)]

I can make the openstack-manuals required change, but I need some pointers on the best place to put that type of information.

[Alexandra Settle (alexandra-settle)]

Hey Kevin,

If I understand this correctly, for the qemu.conf permissions, I recommend adding a section here: https://docs.openstack.org/ocata/install-guide-ubuntu/nova-compute-install.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/465405
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=d1fe0e62e77b2eaf711e0b4c157dc571be9ad13e
Submitter: Jenkins
Branch: master

commit d1fe0e62e77b2eaf711e0b4c157dc571be9ad13e
Author: Kevin Benton <email address hidden>
Date: Tue May 16 22:27:58 2017 -0700

    Always setup libvirt for tap devices when using Neutron

    This logic has been tied to OVS since it was introduced in [1] and
    revised in [2]. However, many other backends may use tap devices that
    aren't related to OVS, such as Calico[3] and Linux Bridge after [4]
    merges.

    This patch just removes the dependency on OVS specifically so
    /dev/net/tun is added to cgroups whenever any Neutron backend is used.
    This is done in other deployment tools like Juju[5] so it's not
    unprecedented.

    1. Ifab268f739b004db13024633e8abeb17691b9e46
    2. Ic1da132fa421f1c70c10a319ee3239831b0f956f
    3.
    http://docs.projectcalico.org/master/getting-started/openstack/installation/ubuntu#compute-node-install
    4. I23c5faaeab69aede1fd038a36f4a0b8f928498ce
    5.
    https://github.com/openstack/charm-nova-compute/blob/2790f81ecd32d9962617c4c3126621fffdc318a0/templates/qemu.conf

    Change-Id: I075595158d8f3b5a6811c4794aa7b91912940db5
    Partial-Bug: #1675343

[Sean Dague (sdague)]

No open reviews found in this bug, unassigning. Please add a comment with active reviews before assigning an individual, or tag the bug in the gerrit review, which will do that automatically. We try not to assign bugs without patches as that discourages other folks from looking into bugs.

[Dr. Jens Harbott (j-harbott)]

The state of this bug is confusing, please update if there is still an issue with devstack.

[Launchpad Janitor (janitor)]

[Expired for devstack because there has been no activity for 60 days.]