[2.5] Container-based KVM pods in MAAS cannot use macvlan attachments

[Mike Pontillo]

When running a KVM pod inside a LXD container, newly-composed VMs with macvlan attachments fail to start. While errors that occur during this process are not yet surfaced appropriately in MAAS (see bug #1788910), they can be found in the syslog, or when a user attempts to manually start the composed VM using virsh or virt-manager. An example error is as follows:

   342: error : virNetDevMacVLanTapOpen:477 : cannot open macvtap tap device /dev/tap11: No such file or directory

After speaking with @stgraber about this issue, it turns out this happens because, when creating a macvtap interface inside a container, macvtap creates a non-namespaced character device with a name matching /dev/tap*, whose major/minor number cannot be correlated in advance with the generated device file name. (On a non-container host, this works fine due to devtmpfs.)

It's not clear to me how this issue might be fixed, but I wanted to record it here for future reference. We cannot work around the issue in MAAS.

We could prevent such attachments from being created, but that would require knowing in advance that the KVM pod is running inside a container. Not only do we not currently have that information, but that would also make it so that MAAS would not work in the event that this bug can be fixed elsewhere in the stack.

[Andres Rodriguez (andreserl)]

I think this will be a problem for the snap

On Fri, Aug 24, 2018 at 6:00 PM Mike Pontillo <email address hidden>
wrote:

> Public bug reported:
>
> When running a KVM pod inside a LXD container, newly-composed VMs with
> macvlan attachments fail to start. While errors that occur during this
> process are not yet surfaced appropriately in MAAS (see bug #1788910),
> they can be found in the syslog, or when a user attempts to manually
> start the composed VM using virsh or virt-manager. An example error is
> as follows:
>
> 342: error : virNetDevMacVLanTapOpen:477 : cannot open macvtap tap
> device /dev/tap11: No such file or directory
>
> After speaking with @stgraber about this issue, it turns out this
> happens because, when creating a macvtap interface inside a container,
> macvtap creates a non-namespaced character device with a name matching
> /dev/tap*, whose major/minor number cannot be correlated in advance with
> the generated device file name. (On a non-container host, this works
> fine due to devtmpfs.)
>
> It's not clear to me how this issue might be fixed, but I wanted to
> record it here for future reference. We cannot work around the issue in
> MAAS.
>
> We could prevent such attachments from being created, but that would
> require knowing in advance that the KVM pod is running inside a
> container. Not only do we not currently have that information, but that
> would also make it so that MAAS would not work in the event that this
> bug can be fixed elsewhere in the stack.
>
> ** Affects: maas
> Importance: Medium
> Status: Won't Fix
>
>
> ** Tags: ux
>
> ** Changed in: maas
> Status: Triaged => Won't Fix
>
> ** Description changed:
>
> When running a KVM pod inside a LXD container, newly-composed VMs with
> macvlan attachments fail to start. While errors that occur during this
> process are not yet surfaced appropriately in MAAS (see bug #1788910),
> they can be found in the syslog, or when a user attempts to manually
> start the composed VM using virsh or virt-manager. An example error is
> as follows:
>
> - 342: error : virNetDevMacVLanTapOpen:477 : cannot open macvtap tap
> + 342: error : virNetDevMacVLanTapOpen:477 : cannot open macvtap tap
> device /dev/tap11: No such file or directory
>
> After speaking with @stgraber about this issue, it turns out this
> happens because, when creating a macvtap interface inside a container,
> macvtap creates a non-namespaced character device with a name matching
> /dev/tap*, whose major/minor number cannot be correlated in advance with
> the generated device file name. (On a non-container host, this works
> fine due to devtmpfs.)
>
> It's not clear to me how this issue might be fixed, but I wanted to
> record it here for future reference. We cannot work around the issue in
> MAAS.
> +
> + We could prevent such attachments from being created, but that would
> + require knowing in advance that the KVM pod is running inside a
> + container. Not only do we not currently have that information, but that
> + would also make it so that MAAS would not work in the event that this
> + bug can be fixed elsewhere in the stack.
>
> --...

[Mike Pontillo (mpontillo)]

The fix would need to be in libvirt, unless we could create a workaround that creates the expected /dev entries fast enough for libvirt to not notice they're not there yet. ;-)

This is where the error happens in libvirt:

https://github.com/libvirt/libvirt/blob/9d399be9496388fd7538904763bcbdf5d29f10d9/src/util/virnetdevmacvlan.c#L447

Instead, libvirt would need to look up the expected major/minor number for the tap device, and ensure it has been created before trying to open it. The major/minor number can be looked up in sysfs, such as:

    /sys/class/net/{ifname}/macvtap/tap{ifindex}/dev

[Mike Pontillo (mpontillo)]

There's another issue that would have to be fixed in the kernel, too. When you create a TAP device in a container with an ifindex that overlaps with another tap device (in another container, or on the host), you get an error back from the kernel:

    RTNETLINK answers: File exists

So the interface indexes have to be namespaced.