"initctl list" shows 11974 instances of network-interface-security after two days of uptime

[Dan Kegel]

On an Ubuntu 12.04.1 system, each time you start and stop a container,
"initctl status" shows two more instances of network-interface and network-interface-security running.
The numbers do not go down after the container shuts down.
Evidently there's an interface leak in lxc-start?

Here's how I ran into this.
Hosting a small number of buildbots in
one-shot ephemeral LXC containers,
in which the LXC container is stop and started after each build,
after two days of uptime, running the commands

while true
do
   time initctl list | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 3
done

outputs

      1 wait-for-state
   6003 network-interface
  11990 network-interface-security

real 0m19.428s

      1 wait-for-state
   6004 network-interface
  11994 network-interface-security

real 0m19.271s

If I stop the buildbots, the numbers stop rising.

This broke my build.

I can work around this by rebooting every night, but I'd sure rather not.

[Serge Hallyn (serge-hallyn)]

Yup, I can reproduce this. However I believe the bug is in the init script, not in lxc. The veth interfaces are not around.

[Serge Hallyn (serge-hallyn)]

Note that the count goes up by 1 for each container. But each container has 2 veths.

My guess is that when lxc passes one of the 2 veths into the container, we need to emit a net-device-removed signal for the passed-in device.

[Serge Hallyn (serge-hallyn)]

Ah, no, both veths have a network-interface job sticking around.

Could this be seen as a kernel/udev bug, that when they veth is destroyed (and maybe even when passed to a new netns) a uevent should be sent?

[Serge Hallyn (serge-hallyn)]

This can be worked around by manually stopping the jobs, for instance:

sudo stop network-interface-security JOB=network-interface INTERFACE=vethCEDGKQ

So lowering priority (per guidelines) since there is a workaround.

[Serge Hallyn (serge-hallyn)]

I've opened bug 1065684 to handle the network-interface-security jobs. Those can just be made to go away immediately.

THe network-interface jobs are a bit more contraversial. The question is whether the kernel should emit a signal, or whether lxc should, when a veth is moved to another network namespace.

[Serge Hallyn (serge-hallyn)]

Looking at the kernel source, I believe the uevent is being sent. I think udev (or upstart-udev-bridge) may be tossing the event because the device is no longer valid.

[Serge Hallyn (serge-hallyn)]

Running 'udevadm monitor' while starting and stopping a container gives:

KERNEL[17237.641503] add /devices/virtual/net/veth7t59jJ (net)
KERNEL[17237.644300] add /devices/virtual/net/veth7t59jJ/queues/rx-0 (queues)
KERNEL[17237.644577] add /devices/virtual/net/veth7t59jJ/queues/tx-0 (queues)
KERNEL[17237.644855] add /devices/virtual/net/vethoLdAjv (net)
KERNEL[17237.644906] add /devices/virtual/net/vethoLdAjv/queues/rx-0 (queues)
KERNEL[17237.644934] add /devices/virtual/net/vethoLdAjv/queues/tx-0 (queues)
KERNEL[17237.649498] add /kernel/slab/nf_conntrack_ffff88007a720000 (slab)
KERNEL[17237.652309] add /devices/virtual/net/lo/queues/rx-0 (queues)
KERNEL[17237.652348] add /devices/virtual/net/lo/queues/tx-0 (queues)
UDEV [17237.657166] add /devices/virtual/net/lo/queues/tx-0 (queues)
UDEV [17237.659337] add /devices/virtual/net/lo/queues/rx-0 (queues)
UDEV [17237.659652] add /kernel/slab/nf_conntrack_ffff88007a720000 (slab)
UDEV [17237.679686] add /devices/virtual/net/veth7t59jJ (net)
UDEV [17237.680836] add /devices/virtual/net/veth7t59jJ/queues/rx-0 (queues)
UDEV [17237.681041] add /devices/virtual/net/veth7t59jJ/queues/tx-0 (queues)
UDEV [17237.698158] add /devices/virtual/net/vethoLdAjv (net)
UDEV [17237.699761] add /devices/virtual/net/vethoLdAjv/queues/rx-0 (queues)
UDEV [17237.699948] add /devices/virtual/net/vethoLdAjv/queues/tx-0 (queues)
KERNEL[17244.427892] remove /devices/virtual/net/eth0/queues/rx-0 (queues)
UDEV [17244.429127] remove /devices/virtual/net/eth0/queues/rx-0 (queues)
KERNEL[17244.429466] remove /devices/virtual/net/eth0/queues/tx-0 (queues)
UDEV [17244.429918] remove /devices/virtual/net/eth0/queues/tx-0 (queues)
KERNEL[17244.431075] remove /devices/virtual/net/vethoLdAjv/queues/rx-0 (queues)
UDEV [17244.431564] remove /devices/virtual/net/vethoLdAjv/queues/rx-0 (queues)
KERNEL[17244.432570] remove /devices/virtual/net/vethoLdAjv/queues/tx-0 (queues)
UDEV [17244.433942] remove /devices/virtual/net/vethoLdAjv/queues/tx-0 (queues)
KERNEL[17244.435509] remove /devices/virtual/net/vethoLdAjv (net)
UDEV [17244.436596] remove /devices/virtual/net/vethoLdAjv (net)
KERNEL[17244.440529] remove /devices/virtual/net/lo/queues/rx-0 (queues)
UDEV [17244.440979] remove /devices/virtual/net/lo/queues/rx-0 (queues)
KERNEL[17244.441098] remove /devices/virtual/net/lo/queues/tx-0 (queues)
UDEV [17244.441492] remove /devices/virtual/net/lo/queues/tx-0 (queues)
KERNEL[17244.706760] remove /kernel/slab/nf_conntrack_ffff88007a720000 (slab)
UDEV [17244.707342] remove /kernel/slab/nf_conntrack_ffff88007a720000 (slab)

Note that while vethoLdAjv has both kernel and udev add and remove, veth7t59jJ
has only the add messages.

My previous comment is likely wrong, because the *renamed* veth7t59jJ (eth0
in the container namespace) does have remove messages, and no add messages.
(My point here being that since we see those messages, for an eth0 in another
network namespace, then we should also be seeing the messages for a veth7t59jJ
while being moved to a new netns if those were being sent - the virtual 'eth0...

[Dan Kegel (dank)]

Running this script periodically seems to work around the problem. Only lightly tested.

[Dan Kegel (dank)]

Oops, that only deleted one of the jobs. This draft deletes both.

[Serge Hallyn (serge-hallyn)]

@Dan,

note that you can also compare the list of running 'network-interface'
jobs to the veth's in /sys/class/net/. For those which are not there,
you can remove the network-interface job using
sudo initctl emit net-device-removed INTERFACE=$thenicyoufound

[Dan Kegel (dank)]

Once more with feeling.

[Serge Hallyn (serge-hallyn)]

Email thread about a potential kernel patch to solve the problem: http://lists.linuxfoundation.org/pipermail/containers/2012-October/030519.html

[Dan Kegel (dank)]

Alas, that archive doesn't show attachments. For the record, is there a better archive somewhere?

[Serge Hallyn (serge-hallyn)]

Hm, I didn't know it did that. The patch wasn't even an attachment, just inline.

I'll attach the new version here. There is another archive at sf.net, but it doesn't seem to have oct 12 messages yet.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1065589

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Serge Hallyn (serge-hallyn)]

I'm running ltp against my proposed patch, and will send it to netdev (per the email thread) later today.

[Dan Kegel (dank)]

Where does this stand? A fully updated 12.04.1 system is still seeing lots of interfaces;
   2015 network-interface
   4028 network-interface-security
and toggling an lxc container up and down four times seemed to result in
one extra network-interface and four extra network-interface-security's.

[Stéphane Graber (stgraber)]

I fixed the userspace side of this in 13.04 and the 3.8 kernel contains the matching kernel fix.
I could probably backport the fix to 12.04, though it won't work until the 13.04 kernel is backported and people explicitly move to it.

[Dan Kegel (dank)]

Thanks.

[Serge Hallyn (serge-hallyn)]

(Marking fix-released for lxc per comment #19)

[penalvch (penalvch)]

Dan Kegel, would utilizing the Raring enablement stack in Precise work for you as per https://wiki.ubuntu.com/Kernel/LTSEnablementStack ?