unregister_netdevice: waiting for lo to become free
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
In Progress
|
Medium
|
Unassigned | ||
Trusty |
In Progress
|
Medium
|
Unassigned | ||
Xenial |
In Progress
|
Medium
|
Unassigned | ||
Zesty |
Won't Fix
|
Medium
|
Unassigned | ||
Artful |
Won't Fix
|
Medium
|
Unassigned | ||
Bionic |
In Progress
|
Medium
|
Unassigned |
Bug Description
This is a "continuation" of bug 1403152, as that bug has been marked "fix released" and recent reports of failure may (or may not) be a new bug. Any further reports of the problem should please be reported here instead of that bug.
--
[Impact]
When shutting down and starting containers the container network namespace may experience a dst reference counting leak which results in this message repeated in the logs:
unregister_
This can cause issues when trying to create net network namespace and thus block a user from creating new containers.
[Test Case]
See comment 16, reproducer provided at https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs. | #1 |
Changed in linux (Ubuntu): | |
status: | New → Incomplete |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #2 |
Can anyone experiencing this issue please provide details such as:
-what release are you using (trusty/
-what kernel version are you using?
-do you have specific steps to reproduce the problem?
description: | updated |
Changed in linux (Ubuntu): | |
assignee: | nobody → Dan Streetman (ddstreet) |
importance: | Undecided → Medium |
status: | Incomplete → In Progress |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : | #3 |
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
cat /etc/apt/
deb [arch=amd64] https:/
# deb-src [arch=amd64] https:/
cat /etc/apt/
# this file was generated by packages.gitlab.com for
# the repository at https:/
deb https:/
deb-src https:/
dpkg -l | grep gitlab
ii gitlab-
dpkg -l | grep docker
ii docker-engine 17.05.0~
Seems to only occur when running docker with privileged inside of gitlab-runner. On "close" of the docker container, the process fails with the following:
Aug 17 17:53:04 WIS-DOCKERPHY kernel: [ 1579.245473] unregister_
Aug 17 17:53:15 WIS-DOCKERPHY kernel: [ 1589.356182] unregister_
Aug 17 17:53:25 WIS-DOCKERPHY kernel: [ 1599.466898] unregister_
Aug 17 17:53:35 WIS-DOCKERPHY kernel: [ 1609.577597] unregister_
Aug 17 17:53:45 WIS-DOCKERPHY kernel: [ 1619.692335] unregister_
Aug 17 17:53:54 WIS-DOCKERPHY dockerd[18386]: time="2017-
Aug 17 17:53:54 WIS-DOCKERPHY dockerd[18386]: time="2017-
Aug 17 17:53:55 WIS-DOCKERPHY kernel: [ 1629.366310] docker0: port 1(veth9f03d7e) entered disabled state
Aug 17 17:53:55 WIS-DOCKERPHY kernel: [ 1629.367889] device veth9f03d7e left promiscuous mode
Aug 17 17:53:55 WIS-DOCKERPHY kernel: [ 1629.367891] docker0: port 1(veth9f03d7e) entered disabled state
Aug 17 17:53:55 WIS-DOCKERPHY kernel: [ 1629.449744] auplink[25254]: segfault at 7ffdd8ddb0d8 ip 00007facdfff1579 sp 00007ffdd8ddb0e0 error 6 in libc-2.
Aug 17 17:53:55 WIS-DOCKERPHY dockerd[18386]: time="2017-
Aug 17 17:53:55 WIS-DOCKERPHY dockerd[18386]: time="2017-
Changed in linux (Ubuntu): | |
status: | In Progress → Confirmed |
tags: | added: apport-collected xenial |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : apport information | #4 |
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 17 17:27 seq
crw-rw---- 1 root audio 116, 33 Aug 17 17:27 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
HibernationDevice: RESUME=
InstallationDate: Installed on 2016-09-20 (331 days ago)
InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
IwConfig: Error: [Errno 2] No such file or directory
MachineType: Dell Inc. PowerEdge R710
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware 1.157.11
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
Uname: Linux 4.11.0-14-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 07/23/2013
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 6.4.0
dmi.board.name: 00NH4P
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.
dmi.product.name: PowerEdge R710
dmi.sys.vendor: Dell Inc.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : CRDA.txt | #5 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : CurrentDmesg.txt | #6 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : JournalErrors.txt | #7 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : Lspci.txt | #8 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : Lsusb.txt | #9 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : ProcCpuinfo.txt | #10 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : ProcCpuinfoMinimal.txt | #11 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : ProcInterrupts.txt | #12 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : ProcModules.txt | #13 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : UdevDb.txt | #14 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andrew Reis (areis422) wrote : WifiSyslog.txt | #15 |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Fabian Holler (fh0) wrote : | #16 |
- kernel-logs.txt Edit (10.4 KiB, text/plain)
With https:/
- linux-image-
- linux-image-
On 4.11.0-14 it was much harder to reproduce. Sometimes only a warning
happened, 1x time I was able to produce an Oops with
https:/
But I don't know if the changes in the branch made it more likely to happen or
if it was only a coincidence.
The kernel message:
unregister_
shows up, some minutes later the oops and/or warning happens.
In a different scenario were the Ubuntu 16.04 servers were running multiple docker containers with Nginx or small network applications in parallel, I was also able to reproduce the kernel Oopses also on:
- linux-image-
- linux-image-
- linux-image-
- linux-image-
I haven't tried again to reproduce it with those kernels on a clean Ubuntu
installation and unfortunately didn't kept the kernel logs.
Kernel logs can be found in the attached file.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Fabian Holler (fh0) wrote : | #17 |
- kernoops-4.4.0-93.txt Edit (19.0 KiB, text/plain)
Attached are the logs for an Oops on Ubuntu 14.04 on kernel linux-image-
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Thomas Morin (tmmorin-orange) wrote : | #18 |
bug 1715660 seems related
( Lots of "waiting for lo to become free" message, then a "task ip:1358 blocked for more than 120 seconds." with a backtrace )
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Fabian Holler (fh0) wrote : | #19 |
I could not reproduce the bug with the described method with kernel 4.4.0-81-generic and neither with 4.13.0-
So the bug might have been reintroduced between 4.4.0-82 and 4.4.0-93 and 4.13 seems to contain a fix.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #20 |
Thank you for the reproducer, I have been able to reproduce this using docker-samba-loop scripts. This should help me to dig into the kernel to see where the dst leak is coming from.
description: | updated |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Charles (bityard) wrote : | #21 |
We're seeing this on our build servers. These are VMWare virtual machines which run docker containers in privileged mode. Inside the containers are Jenkins agents and a build environment. Part of the build process uses unshare to create these namespaces: ipc uts mount pid net. If you have any questions about this arrangement, I can try to provide more detail and maybe example code.
The kernel is 4.4.0-79-generic x86_64. We are using docker 17.05.0-ce.
In one case, the whole VM hung when this was encountered.
Are there any workarounds for 16.04 users beyond rebooting the host?
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Jens Elkner (jelmd) wrote : | #22 |
Since xenial we have this very annoying behavior (unregister_
[ +10.244888] unregister_
[Oct 2 17:16] INFO: task lxc-autostart:5321 blocked for more than 120 seconds.
[ +0.007147] Tainted: P OE 4.4.0-96-generic #119-Ubuntu
[ +0.006796] "echo 0 > /proc/sys/
[ +0.007913] lxc-autostart D ffff883fb99fbcb8 0 5321 1 0x00000000
[ +0.000007] ffff883fb99fbcb8 ffffffffffffffff ffff883ff2a3aa00 ffff881fed2fe200
[ +0.000004] ffff883fb99fc000 ffffffff81ef79a4 ffff881fed2fe200 00000000ffffffff
[ +0.000004] ffffffff81ef79a8 ffff883fb99fbcd0 ffffffff8183f165 ffffffff81ef79a0
[ +0.000004] Call Trace:
[ +0.000014] [<ffffffff8183f
[ +0.000005] [<ffffffff8183f
[ +0.000016] [<ffffffff81841
[ +0.000014] [<ffffffff81841
[ +0.000009] [<ffffffff8172d
[ +0.000009] [<ffffffff810a1
[ +0.000003] [<ffffffff810a1
[ +0.000007] [<ffffffff8107f
[ +0.000011] [<ffffffff81080
[ +0.000015] [<ffffffff81003
[ +0.000004] [<ffffffff81080
[ +0.000009] [<ffffffff81843
[ +4.882232] unregister_
...
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #23 |
Hi everyone,
so this is an interesting problem. There are 2 parts to this:
1. in the container being destroyed, some socket(s) remain open for a period of time, which prevents the container from fully exits until all its sockets have exited. While this happens you will see the 'waiting for lo to become free' message repeated.
2. while the previous container is waiting for its sockets to exit, so it can exit, any new containers are blocked from starting.
For issue #2, I'm not clear just yet on what is blocking the new containers from starting. I thought it could be the rtnl_lock since the exiting container does loop in the rtnl_unlock path, however while it's looping it also calls __rtnl_
For issue #1, the sockets are lingering because - at least for the reproducer case from comment 16 - there is a TCP connection that never is closed, so the kernel continues to probe the other end until it times out, which takes around 2 minutes. For this specific reproducer, there are 2 easy ways to work around this. First for background, the reproducer uses docker to create a CIFS mount, which uses a TCP connection. This is the script run in the 'client' side, where the hang happens:
date ; \
mount.cifs //$SAMBA_
date ; \
ls -la /mnt ; \
umount /mnt ; \
echo "umount ok"
this is repeated 50 times in the script, but the 2nd (and later) calls hang (for 2 minutes each, if you wait that long).
A. The reason the TCP connection to the CIFS server lingers is beacuse it is never closed; the reason it isn't closed is because the script exits the container immediately after unmounting. The kernel cifs driver, during fs unmount, closes the TCP socket to the CIFS server, which queues a TCP FIN to the server. Normally, this FIN will reach the server who will respond and the TCP connection will close. However, since the container starts exiting immediately, that FIN never reaches the CIFS server - the interface is taken down too quickly. So, the TCP connection remains. To avoid this, simply add a short sleep between unmounting and exiting the container, e.g. (new line prefixed with +):
date ; \
mount.cifs //$SAMBA_
date ; \
ls -la /mnt ; \
umount /mnt ; \
+ sleep 1 ; \
echo "umount ok"
that avoids the problem, and allows the container to exit immediately and the next container to start immediately.
B. Instead of delaying enough to allow the TCP connection to exit, the kernel's TCP configuration can be changed to more quickly timeout. The specific setting to change is tcp_orphan_retries, which controls how many times the kernel keeps trying to talk to a remote host over a closed socket. Normally, this defaults to 0 - this will cause the kernel to actually use a value of 8 (retries). So, change it from 0 to 1, to actually reduce it from 8 to 1 (confusing, I know). Like (n...
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Charles (bityard) wrote : | #24 |
Is there any way for a non-kernel-dev type to see exactly which resource unregister_
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Fabian Holler (fh0) wrote : | #25 |
Hello Dan,
thanks for the analysis!
That the startup of containers is delayed is annoying.
The much bigger issue is that it can reproducible cause a kernel Oops and crash a whole machine.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Fabian Holler (fh0) wrote : | #26 |
According to https:/
https:/
https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #27 |
My analysis so far of the problem:
1. container A has an outstanding TCP connection (thus, a socket and dst which hold reference on the "lo" interface from the container). When the container is stopped, the TCP connection takes ~2 minutes to timeout (with default settings).
2. when container A is being removed, its net namespace is removed via net/core/
3. When a new container is started, part of its startup is to call net/core/
There are a few ways to possibly address this:
a) when a container is being removed, all its TCP connections should abort themselves. Currently, TCP connections don't directly register for interface unregister events - they explictly want to stick around, so if an interface is taken down and then brought back up, the TCP connection remains, and the communication riding on top of the connection isn't interrupted. However, the way TCP does this is to move all its dst references off the interface that is unregistering, to the loopback interface. This works for the initial network namespace, where the loopback interface is always present and never removed. However, this doesn't work for non-default network namespaces - like containers - where the loopback interface is unregistered when the container is being removed. So this aspect of TCP may need to change, to correctly handle containers. This also may not cover all causes of this hang, since sockets handle more than just tcp connections.
b) when a container is being removed, instead of holding the net_mutex while the cleaning up (and calling rtnl_unlock), it could release the net_mutex first (after removing all netns marked for cleanup from the pernet list), then call rtnl_unlock. This needs examination to make sure it would not introduce any races or other problems.
c) rtnl_unlock could be simplified - currently, it includes significant side-effects, which include the long delay while waiting to actually remove all references to the namespace's interfaces (including loopback). Instead of blocking rtnl_unlock() to do all this cleanup, the cleanup could be deferred. This also would need investigation to make sure no caller is expecting to be able to free resources that may be accessed later from the cleanup action (which I believe is the case).
As...
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #28 |
> Is there any way for a non-kernel-dev type to see exactly which resource
> unregister_
not really, no. It's simply waiting for its reference count to drop to 0, and just broadcasts unregister events periodically hoping that everyone with a reference is listening and does whatever they need to do to drop their references. In this case the socket's dst actually is listening, but the way the dst handles it assumes that the loopback interface will never, ever unregister - the dst shifts its reference count from the interface going down, to its netns's loopback interface. If the interface going down is the netns's loopback interface, then that reference shift doesn't help, of course.
So no, there is no simple way for you to tell what it's waiting for.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #29 |
> That the startup of containers is delayed is annoying.
> The much bigger issue is that it can reproducible cause a kernel Oops and
> crash a whole machine.
Yes as you found this uncovers other bugs in the kernel, but those are separate and looks like they are getting fixed.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Fabian Holler (fh0) wrote : | #30 |
Ok, seems that I'm than in the wrong ticket with my issues. :-)
Should separate Ubuntu bug reports be created (if they don't exist already) regarding the kernel
crashes?
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #31 |
> Should separate Ubuntu bug reports be created (if they don't exist already) regarding
> the kernel crashes
This bug is only to fix the hang/delay in new container creation. Please open a new bug for kernel crashes.
Changed in linux (Ubuntu Artful): | |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu Bionic): | |
status: | New → In Progress |
Changed in linux (Ubuntu Zesty): | |
status: | New → In Progress |
Changed in linux (Ubuntu Xenial): | |
status: | New → In Progress |
Changed in linux (Ubuntu Bionic): | |
importance: | Undecided → Medium |
assignee: | nobody → Dan Streetman (ddstreet) |
Changed in linux (Ubuntu Zesty): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Trusty): | |
assignee: | nobody → Dan Streetman (ddstreet) |
Changed in linux (Ubuntu Xenial): | |
assignee: | nobody → Dan Streetman (ddstreet) |
Changed in linux (Ubuntu Trusty): | |
status: | New → In Progress |
Changed in linux (Ubuntu Zesty): | |
assignee: | nobody → Dan Streetman (ddstreet) |
Changed in linux (Ubuntu Xenial): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu Trusty): | |
importance: | Undecided → Medium |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Jiri Horky (jiri-horky) wrote : | #32 |
his information might be relevant.
We are able to reproduce the problem with unregister_
BOOT_IMAGE=
Once we hit this state, it stays in this state and reboot is needed. No more containers can be spawned. We reproduce it by running images doing ipsec/openvpn connections + downloading a small file inside the tunnels. Then the instances exist (usually they run < 10s). We run 10s of such containers a minute on one machine. With the abovementioned settings (only 1cpu), the machine hits it in ~2 hours.
Another reproducer with the same kernel, but without limiting number of CPUs, is to jus run iperf in UDP mode for 3 seconds inside the container (so there is no TCP communication at all). If we run 10 of such containers in parallel, wait for all of them to finish and do it again, we hit the trouble in less than 10 minutes (on 40 cores machine).
In both of our reproducers, we added "ip route flush table all; ifconfig down; sleep 10" before existing from containers. It does not seem to have any effect.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #33 |
> In both of our reproducers, we added "ip route flush table all;
that won't help
> ifconfig down;
that's not a valid cmdline
> sleep 10" before existing from containers.
sleeping without closing existing open sockets doesn't help
what are you using for container mgmt? docker? lxc? lxd?
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #34 |
simpler reproducer for this using LXD instead of Docker:
1. install/setup LXD
$ sudo apt install lxd
$ sudo lxd init
2. create two containers
$ lxc launch ubuntu:xenial server
$ lxc launch ubuntu:xenial client
3. make the client privileged (so it can mount smb/cifs)
$ lxc config set client security.privileged true
$ lxc stop client
$ lxc start client
4. set up samba server in container
$ lxc exec server apt update
$ lxc exec server apt install cifs-utils samba
$ lxc exec server mkdir /dummy
$ lxc exec server touch /dummy/emptyfile
$ lxc exec server vim /etc/samba/smb.conf
add this section at the end of the container's smb.conf:
[dummy]
path = /dummy
browseable = yes
read only = yes
guest ok = yes
$ lxc stop server
$ lxc start server
5. note server container's ip
$ lxc list server
e.g.
| server | RUNNING | 10.109.201.68 (eth0)
6. setup client container
$ lxc exec client mkdir /dummy
$ lxc exec apt update
$ lxc exec apt install cifs-utils
7. stop client container
$ lxc stop client
8. full cycle of client container start through stop
$ lxc start client ; sleep 2 ; lxc exec client -- mount -o guest //10.109.
9. repeat client cycle; it should hang due to this bug
$ lxc start client ; sleep 2 ; lxc exec client -- mount -o guest //10.109.
Note this force-stops the container, instead of normally stopping it; if the container is allowed to exit normally, the problem isn't reproduced (with these steps, at least). Also note the 'sleep 2' is there to give enough time for the client container network to come up, it may need to be increased if your client container takes longer to assign its ip address (you can edit the client container nw config to use a static address if you want, which should be faster).
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Stefan Rinkes (darinkes) wrote : | #35 |
Hi,
In a test-setup I can easily reproduce this issue very quickly.
At my scenario it looks like the issue is just triggered if I change the IP inside the docker container
(Needed to test StrongSwan's MOBIKE feature), send some data (ICMP) and then kill the docker container.
So if there is something to test or debug, I would be happy to help fixing this issue.
Also already tried the latest kernel from http://
Linux docker 4.15.0-
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #36 |
> So if there is something to test or debug, I would be happy to help fixing this issue.
yes, I'm hoping to have a kernel build with some added debug to track/list exactly what socket(s) are still holding lo device references when the container is killed/shutdown, so testing with that would help clarify the specifics of this problem, once I have the debug kernel ready and built. I will update this bug once I have something ready to test.
> Also already tried the latest kernel from http://
and I assume that did not help, correct? This problem still exists upstream (assuming we're all encountering the same problem that I'm debugging) and will require an upstream patch.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Daniel Borkmann (dansel) wrote : | #37 |
Hi Dan,
the test kernel you were using had this one [1] included? It's upstream commit f186ce61bb82 ("Fix an intermittent pr_emerg warning about lo becoming free."). Meaning, it's a rather a dst leak somewhere in latest upstream as well?
Thanks,
Daniel
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #38 |
> the test kernel you were using had this one [1] included? It's upstream commit f186ce61bb82
no that commit doesn't help, because the issue isn't one of dst refcount leaks or delayed dst gc, the problem is kernel sockets that haven't closed yet holding a reference to the netns lo interface device via their dsts.
I do think I have a possible fix for this though, and I have some test kernels building in this ppa (please note they are currently building and will take a few more hours to finish):
https:/
can anyone please test with any of those kernels (they all contain the patch to maybe fix this)? In my testing it eliminated the problem completely, but I'd like to get some more diverse real-world testing before starting an upstream discussion on the patch.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Stefan Rinkes (darinkes) wrote : | #39 |
Ok, waiting for the build to finish and then will test&report ASAP.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Stefan Rinkes (darinkes) wrote : | #40 |
$ sudo add-apt-repository ppa:ddstreet/
Cannot add PPA: 'ppa:~ddstreet/
ERROR: '~ddstreet' user or team does not exist.
DISTRIB_ID=Ubuntu
DISTRIB_
DISTRIB_
DISTRIB_
Changed in linux (Ubuntu Zesty): | |
status: | In Progress → Won't Fix |
assignee: | Dan Streetman (ddstreet) → nobody |
29 comments hidden
Loading more comments
|
view all 109 comments |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Jiri Horky (jiri-horky) wrote : | #70 |
Hi Dan,
how should I read that it got to "Won't fix" state? No more time to debug it?
Thanks
Jirka H.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #71 |
Hi Jiri,
no the "Won't Fix" is only for the Zesty release, as that's EOL now.
I actually have a test kernel building in the ppa:
https:/
it's version 4.13.0-
this patch isn't quite finished, I still have to redo the netns cleanup workqueue because the cleanup work can now hang (indefinitely) so it can't be a single-threaded workqueue with the same work queued. But, it should at least verify if it works around the container create/destroy hangs.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #72 |
Just to re-summarize this:
The problem here is when a net namespace is being cleaned up, the thread cleaning it up gets hung inside net/core/dev.c netdev_
The thing(s) holding dev references here are dst objects from sockets (at least, that's what is failing to release the dev refs, in my debugging). When a net namespace is cleaned up, all its sockets' call dst_dev_put() for their dsts, which moves the netdev refs from the actual net namespace interfaces to the netns loopback device (this is why the message is typically 'waiting for lo to become free', but sometimes it waits for non-lo to become free if the dst_dev_put hasn't been called for a dst).
The dsts are then put, which should reduce their ref count to 0 and free them, which then also reduces the lo device's ref count. In correct operation, this reduces the lo device's ref count to 0, which allows netdev_
This is where the problem arrives:
1. the first possibility a kernel socket that hasn't closed, and still has dst(s). This is possible because while normal (user) sockets take a reference to their net namespace, and the net namespace does not begin cleanup until all its references are released (meaning, no user sockets are still open), kernel sockets do not take a reference to their net namespace. So, net namespace cleanup begins while kernel sockets are still open. While the net namespace is cleaning up, it ifdown's all its interfaces, which cause the kernel sockets to close, and free their dsts which releases all netdev interface refs, which allows the net namespace to close. If, however, one of the kernel sockets doesn't pay attention to the net namespace cleaning up, it may hang around, failing to free its dsts, and failing to release the netdev interface reference(s). That causes this bug. In some cases (maybe most or all, I'm not sure) the socket eventually times out, and does release its dsts, allowing the net namespace to finish cleaning up and exiting.
My recent patch 4ee806d51176ba7
2. alternately, even if all kernel sockets behave correctly and cleanup immediately during net namespace cleanup, it's possible for a dst object to leak a reference. This has happened before, there are likely existing dst leaks, and there will likely be dst leaks accidentally introduced in the future. When a dst leaks, its refcount never reaches 0, and so it never releases its reference on its interface (which remember, is ...
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #73 |
Jiri,
I just added a new artful version to the ppa, version 4.13.0-
I also added debug to this new build that will show each net namespace alloc and free, like:
[ 207.625857] net_namespace: alloced net ffff8d2eb3cae000
...
[ 208.152882] net_namespace: freeing net ffff8d2eb3cae000
and it has the same dbg during reproduction of this bug, where it hangs, like:
[ 1363.377355] unregister_
[ 1363.377393] (netns ffff96ddf1c10000): open sk ffff96ddf1e77380 family 2 state 4 dst_cache (null) rx_dst ffff96ddf532d800 creator generic_
[ 1363.377406] (netns ffff96ddf1c10000): dst ffff96ddf532d800 expires 0 error 0 obsolete 2 flags 0 refcnt 1 ops ipv4_dst_
If you're able to test with today's build, let me know if you are able to reproduce the problem (see the 'namespace leak' message) and if it has worked around the container hang correctly. Thanks!
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Jiri Horky (jiri-horky) wrote : | #74 |
Hi there, great to hear the progress!
I have just reproduced it using the previous #43+hf1711407v2
Apr 13 19:12:14 prg41-004 kernel: [13621.398319] unregister_
Apr 13 19:12:24 prg41-004 kernel: [13631.478266] unregister_
The seconds seems to be rather milliseconds.
Waiting here for the newest kernel and will try it right away.
Jiri Horky
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #75 |
> I have just reproduced it using the previous #43+hf1711407v2
Excellent. And just to clarify again, I do *not* expect the kernel to fix/avoid the problem - I fully expect it to happen - the change in these kernels is strictly to make sure that the system is still usable after the problem happens. Can you verify that after reproducing? I assume that before, when you reproduced the problem, you were no longer able to create new (or destroy old) containers (i.e., new net namespaces) - that should now be fixed, and you should be able to keep creating new containers with no issues.
> The seconds seems to be rather milliseconds.
yep sorry, kernel building now should fix that to print seconds instead of ms.
> Waiting here for the newest kernel and will try it right away.
thanks!
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Jiri Horky (jiri-horky) wrote : | #76 |
Hi Dan,
so here are the information for the latest kernel (4.13.0-
The bug got reproduced:
[ 1085.626663] unregister_
[ 1085.626696] (netns ffff8d2dc3d34800): dst ffff8d2dd006db00 expires 0 error 0 obsolete 2 flags 0 refcnt 1 ops ipv4_dst_
If I grep the netns: ffff8d2dc3d34800, it is visible that during first 10 minutes, it outputs more info, then until ~ 1 hour, it says the same message more frequently and then only outputs it ever hour.
[ 901.854919] net_namespace: alloced net ffff8d2dc3d34800
[ 1085.626663] unregister_
[ 1085.626696] (netns ffff8d2dc3d34800): dst ffff8d2dd006db00 expires 0 error 0 obsolete 2 flags 0 refcnt 1 ops ipv4_dst_
[ 1095.706659] unregister_
[ 1095.706692] (netns ffff8d2dc3d34800): dst ffff8d2dd006db00 expires 0 error 0 obsolete 2 flags 0 refcnt 1 ops ipv4_dst_
[ 1105.786651] unregister_
[ 1105.786669] (netns ffff8d2dc3d34800): dst ffff8d2dd006db00 expires 0 error 0 obsolete 2 flags 0 refcnt 1 ops ipv4_dst_
[ 1115.866654] unregister_
....
[ 1631.003906] unregister_
[ 1641.147950] unregister_
[ 1651.291981] unregister_
[ 1661.372025] unregister_
[ 1671.452071] unregister_
[ 1681.564110] unregister_
...
[ 4628.301622] unregister_
[ 8252.383668] unregister_
[11877.333159] unregister_
[15502.281254] unregister_
[19127.230600] unregister_
[22752.179625] unregister_
[26377.129256] unregister_
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #77 |
> The machine continue to run and the docker is still able to create new instances,
> i.e. there is no complete lock out.
Great!
> The machine continued to run further, and after an ~hour , 3 namespaces leaked.
> The machine now has load "3", which seems to be an artifact of this. Other than that,
> it seems to continue to run!
Hmm, can you check 'top' or 'htop' or similar to see if the kworker thread(s) are actually using cpu time? They should be sleeping almost all the time, not spinning on anything.
I'll work on getting that patch upstream (but likely not nearly as verbose, probably just a single WARNING for leaked namespace).
After that, I'll keep working on adding dst leak debugging to actually try to find where your dst leak is coming from.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Jiri Horky (jiri-horky) wrote : | #78 |
Hi Dan,
the CPU usage is zero, there is nothing popping up. The tasks causing the high load value are in D state. If I do "echo w > /proc/sysrq-
[392149.562095] kworker/u81:40 D 0 4290 2 0x80000000
[392149.562112] Workqueue: netns cleanup_net
[392149.562125] Call Trace:
[392149.562135] __schedule+
[392149.562146] schedule+0x2c/0x80
[392149.562157] schedule_
[392149.562167] ? call_timer_
[392149.562177] msleep+0x2d/0x40
[392149.562188] ? msleep+0x2d/0x40
[392149.562199] netdev_
[392149.562210] rtnl_unlock+
[392149.562221] default_
[392149.562233] ? do_wait_
[392149.562245] cleanup_
[392149.562257] process_
[392149.562270] worker_
[392149.562281] kthread+0x128/0x140
[392149.562292] ? process_
[392149.562300] ? kthread_
[392149.562305] ? do_syscall_
[392149.562310] ? SyS_exit_
[392149.562316] ret_from_
I am not sure how fine is to have e.g. thousands of such leaked namespaces - if I can't run out of some resources (kworker threads?). I am going to keep the machine running + generate even higher load there to see if I hit any such limit.
Thanks
JH
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #79 |
> the CPU usage is zero, there is nothing popping up. The tasks causing the high load
> value are in D state
ok that seems expected then. This doesn't 'fix' the problem of the leaked dst (and namespace), just prevents blocking further net namespace creation/deletion.
> if I can't run out of some resources (kworker threads?)
there is a limit, yes, and once it's hit namespace cleanup will essentially stop, and all further net namespaces will be leaked. However, there's really just no way to avoid that without fixing the actual dst leaks, which I'd still like to try to add debug to help identify.
I've been busy elsewhere last week but I'll prepare the patch to send upstream this week, and then look at adding debug to actually find the dst leak.
Thanks!
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Gonzalo Servat (gservat) wrote : | #80 |
Hi Dan,
We run a bunch of instances both on AWS and GCP, and we run a significant number of containers on both. We've only ever seen this problem on GCP and never on AWS (it's baffling!). The kernels are as-close-
-what release are you using (trusty/
Xenial
-what kernel version are you using?
4.13.0-1008-gcp
-do you have specific steps to reproduce the problem?
Unfortunately we don't. It just happens on its own after 1/2 weeks. The following shows in dmesg:
[1015401.681728] unregister_
[1015411.761772] unregister_
[1015421.841740] unregister_
[1015431.953729] unregister_
etc etc
We've tried a few things including upgrading kernel and disabling IPv6 to no avail.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #81 |
Hi @gservat,
> To answer your questions:
There's actually been quite a lot of discussion/work in this bug since comment 2, so I don't actually need that info anymore (except for reproduction steps, that's always welcome).
> It just happens on its own after 1/2 weeks. The following shows in dmesg:
Are you having any problems with the system *other* than just the log messages?
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Gonzalo Servat (gservat) wrote : | #82 |
Hi Dan,
> There's actually been quite a lot of discussion/work in this bug since comment 2, so I don't
> actually need that info anymore (except for reproduction steps, that's always welcome).
Fair enough. I wish I had some reproduction steps but unfortunately it just happens randomly and not because of any one action.
> Are you having any problems with the system *other* than just the log messages?
Apologies here. I should have mentioned how it affects us. The symptoms seen are a number of docker commands that just hang indefinitely (e.g. docker ps, spinning up new containers). Occasionally, after a long time, we have seen it come back and continue working OK (like whatever was holding the lock released it finally), but most of the time it hangs indefinitely. On occasion we can also use the -n parameter (e.g. docker ps -n #) and it may or may not return a partial listing of running containers. We've also found (again, not always) that we might be able to stop a container if we use the container name or ID. When we strace the process, we just see:
write(5, "GET /v1.24/
futex(0xc8200fa590, FUTEX_WAKE, 1) = 1
futex(0x21fce50, FUTEX_WAIT, 0, NULL
... and that's as far as it goes. I do have a Docker stacktrace if you want to see that.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #83 |
> The symptoms seen are a number of docker commands that just hang indefinitely (e.g.
> docker ps, spinning up new containers). Occasionally, after a long time, we have
> seen it come back and continue working OK (like whatever was holding the lock
> released it finally), but most of the time it hangs indefinitely
ok, great. You didn't mention what release you're using on gcp, but I have a 4.13 test kernel in this ppa, with patches to specifically fix/workaround the hanging problem:
https:/
Note that *only* version 4.13.0-
The details of the cause of this problem is back in comment 72, if you want that info. I'm preparing to send the workaround patch upstream; after that, there will be (likely ongoing) work to debug the various dst/object leaks and kernel socket "leaks" that are actually causing the symptoms.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Gonzalo Servat (gservat) wrote : Re: [Bug 1711407] Re: unregister_netdevice: waiting for lo to become free | #84 |
Hi Dan,
We're using a newer kernel:
4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 x86_64
x86_64 GNU/Linux
Would you be able to build a test kernel against this version?
Thanks very much, Dan.
On Tue, May 1, 2018 at 12:14 AM, Dan Streetman <email address hidden>
wrote:
> > The symptoms seen are a number of docker commands that just hang
> indefinitely (e.g.
> > docker ps, spinning up new containers). Occasionally, after a long time,
> we have
> > seen it come back and continue working OK (like whatever was holding the
> lock
> > released it finally), but most of the time it hangs indefinitely
>
> ok, great. You didn't mention what release you're using on gcp, but I
> have a 4.13 test kernel in this ppa, with patches to specifically
> fix/workaround the hanging problem:
> https:/
>
> Note that *only* version 4.13.0-
> has the patches to workaround the hanging. If you need a different
> version, let me know and I can apply the patches and build for whatever
> release you need (just give me the 'uname -a' output).
>
> The details of the cause of this problem is back in comment 72, if you
> want that info. I'm preparing to send the workaround patch upstream;
> after that, there will be (likely ongoing) work to debug the various
> dst/object leaks and kernel socket "leaks" that are actually causing the
> symptoms.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> unregister_
>
> Status in linux package in Ubuntu:
> In Progress
> Status in linux source package in Trusty:
> In Progress
> Status in linux source package in Xenial:
> In Progress
> Status in linux source package in Zesty:
> Won't Fix
> Status in linux source package in Artful:
> In Progress
> Status in linux source package in Bionic:
> In Progress
>
> Bug description:
> This is a "continuation" of bug 1403152, as that bug has been marked
> "fix released" and recent reports of failure may (or may not) be a new
> bug. Any further reports of the problem should please be reported
> here instead of that bug.
>
> --
>
> [Impact]
>
> When shutting down and starting containers the container network
> namespace may experience a dst reference counting leak which results
> in this message repeated in the logs:
>
> unregister_
> 1
>
> This can cause issues when trying to create net network namespace and
> thus block a user from creating new containers.
>
> [Test Case]
>
> See comment 16, reproducer provided at https:/
> samba-loop
>
> To manage notifications about this bug go to:
> https:/
> 1711407/
>
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #85 |
> Would you be able to build a test kernel against this version?
very sorry, i've been quite busy the last couple weeks. I'll get a test kernel built for you next week.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Gonzalo Servat (gservat) wrote : | #86 |
Thanks for the ping, Dan. No worries. Looking forward to it as this issue
seems to be biting us more frequently nowadays so really interested to see
if this kernel helps.
On Sat, May 19, 2018 at 7:19 AM, Dan Streetman <email address hidden>
wrote:
> > Would you be able to build a test kernel against this version?
>
> very sorry, i've been quite busy the last couple weeks. I'll get a test
> kernel built for you next week.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> unregister_
>
> Status in linux package in Ubuntu:
> In Progress
> Status in linux source package in Trusty:
> In Progress
> Status in linux source package in Xenial:
> In Progress
> Status in linux source package in Zesty:
> Won't Fix
> Status in linux source package in Artful:
> In Progress
> Status in linux source package in Bionic:
> In Progress
>
> Bug description:
> This is a "continuation" of bug 1403152, as that bug has been marked
> "fix released" and recent reports of failure may (or may not) be a new
> bug. Any further reports of the problem should please be reported
> here instead of that bug.
>
> --
>
> [Impact]
>
> When shutting down and starting containers the container network
> namespace may experience a dst reference counting leak which results
> in this message repeated in the logs:
>
> unregister_
> 1
>
> This can cause issues when trying to create net network namespace and
> thus block a user from creating new containers.
>
> [Test Case]
>
> See comment 16, reproducer provided at https:/
> samba-loop
>
> To manage notifications about this bug go to:
> https:/
> 1711407/
>
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #87 |
> We're using a newer kernel:
> 4.4.0-109-generic
ok, i backported the workaround to the xenial 4.4 kernel, and built it at the same ppa:
https:/
it's version 4.4.0-127.
can you test with that and let me know if it works around the hangs? Note that it is still expected to show errors in the logs when the problem is reproduced; this workaround is strictly to avoid internal hangs when the error happens, so you should be able to still create/destroy containers (net namespaces).
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Gonzalo Servat (gservat) wrote : | #88 |
Many thanks, Dan! Going to roll it out to a few test boxes and try it out over the next few days. Will report back as soon as I have some feedback.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
lirongqing (lirongqing) wrote : | #89 |
I have the same issue, and I dump the vmcore, and find dst cache, hope it has some help.
this leaked dst is in dst_busy_list, except dst_busy_list, nowhere I can find it in
=======
First case:
crash> rtable 0xffff880036fbba00 -x
struct rtable {
dst = {
callback_head = {
next = 0xffff880036fbb300,
func = 0x0
},
child = 0x0,
dev = 0xffff8817f15c3000,
ops = 0xffffffff8191f2c0 <ipv4_dst_ops>,
_metrics = 0xffffffff815ef7d1,
expires = 0x0,
path = 0xffff880036fbba00,
from = 0x0,
xfrm = 0x0,
input = 0xffffffff8149f777 <dst_discard>,
{
output = 0xffffffff8149f762 <dst_discard_sk>,
_
output = 0xffffffff8149f762 <dst_discard_sk>
},
{<No data fields>}
},
flags = 0x0,
pending_confirm = 0x0,
error = 0x0,
obsolete = 0x2,
header_len = 0x0,
trailer_len = 0x0,
tclassid = 0x0,
__pad_
__refcnt = {
counter = 0x1
},
__use = 0x0,
lastuse = 0x1000860b0,
{
next = 0xffff882fdaaa9800,
rt_next = 0xffff882fdaaa9800,
rt6_next = 0xffff882fdaaa9800,
dn_next = 0xffff882fdaaa9800
},
rh_reserved1 = 0x2,
rh_reserved2 = 0x85987845,
rh_reserved3 = 0x0,
rh_reserved4 = 0x0
},
rt_genid = 0x11,
rt_flags = 0x80000000,
rt_type = 0x2,
rt_is_input = 0x1,
rt_uses_gateway = 0x0,
rt_iif = 0x0,
rt_gateway = 0x0,
rt_pmtu = 0x0,
rt_uncached = {
next = 0xffff880036fbbac0,
prev = 0xffff880036fbbac0
}
}
crash>
=======
seconds case:
crash> rtable ffff8807ec0d9a00 -x
struct rtable {
dst = {
callback_head = {
next = 0xffff8807ec0db900,
func = 0xffffffff8119e250 <file_free_rcu>
},
child = 0x0,
dev = 0xffff8817ed384800,
ops = 0xffffffff81b18bc0 <ipv4_dst_ops>,
_metrics = 0xffffffff817ca681,
expires = 0x0,
path = 0xffff8807ec0d9a00,
from = 0x0,
xfrm = 0x0,
input = 0xffffffff8162b200 <dst_discard>,
{
output = 0xffffffff8162b1e0 <dst_discard_sk>,
_
output = 0xffffffff8162b1e0 <dst_discard_sk>
},
{<No data fields>}
},
flags = 0x6,
pending_confirm = 0x0,
error = 0x0,
obsolete = 0x2,
header_len = 0x0,
trailer_len = 0x0,
tclassid = 0x0,
__pad_
__refcnt = {
counter = 0x1
},
__use = 0x0,
lastuse = 0x1fcce9056,
{
next = 0xffff880404f49400,
rt_next = 0xffff880404f49400,
rt6_next = 0xffff880404f49400,
dn_next = 0xffff880404f49400
},
rh_reserved1 = 0x0,
rh_reserved2 = 0x0,
rh_reserved3 = 0x0,
rh_reserved4 = 0x0
},
rt_genid = 0x16,
rt_flags = 0x80000000,
rt_type = 0x2,
rt_is_input = 0x0,
rt_uses_gateway = 0x0,
rt_iif = 0x0,
rt_gateway = 0x0,
rt_pmtu = 0x0,
rt_uncached = {
next = 0xffff8807ec0d9ac0,
prev = 0xffff8807ec0d9ac0
}
}
crash>
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
lirongqing (lirongqing) wrote : | #90 |
is it possibility that sk->sk_dst_cache is overwritten? like in __sk_dst_check,
when tcp timer tries to resend a packet, at the same time, tcp_close is called, and a reset packet will send, and ip_queue_xmit will be called concurrent;
cpu 1 cpu 2
tcp_close
tcp_
ip_queue_xmit
dst = __sk_dst_get(sk);
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #91 |
> is it possibility that sk->sk_dst_cache is overwritten?
maybe ;-)
> like in __sk_dst_check,
> when tcp timer tries to resend a packet, at the same time, tcp_close is called
tcp_close() locks the sock; tcp_retransmit_
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #92 |
> So they should not be operating on the same sock concurrently.
But, of course, all this is caused by bugs, so yes it's certainly possible a bug allowed what you said to happen, but you'll need to dig in deeper to find where such a bug is (if any), since the locking design is correct as far as I can tell.
1 comments hidden
Loading more comments
|
view all 109 comments |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andy Whitcroft (apw) wrote : Closing unsupported series nomination. | #94 |
This bug was nominated against a series that is no longer supported, ie artful. The bug task representing the artful nomination is being closed as Won't Fix.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.
Changed in linux (Ubuntu Artful): | |
status: | In Progress → Won't Fix |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
TomaszChmielewski (mangoo-wpkg) wrote : | #95 |
I'm hitting this bug frequently on AWS with Ubuntu 16.04.5 LTS:
Linux uni02.sys.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
TomaszChmielewski (mangoo-wpkg) wrote : | #96 |
Also seeing it on Ubuntu 18.04 on AWS:
# uname -a
Linux uni09.sys.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Gonzalo Servat (gservat) wrote : | #97 |
Dan,
We had been running 80-100 instances on the hotfix kernel and I was calling victory on the workaround as it has been a month without this issue popping up, but I just had an instance lockup on, say, "docker ps" and dmesg shows:
[2924480.806202] unregister_
[2924490.870209] unregister_
[2924501.006177] unregister_
[2924511.134204] unregister_
[2924521.218189] unregister_
:-(
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Gonzalo Servat (gservat) wrote : | #98 |
Using:
linux-image-
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Dan Streetman (ddstreet) wrote : | #99 |
unfortunately I've moved away from kernel work, for now at least, and no longer have time to continue this bug; if anyone else would like to pick up the work, I'm happy to answer questions.
Changed in linux (Ubuntu Xenial): | |
assignee: | Dan Streetman (ddstreet) → nobody |
Changed in linux (Ubuntu Artful): | |
assignee: | Dan Streetman (ddstreet) → nobody |
Changed in linux (Ubuntu): | |
assignee: | Dan Streetman (ddstreet) → nobody |
Changed in linux (Ubuntu Trusty): | |
assignee: | Dan Streetman (ddstreet) → nobody |
Changed in linux (Ubuntu Bionic): | |
assignee: | Dan Streetman (ddstreet) → nobody |
tags: | added: cscc |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Jim Heald (steelcowboy1) wrote : | #100 |
Hi! I'm on Debian and I'm getting this issue with the following kernel:
4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26)
I feel like this isn't the best place to report this, but can someone point me to the best place to report this issue? Can't find any discussion on it from the past year but I don't know how I could have triggered it...
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Thiago Vincenzi Conrado (tconrado28) wrote : | #101 |
hey guys, we create our namespace using ip netns add.
Each nameSpace has it's ipv6 and ipv4.
We notice that the ipv6 ping continues to report, but the namespace at /run/netns does not exist anymore.
is it possible to find the namespace using the ipv6 info?
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Thiago Vincenzi Conrado (tconrado28) wrote : | #102 |
4.15.0-54-generic did work great in the past
we moved to 5.3.0-28-generic and this bug is notable...
funny enough one of our machines was able to recover... both have the very same OS version
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
5.3.0-28-generic
maybe a downgrade de to Ubuntu 18.04.2 is a "solution"
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Thiago Vincenzi Conrado (tconrado28) wrote : | #103 |
4.15.0-54-generic is bug free
5.3.0-28-generic has the bug
funny enough one of our machines was able to recover... both have the very same OS version
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
5.3.0-28-generic
maybe a downgrade de to Ubuntu 18.04.2 is a "solution"
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Sandro (supersandro2000) wrote : | #104 |
Bug also happens on 5.4.0-42-generic.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Nivedita Singhvi (niveditasinghvi) wrote : | #105 |
Is anyone still seeing a similar issue on current mainline?
tags: | added: sts |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Mariano López (justanotherboy) wrote : | #106 |
Our serves are affected by this bug too in focal and it triggered today in one of our servers.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
We have seen this in:
- Ubuntu 5.4.0-89.
- Ubuntu 5.4.0-88.99-generic
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Nivedita Singhvi (niveditasinghvi) wrote : | #107 |
We are seeing definitely a problem on kernels after 4.15.0-159-generic,
which is the last known good kernel. 5.3* kernels are affected, but I
do not have data on most recent upstream.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Nivedita Singhvi (niveditasinghvi) wrote : | #108 |
As several different forums are discussing this issue,
I'm using this LP bug to continue investigation in to
current manifestation of this bug (after 4.15 kernel).
I suspect it's in one of the other places not fixed, as
my colleague Dan stated a while ago.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Mariano López (justanotherboy) wrote : | #109 |
Hi Nivedita,
Can you point me to the threads discussing this problem? I'm aware of this one, https:/
Thanks!
This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:
apport-collect 1711407
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.