auplink crashed with SIGSEGV in ftw_startup()

[modolo]

Hi!

I take docker from https://docs.docker.com/installation/ubuntulinux/ with follow instrunctions:

$ wget -qO- https://get.docker.com/ | sh

Now, I have to start docker winth "$ sudo service docker restart" after reboot.

Thanks,
Marcelo Módolo

ProblemType: Crash
DistroRelease: Ubuntu 15.04
Package: aufs-tools 1:3.2+20130722-1.1
ProcVersionSignature: Ubuntu 3.19.0-13.13-generic 3.19.3
Uname: Linux 3.19.0-13-generic x86_64
ApportVersion: 2.17-0ubuntu2
Architecture: amd64
Date: Sat Apr 11 00:13:12 2015
Dependencies:
 gcc-5-base 5-20150401-0ubuntu1
 libc6 2.21-0ubuntu4
 libgcc1 1:5-20150401-0ubuntu1
 multiarch-support 2.21-0ubuntu4
ExecutablePath: /sbin/auplink
InstallationDate: Installed on 2015-03-21 (20 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Alpha amd64 (20150318)
ProcCmdline: auplink /var/lib/docker/aufs/mnt/f2fce76cf809d39a6e9f43fb02f7ab85b8ce4ce43eeb7c69b8fae97ed3292cc4 flush
ProcEnviron:
 LANG=pt_BR.UTF-8
 LANGUAGE=pt_BR:pt:en
 PATH=(custom, no user)
SegvAnalysis:
 Segfault happened at: 0x7f754caf3759 <ftw_startup+105>: callq 0x7f754ca89200 <__memset_sse2>
 PC (0x7f754caf3759) ok
 source "0x7f754ca89200" (0x7f754ca89200) ok
 destination "(%rsp)" (0x7ffedd57fa20) not located in a known VMA region (needed writable region)!
 Stack memory exhausted (SP below stack segment)
SegvReason: writing unknown VMA
Signal: 11
SourcePackage: aufs-tools
StacktraceTop:
 ftw_startup (dir=0x2550010 "/var/lib/docker/aufs/mnt/f2fce76cf809d39a6e9f43fb02f7ab85b8ce4ce43eeb7c69b8fae97ed3292cc4", is_nftw=1, func=0x401458, descriptors=1048566, flags=19) at ../sysdeps/wordsize-64/../../io/ftw.c:654
 ?? ()
 ?? ()
 __libc_start_main (main=0x40126d, argc=3, argv=0x7ffeddd7fd08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffeddd7fcf8) at libc-start.c:289
 ?? ()
Title: auplink crashed with SIGSEGV in ftw_startup()
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

[Apport retracing service (apport)]

Thank you for taking the time to report this crash and helping to make this software better. This particular crash has already been reported and is a duplicate of bug #1441038, so is being marked as such. Please look at the other bug report to see if there is any missing information that you can provide, or to see if there is a workaround for the bug. Additionally, any further discussion regarding the bug should occur in the other report. Please continue to report any other bugs you may find.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Syver Enstad (syver-enstad)]

Sorry, newbie error, clicked on duplicate bug to see the details and inadvertently deleted the duplicate bug status. I don't know how to undo

[Sławek Ehlert (slafs)]

bug #1441038 no longer exists. Is there a new one we could follow?

[Ville Ranki (ville-ranki)]

"Me too."

Is this related do docker? If so, I installed it from Ubuntu's repositories. Now this crash happens on every boot so it's quite annoying.

Running Ubuntu vivid (15.04).

[Norm Walsh (ndw)]

"Me too, too". Ubuntu Wily.

[Jonas G. Drange (jonas-drange)]

Confirmed for Xenial.

[XerebZ (xerebz)]

Seeing this on Xubuntu 16.04 Xenial

[Salih EMIN (salih-emin)]

It happens when I build locally a docker image. It disconnects me from the network but fortunately it reconnects back

[Graham Weldon (graham-gj58)]

Confirmed on Xenial

[[e-mail] (personalpc)]

Confirmed on a fresh install of Xenial

[Andrea Del Bene (an-delbene)]

+1 on a fresh install of Xenial

[Peter Cornelius (aranaea)]

+1 on Xenial

[RH (rh-t)]

+1 on Xenial (16.04)

[Rostás Balázs (rostas-balazs)]

+1 on Xenial

[Robert Smith (rsmith31415)]

Just to include additional information, this error occurs after exiting a Docker container or after booting on Ubuntu 16.04.

[Andreo de Aguiar Vieira (andreoav)]

+1 on Xenial - Docker installed

[Kaluzki Demjan (kaluzkidemjan)]

+1

16.04
curl -fsSL https://get.docker.com/ | sh
sudo usermod -aG docker <user>

[Joern Heissler (joernheissler)]

Created new ticket https://sourceforge.net/p/aufs/bugs/26/ with explanation and patch.

[llb4ll (beatrix-vad)]

+1 on Xenial - Docker installed

[John Paul Adrian Glaubitz (glaubitz)]

People, just a heads-up: Writing "me, too" or "+1" in this bug report is absolutely pointless without providing any additional information which is missing here. It just produces noise. If you just want to say "This problem affects me as well", please just click " This bug affects 68 people. Does this bug affect you?".

Anything else just clutters the bug report and makes the work for the people who want to work on this harder.

[Krishnan Sriram Rama (krishnan-srm)]

This crashed on me too.

I am on 16.10 and I use Dell Latitude E7240. This crash happens everytime I restart my machine. In addition, everytime I do a "docker run", it crashes.

I'd appreciate if someone can provide a fix or workaround?

[Steven Peters (scpeters)]

I recently experienced this crash after upgrading to 16.04. While debugging, I noticed that my `nvidia-docker` installation wasn't working, so I upgraded that software, and I don't see the system problem dialog on launch anymore. I'm not sure why it worked.

[Fábio Antunes (fabio-h-f-antunes)]

Has anyone with this issue noticed data corruption affecting docker images or containers?

[Julian Raschke (jlnr)]

Also affects the current build of 17.10.

[Craig Jellick (craig-jellick)]

If this affects you and you are using aufs as your docker storage driver just because it is the default, you can just switch to the overlay2 storage driver to circumvent this crash.

Note that switching storage drivers will cause you to "lose" all you docker images. You'll have to repull them.

Can switch the storage driver on ubuntu (w/systemd) by putting this:
```
{
    "storage-driver": "overlay2"
}
```
in /etc/docker/daemon.json and restarting docker.

[Cedric Jehasse (cedricj)]

The trace i get is this:

SegvAnalysis:
 Segfault happened at: 0x7f022f4a25b9 <ftw_startup+105>: callq 0x7f022f438240 <__memset_sse2>
 PC (0x7f022f4a25b9) ok
 source "0x7f022f438240" (0x7f022f438240) ok
 destination "(%rsp)" (0x7fffbca06f40) not located in a known VMA region (needed writable region)!
 Stack memory exhausted (SP below stack segment)
SegvReason: writing unknown VMA
SourcePackage: aufs-tools
Stacktrace:
 #0 ftw_startup (dir=0x1081010 "/var/lib/docker/aufs/mnt/5f3306c6fad9d2fc750be5dd28b3d8d58db4eaa0d5947583d318829e2ea4326e", is_nftw=1, func=0x40149c, descriptors=1048566, flags=19) at ../sysdeps/wordsize-64/../../io/ftw.c:654
         data = {dirstreams = 0x7fffbca06f40, actdir = 0, maxdir = 1048566, dirbuf = 0xff0000 <error: Cannot access memory at address 0xff0000>, dirbufsize = 0, ftw = {base = 0, level = 0}, flags = 0, cvt_arr = 0x0, func = 0x0, dev = 0, known_objects = 0x0}
         st = {st_dev = 0, st_ino = 0, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 1523, st_blksize = 4, st_blocks = 17345968, st_atim = {tv_sec = 1531, tv_nsec = 17345968}, st_mtim = {tv_sec = 48, tv_nsec = 17305728}, st_ctim = {tv_sec = 139647365056152, tv_nsec = 1048576}, __glibc_reserved = {19, 1048566, 4199580}}
         result = 0
         cwdfd = -1
         cwd = 0x0
         cp = <optimized out>
 #1 0x0000000000401d52 in ?? ()
 No symbol table info available.
 #2 0x00000000004013ec in ?? ()
 No symbol table info available.
 #3 0x00007f022f3c9830 in __libc_start_main (main=0x401266, argc=3, argv=0x7fffbd207228, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffbd207218) at ../csu/libc-start.c:291
         result = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -7392484191596224330, 4198768, 140736366408224, 0, 0, 7392628052837490870, 7452542804783751350}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x402820, 0x7f022f783ab0 <_dl_fini>}, data = {prev = 0x0, cleanup = 0x0, canceltype = 4204576}}}
         not_first_call = <optimized out>
 #4 0x0000000000401199 in ?? ()
 No symbol table info available.

I looked at the ftw_startup implementation in glibc, and think this line could be exhausting the stack memory:
 data.dirstreams = (struct dir_data **) alloca (data.maxdir
       * sizeof (struct dir_data *));

The stacktrace shows data.maxdir is 1048566.
data.maxdir is coming from the nopendfd parameter of ntfw.

Looking at plink.c from aufs-tools package shows it uses the current number of files limit as nopenfd:
 err = getrlimit(RLIMIT_NOFILE, &rlim);
 if (err)
  AuFin("getrlimit");
 nftw(cwd, func, rlim.rlim_cur - 10,
      FTW_PHYS | FTW_MOUNT | FTW_ACTIONRETVAL);

It looks like rlim.rlim_cur is 1048576 when auplink crashes.

Is auplink started by dockerd?
I can see on my system dockerd has it's max open files set to 1048576:
cat /proc/2603/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unl...