segfault at 0 ip 00007fe70ae4e3b2 sp 00007fe70884fb70 error 4 in liblxcfs.so[7fe70ae46000+f000]

Bug #1807628 reported by Haw Loeung on 2018-12-10
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
lxcfs (Ubuntu)
Undecided
Christian Brauner

Bug Description

Hi,

lxcfs crashed earlier today requiring us to restart a bunch of LXC containers. I'm not able to upload using apport-bug but here's the attached crash report.

I commented on https://github.com/lxc/lxcfs/issues/73#issuecomment-445598111 and repeating what I said here:

| Dec 8 06:25:03 orlo kernel: [25247258.665022] lxcfs[3871]: segfault at 0 ip 00007fe70ae4e3b2 sp 00007fe70884fb70 error 4 in liblxcfs.so[7fe70ae46000+f000]
| Dec 8 06:25:09 orlo systemd[1]: lxcfs.service: Main process exited, code=killed, status=11/SEGV
| Dec 8 06:25:09 orlo systemd[1]: lxcfs.service: Unit entered failed state.
| Dec 8 06:25:09 orlo systemd[1]: lxcfs.service: Failed with result 'signal'.
| Dec 8 06:25:10 orlo systemd[1]: lxcfs.service: Service hold-off time over, scheduling restart.
| Dec 8 06:25:10 orlo lxcfs[10839]: hierarchies:
| Dec 8 06:25:10 orlo lxcfs[10839]: 0: fd: 5: perf_event
| Dec 8 06:25:10 orlo lxcfs[10839]: 1: fd: 6: blkio
| Dec 8 06:25:10 orlo lxcfs[10839]: 2: fd: 7: freezer
| Dec 8 06:25:10 orlo lxcfs[10839]: 3: fd: 8: devices
| Dec 8 06:25:10 orlo lxcfs[10839]: 4: fd: 9: cpuset
| Dec 8 06:25:10 orlo lxcfs[10839]: 5: fd: 10: cpu,cpuacct
| Dec 8 06:25:10 orlo lxcfs[10839]: 6: fd: 11: pids
| Dec 8 06:25:10 orlo lxcfs[10839]: 7: fd: 12: memory
| Dec 8 06:25:10 orlo lxcfs[10839]: 8: fd: 13: net_cls,net_prio
| Dec 8 06:25:10 orlo lxcfs[10839]: 9: fd: 14: hugetlb
| Dec 8 06:25:10 orlo lxcfs[10839]: 10: fd: 15: name=systemd

So now after restarting it, the containers are showing this:

| Error: /proc must be mounted
| To mount /proc at boot you need an /etc/fstab line like:
| proc /proc proc defaults
| In the meantime, run "mount proc /proc -t proc"

Package version:

| ubuntu@orlo:~$ cfs
| lxcfs:
| Installed: 2.0.8-0ubuntu1~16.04.2
| Candidate: 2.0.8-0ubuntu1~16.04.2
| Version table:
| 3.0.2-0ubuntu1~16.04.1 100
| 100 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
| *** 2.0.8-0ubuntu1~16.04.2 500
| 500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
| 100 /var/lib/dpkg/status
| 2.0.0-0ubuntu2 500
| 500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

Haw Loeung (hloeung) wrote :
description: updated
Stéphane Graber (stgraber) wrote :
Download full text (4.5 KiB)

Trace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fe70ae4e3b2 in cg_readdir (path=<optimized out>, buf=0x7fe690003610, filler=0x7fe70b850ce0 <fill_dir>, offset=<optimized out>, fi=<optimized out>)
    at bindings.c:1793
1793 bindings.c: No such file or directory.
[Current thread is 1 (Thread 0x7fe708850700 (LWP 3871))]
(gdb) bt full
#0 0x00007fe70ae4e3b2 in cg_readdir (path=<optimized out>, buf=0x7fe690003610, filler=0x7fe70b850ce0 <fill_dir>, offset=<optimized out>, fi=<optimized out>)
    at bindings.c:1793
        d = 0x7fe690000940
        list = 0x0
        i = 0
        ret = <optimized out>
        nextcg = 0x0
        fc = 0x7fe6500240d0
        clist = 0x0
        __func__ = "cg_readdir"
        initpid = <optimized out>
#1 0x000055cfc579a411 in do_proc_readdir (fi=<optimized out>, offset=<optimized out>, filler=<optimized out>, buf=<optimized out>, path=<optimized out>)
    at lxcfs.c:307
        proc_readdir = <optimized out>
        error = <optimized out>
#2 lxcfs_readdir (path=<optimized out>, buf=0x7fe690003610, filler=0x7fe70b850ce0 <fill_dir>, offset=0, fi=0x7fe70884fc80) at lxcfs.c:504
No locals.
#3 0x00007fe70b856232 in fuse_fs_readdir (fs=0x55cfc69e2fd0,
    path=0x7fe650024170 "/cgroup/name=systemd/lxc/juju-c5f7d5-1-lxd-1/user.slice/user-113.slice/session-463849.scope", buf=0x7fe690003610,
    filler=0x7fe70b850ce0 <fill_dir>, off=0, fi=0x7fe70884fc80) at fuse.c:2044
No locals.
#4 0x00007fe70b8563bc in readdir_fill (fi=0x7fe70884fc80, dh=0x7fe690003610, off=0, size=4096, ino=3274299, req=0x7fe650024ad0, f=0x55cfc69e2aa0)
    at fuse.c:3502
        d = {id = 472446402651, cond = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 532575944823, __woken_seq = 0,
              __mutex = 0x5c0b63bf, __nwaiters = 918218053, __broadcast_seq = 0},
            __size = '\000' <repeats 16 times>, "w\000\000\000|", '\000' <repeats 11 times>, "\277c\v\\\000\000\000\000E\345\272\066\000\000\000",
            __align = 0}, finished = 1544250303}
        path = 0x7fe650024170 "/cgroup/name=systemd/lxc/juju-c5f7d5-1-lxd-1/user.slice/user-113.slice/session-463849.scope"
        err = <optimized out>
#5 fuse_lib_readdir (req=0x7fe650024ad0, ino=3274299, size=4096, off=0, llfi=<optimized out>) at fuse.c:3528
        err = 0
        f = 0x55cfc69e2aa0
        fi = {flags = 0, fh_old = 140628235127104, writepage = 0, direct_io = 0, keep_cache = 0, flush = 0, nonseekable = 0, flock_release = 0, padding = 0,
          fh = 140628235127104, lock_owner = 0}
        dh = 0x7fe690003610
#6 0x00007fe70b85d0f6 in do_readdir (req=<optimized out>, nodeid=<optimized out>, inarg=<optimized out>) at fuse_lowlevel.c:1390
        arg = <optimized out>
        fi = {flags = 0, fh_old = 140628235138576, writepage = 0, direct_io = 0, keep_cache = 0, flush = 0, nonseekable = 0, flock_release = 0, padding = 0,
          fh = 140628235138576, lock_owner = 0}
#7 0x00007fe70b85e679 in fuse_ll_process_buf (data=0x55cfc69e3160, buf=0x7fe70884ff00, ch=<optimized out>) at fuse_lowlevel.c:2442
        f = 0x55cfc69e3160
        bufv = {count = 1, idx = 0, off = 0, buf = {{size = 80, flags = (unknown: 0), mem = ...

Read more...

Changed in lxcfs (Ubuntu):
assignee: nobody → Christian Brauner (cbrauner)
Christian Brauner (cbrauner) wrote :

I sent a fix: https://github.com/lxc/lxcfs/pull/262

I think the issue is

 at bindings.c:1793
        d = 0x7fe690000940
        list = 0x0
        i = 0
        ret = <optimized out>
        nextcg = 0x0
        fc = 0x7fe6500240d0
        clist = 0x0
        __func__ = "cg_readdir"
        initpid = <optimized out>

list == NULL

The other variables are fine.

Changed in lxcfs (Ubuntu):
status: New → In Progress
Haw Loeung (hloeung) wrote :

Any updates on this one? This hit us again (see latest internal incident report).

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers