Backport breaking change for LXCFS

[Patrick Ancillotti]

Per : https://github.com/lxc/lxcfs/issues/24

We should back port the change from -master to vivid as it's currently breaking on my default build of vivid. Any more information needed, let me know !

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug. I'll SRU this asap.

[Serge Hallyn (serge-hallyn)]

Can you please show 'dpkg -l | grep lxcfs' on the vivid hosts you are using?

I want to make sure you're not using a ppa version, because the upstream commits which supposedly fixed this were changing commits which are not yet in the vivid archive package.

[Patrick Ancillotti (patricka)]

No problem.

Here's the output:

root@core01:~# dpkg -l | grep lxcfs
ii lxcfs 0.7-0ubuntu4 amd64 FUSE based filesystem for LXC

root@core01:~# apt-cache show lxcfs
Package: lxcfs
Priority: optional
Section: admin
Installed-Size: 141
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Version: 0.7-0ubuntu4
Depends: cgmanager, libc6 (>= 2.17), libcgmanager0 (>= 0.33), libdbus-1-3 (>= 1.0.2), libfuse2 (>= 2.8.1), libnih-dbus1 (>= 1.0.0), libnih1 (>= 1.0.0), init-system-helpers (>= 1.18~), lsb-base (>= 4.1+Debian11ubuntu7)
Filename: pool/main/l/lxcfs/lxcfs_0.7-0ubuntu4_amd64.deb
Size: 22474
MD5sum: abae7cd57011adf7ed6e79cffba8d9e1
SHA1: 31f72340cda3d43ebab25b0df6efb9d9bace21c0
SHA256: 58985dd9738827bf8c6146e8606c529c605c94c521127660d551899b80b12478
Description-en: FUSE based filesystem for LXC
 LXCFS provides a FUSE based filesystem to improve the LXC experience
 within the containers.
 .
 This filesystem offers both a cgroupfs-like view for use by
 unprivileged containers which wouldn't otherwise be allowed to mount
 cgroupfs. And a set of files that are meant to be bind-mounted over
 their /proc equivalent to make them cgroup-aware.
Description-md5: f3cabfc6ca3d3433720d9d811af950bd
Homepage: https://linuxcontainers.org
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 9m

Which looks to be: https://launchpad.net/ubuntu/+source/lxcfs/0.7-0ubuntu4

[Serge Hallyn (serge-hallyn)]

Yup, thank you. In that case the safest thing will probably to backport all the patches, rather than trying to port.

[Serge Hallyn (serge-hallyn)]

There are just too many commits to SRU them all for this bug. There are two which were specifically listed for this issue, however the first cannot be useful, and the second doesn't apply to the code (which didn't yet use direct_io)

So I'm afraid we have to debug this from scratch on vivid. I've still not managed to reproduce this myself. Can you please show which files you can reproduce this with, exactly what the error is, and perhaps a gdb backtrace of lxcfs at that point?

[Patrick Ancillotti (patricka)]

If you want to roll a patch. I can test it. I've managed to consistently recreate it on my vivid host. I'll see what I can muster re: backtrace and update shortly.

[Patrick Ancillotti (patricka)]

Reproduce for me :

- start lxcfs (current from package)
- start container (ubuntu 14.04 download)

lxcfs dumps.

Here's the backtrace:

root@core01:~# gdb /usr/bin/lxcfs
GNU gdb (Ubuntu 7.9-1ubuntu1) 7.9
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/lxcfs...(no debugging symbols found)...done.
(gdb) run -f -s -o allow_other /var/lib/lxcfs
Starting program: /usr/bin/lxcfs -f -s -o allow_other /var/lib/lxcfs
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
fuse: read too many bytes
fuse: writing device: Invalid argument
*** Error in `/usr/bin/lxcfs': free(): invalid next size (normal): 0x00005555557824b0 ***

Program received signal SIGABRT, Aborted.
0x00007ffff6d5d267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0 0x00007ffff6d5d267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007ffff6d5eeca in __GI_abort () at abort.c:89
#2 0x00007ffff6da0c53 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7ffff6eb91a8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff6da8c69 in malloc_printerr (ptr=<optimized out>, str=0x7ffff6eb9300 "free(): invalid next size (normal)", action=1) at malloc.c:4965
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3834
#5 0x00007ffff6dac89c in __GI___libc_free (mem=<optimized out>) at malloc.c:2950
#6 0x00007ffff753a681 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#7 0x00007ffff753f3de in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#8 0x00007ffff754784e in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#9 0x00007ffff754825b in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#10 0x00007ffff7544a9c in fuse_session_loop () from /lib/x86_64-linux-gnu/libfuse.so.2
#11 0x00007ffff753d1d8 in fuse_loop () from /lib/x86_64-linux-gnu/libfuse.so.2
#12 0x00007ffff754c97f in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#13 0x0000555555556468 in ?? ()
#14 0x00007ffff6d48a40 in __libc_start_main (main=0x555555556330, argc=6, argv=0x7fffffffe608, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe5f8) at libc-start.c:289
#15 0x00005555555564a9 in ?? ()
(gdb)

If I then to do the exact same thing, but using the current build from lxcfs on github, I don't get the same dump (or any dump).

[Serge Hallyn (serge-hallyn)]

> If you want to roll a patch. I can test it. I've managed to consistently
> recreate it on my vivid host. I'll see what I can muster re: backtrace
> and update shortly.

If you can come up with a pach that would be great.

We currently have 3 lxcfs bugs in vivid fixed in wily, and I *am* wondering
whether we could push the wily package into vivid under the lxc MRE, but
I'm not sure that's possible, and if you come up with a clean patch then
its SRU could probably be completed before we come to agreement about MRE.

[Serge Hallyn (serge-hallyn)]

(I will unfortunately be out thursday and friday)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Serge Hallyn (serge-hallyn)]

Note that rather than SRU this patch we're hpoing that bug 1454862 (and MRE to backport 0.9 into vivid) will happen.