ubuntu_lttng_smoke_test failed on Artful: Kernel tracer not available

[Po-Hsu Lin]

Similar to bug 1677126, ubuntu_lttng_smoke_test failed on Artful too.

 == lttng smoke test list kernel events ==
 Error: Unable to list kernel events: Kernel tracer not available
 Error: Command error
 FAILED (lttng list --kernel)
 FAILED (lttng list --kernel more output expected)

 == lttng smoke test trace open/close system calls ==
 Session test-kernel-session created.
 Traces will be written in /tmp/lttng-kernel-trace-2107-session
 PASSED (lttng create)
 Error: Event open: Kernel tracer not available (channel channel0, session test-kernel-session)
 Error: Event openat: Kernel tracer not available (channel channel0, session test-kernel-session)
 Error: Event close: Kernel tracer not available (channel channel0, session test-kernel-session)
 FAILED (lttng enable-event)
 Session test-kernel-session destroyed

 == lttng smoke test trace context switches ==
 Session test-kernel-session created.
 Traces will be written in /tmp/lttng-kernel-trace-2107-session
 PASSED (lttng create)
 Error: Event sched_switch: Kernel tracer not available (channel channel0, session test-kernel-session)
 FAILED (lttng enable-event)

Complete test result: http://pastebin.ubuntu.com/24924042/

[Colin Ian King (colin-king)]

Jun 23 10:34:38 ubuntu lttng-sessiond[3446]: Error: Unable to load required module lttng-ring-buffer-client-discard
Jun 23 10:34:38 ubuntu lttng-sessiond[3446]: Warning: No kernel tracer available
Jun 23 10:34:38 ubuntu sudo[4213]: pam_unix(sudo:session): session closed for user root

[Colin Ian King (colin-king)]

DKMS make.log for lttng-modules-2.9.0 for kernel 4.11.0-8-generic (x86_64)
Fri 23 Jun 11:40:59 BST 2017
make: Entering directory '/usr/src/linux-headers-4.11.0-8-generic'
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-ring-buffer-client-discard.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-ring-buffer-metadata-client.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-ring-buffer-client-overwrite.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-ring-buffer-client-mmap-discard.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-ring-buffer-client-mmap-overwrite.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-ring-buffer-metadata-mmap-client.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-clock.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-events.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-abi.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-probes.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-context.o
In file included from /var/lib/dkms/lttng-modules/2.9.0/build/lttng-abi.c:53:0:
/var/lib/dkms/lttng-modules/2.9.0/build/wrapper/kref.h: In function ‘lttng_kref_get’:
/var/lib/dkms/lttng-modules/2.9.0/build/wrapper/kref.h:39:24: error: passing argument 1 of ‘atomic_add_unless’ from incompatible pointer type [-Werror=incompatible-pointer-types]
  if (atomic_add_unless(&kref->refcount, 1, INT_MAX) != INT_MAX) {
                        ^
In file included from ./include/linux/tracepoint-defs.h:10:0,
                 from ./arch/x86/include/asm/msr.h:67,
                 from ./arch/x86/include/asm/processor.h:20,
                 from ./arch/x86/include/asm/cpufeature.h:4,
                 from ./arch/x86/include/asm/thread_info.h:52,
                 from ./include/linux/thread_info.h:25,
                 from ./arch/x86/include/asm/preempt.h:6,
                 from ./include/linux/preempt.h:80,
                 from ./include/linux/spinlock.h:50,
                 from ./include/linux/seqlock.h:35,
                 from ./include/linux/time.h:5,
                 from ./include/linux/stat.h:18,
                 from ./include/linux/module.h:10,
                 from /var/lib/dkms/lttng-modules/2.9.0/build/lttng-abi.c:40:
./include/linux/atomic.h:504:19: note: expected ‘atomic_t * {aka struct <anonymous> *}’ but argument is of type ‘refcount_t * {aka struct refcount_struct *}’
 static inline int atomic_add_unless(atomic_t *v, int a, int u)
                   ^~~~~~~~~~~~~~~~~
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-context-pid.o
  CC [M] /var/lib/dkms/lttng-modules/2.9.0/build/lttng-context-procname.o
cc1: some warnings being treated as errors
scripts/Makefile.build:294: recipe for target '/var/lib/dkms/lttng-modules/2.9.0/build/lttng-abi.o' failed
make[1]: *** [/var/lib/dkms/lttng-modules/2.9.0/build/lttng-abi.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Makefile:1526: recipe for target '_module_/var/lib/dkms/lttng-modules/2.9.0/build' failed
make: *** [_module_/var/lib/dkms/lttng-modules/2.9.0/build] Error 2
make: Leaving directory '/usr/src/linux-headers-4.11.0-8-generic'

[Colin Ian King (colin-king)]

With the correct upstream compat patches we still get issues.

  * Cherry picks to fix build isues with 4.11 and 4.12
    - [75833a8] Fix: update mm_vmscan instrumentation for kernel 4.11
    - [e52717e] Fix: update btrfs instrumentation for kernel 4.11
    - [52a391b] Fix: changes to the vm_op fault cb prototype in
                libringbuffer
    - [8a9f549] Fix: update scsi instrumentation for kernel 4.11
    - [678dd1c] Fix: timers cputime_t arguments replaced by ull in kernel
                4.11
    - [4c4d77d] Fix: atomic_add_unless() returns true/false rather than
                prior value
    - [7793231] Fix: kref changes for kernel 4.11
    - [673e9a0] Fix: update sched instrumentation for kernel 4.12
    - [e2833be] Fix: update ringbuffer for kernel 4.12
    - [5807b1a] Fix: update btrfs instrumentation for kernel 4.12
    - [d245707] Fix: update block instrumentation for kernel 4.12
    - [5d84966] Fix: update ftrace probe for kernel 4.12
    - [bc1fb1b] Fix: Build ftrace probe on kernels prior to 4.12

This works fine for 4.11 and passes the regression tests. However with 4.12 we get a DKMS build failure:

WARNING: "lttng_ftrace_destroy_private" [/var/lib/dkms/lttng-modules/2.9.0/build/lttng-tracer.ko] undefined!
WARNING: "lttng_ftrace_register" [/var/lib/dkms/lttng-modules/2.9.0/build/lttng-tracer.ko] undefined!
WARNING: "lttng_ftrace_unregister" [/var/lib/dkms/lttng-modules/2.9.0/build/lttng-tracer.ko] undefined!
WARNING: modpost: Found 1 section mismatch(es).

The reason is because of linux commits:

  ec19b85913486993d7d6f747beed1a711afd47d8
  bca6c8d0480a8aa5c86f8f416db96c71f6b79e29
  b5f081b563a6cdcb85a543df8c851951a8978275
  6e4443199e5354255e8a4c1e8e5cfc8ef064c3ce

but most specifically:

commit 92a68fa047ca5b8e1991af2d50b23ad9452613cd
Author: Steven Rostedt (VMware) <email address hidden>
Date: Fri Mar 31 19:21:41 2017 -0400

    ftrace: Move the function commands into the tracing directory

    As nothing outside the tracing directory uses the function command mechanism,
    I'm moving the prototypes out of the include/linux/ftrace.h and into the
    local kernel/trace/trace.h header. I plan on making them hook to the
    trace_array structure which is local to kernel/trace, and I do not want to
    expose it to the rest of the kernel. This requires that the command functions
    must also be local to tracing. But luckily nothing else uses them.

    Signed-off-by: Steven Rostedt (VMware) <email address hidden>

lttng seems to expect the symbols from the lttng lttng-ftrace.o, but these are dependent on the lttng-ftrace being built, which it is not because of:

commit 5d8496675c9e8ccf169871f4f72022df9677624a
Author: Michael Jeanson <email address hidden>
Date: Thu May 25 16:56:52 2017 -0400

    Fix: update ftrace probe for kernel 4.12

..which effectively disables it:

+ ifeq ($(ftrace),warn)
+ $(warning Files $(ftrace_dep) not found. Probe "ftrace" is disabled. Use full kernel source tree to enable it.)
+ ftrace =
+ endif # $(ftrace),warn

and implies we need the full kernel source to build it. Not so good. I'll reach out to upstream about this.

[Colin Ian King (colin-king)]

https://launchpadlibrarian.net/325600945/lttng-modules_2.9.0-1ubuntu1_2.9.0-1ubuntu2.diff.gz

[Colin Ian King (colin-king)]

https://launchpad.net/~colin-king/+archive/ubuntu/lttng-artful-lp1699755/+packages