systemtap doesn't work with HWE kernels

[Jorge Niedbalski]

[Environment]

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial

Linux juju-niedbalski-xenial-machine-12 4.8.0-46-generic #49~16.04.1-Ubuntu SMP Fri Mar 31 14:51:03 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[Description]

Xenial system with LTS-hwe xenial kernel enabled. Trying to run the following script:

$ sudo stap -e 'probe oneshot { println("hello world") }'

Raises the following errors:

In file included from /usr/share/systemtap/runtime/linux/runtime.h:204:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapjb3FJA/stap_06a893e40e8ca093be8e351d40e8d862_960_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h: In function ‘__access_process_vm_’:
/usr/share/systemtap/runtime/linux/access_process_vm.h:35:29: error: passing argument 1 of ‘get_user_pages’ makes integer from pointer without a cast [-Werror=int-conversion]
       ret = get_user_pages (tsk, mm, addr, 1, write, 1, &page, &vma);
                             ^
In file included from ./include/linux/pid_namespace.h:6:0,
                 from ./include/linux/ptrace.h:8,
                 from ./include/linux/ftrace.h:13,
                 from ./include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapjb3FJA/stap_06a893e40e8ca093be8e351d40e8d862_960_src.c:25:
./include/linux/mm.h:1315:6: note: expected ‘long unsigned int’ but argument is of type ‘struct task_struct *’
 long get_user_pages(unsigned long start, unsigned long nr_pages,
      ^
In file included from /usr/share/systemtap/runtime/linux/runtime.h:204:0,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapjb3FJA/stap_06a893e40e8ca093be8e351d40e8d862_960_src.c:25:
./include/linux/mm.h:1315:6: note: expected ‘struct page **’ but argument is of type ‘int’
 long get_user_pages(unsigned long start, unsigned long nr_pages,
      ^
In file included from /usr/share/systemtap/runtime/linux/runtime.h:204:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapjb3FJA/stap_06a893e40e8ca093be8e351d40e8d862_960_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h:35:54: error: passing argument 6 of ‘get_user_pages’ makes pointer from integer without a cast [-Werror=int-conversion]
       ret = get_user_pages (tsk, mm, addr, 1, write, 1, &page, &vma);
                                                      ^
In file included from ./include/linux/pid_namespace.h:6:0,
                 from ./include/linux/ptrace.h:8,
                 from ./include/linux/ftrace.h:13,
                 from ./include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapjb3FJA/stap_06a893e40e8ca093be8e351d40e8d862_960_src.c:25:
./include/linux/mm.h:1315:6: note: expected ‘struct vm_area_struct **’ but argument is of type ‘int’
 long get_user_pages(unsigned long start, unsigned long nr_pages,
      ^
In file included from /usr/share/systemtap/runtime/linux/runtime.h:204:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapjb3FJA/stap_06a893e40e8ca093be8e351d40e8d862_960_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h:35:13: error: too many arguments to function ‘get_user_pages’
       ret = get_user_pages (tsk, mm, addr, 1, write, 1, &page, &vma);
             ^
In file included from ./include/linux/pid_namespace.h:6:0,
                 from ./include/linux/ptrace.h:8,
                 from ./include/linux/ftrace.h:13,
                 from ./include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapjb3FJA/stap_06a893e40e8ca093be8e351d40e8d862_960_src.c:25:
./include/linux/mm.h:1315:6: note: declared here
 long get_user_pages(unsigned long start, unsigned long nr_pages,
      ^
In file included from /usr/share/systemtap/runtime/linux/runtime.h:204:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapjb3FJA/stap_06a893e40e8ca093be8e351d40e8d862_960_src.c:25:
/usr/share/systemtap/runtime/linux/access_process_vm.h:55:7: error: implicit declaration of function ‘page_cache_release’ [-Werror=implicit-function-declaration]
       page_cache_release (page);
       ^
In file included from /usr/share/systemtap/runtime/linux/print.c:17:0,
                 from /usr/share/systemtap/runtime/print.c:17,
                 from /usr/share/systemtap/runtime/runtime_context.h:22,
                 from /tmp/stapjb3FJA/stap_06a893e40e8ca093be8e351d40e8d862_960_src.c:55:
/usr/share/systemtap/runtime/transport/transport.c: In function ‘_stp_lock_inode’:
/usr/share/systemtap/runtime/transport/transport.c:506:19: error: ‘struct inode’ has no member named ‘i_mutex’
  mutex_lock(&inode->i_mutex);
                   ^
/usr/share/systemtap/runtime/transport/transport.c: In function ‘_stp_unlock_inode’:
/usr/share/systemtap/runtime/transport/transport.c:515:21: error: ‘struct inode’ has no member named ‘i_mutex’
  mutex_unlock(&inode->i_mutex);

[Rafael David Tinoco (rafaeldtinoco)]

In my case, for a simple process trace (from utrace/uprobe):

sudo stap -e 'probe process("ls").function("*").call { log (probefunc()." ".$$parms) }' -c 'ls -l'

Generates:

http://pastebin.ubuntu.com/24414785/

And you can clearly see incompatibility between systemtap runtime (used to create stap kernel modules):

----

static int
__access_process_vm_ (struct task_struct *tsk, unsigned long addr, void *buf,
              int len, int write,
              void (*writer) (struct vm_area_struct * vma,
                      struct page * page, unsigned long vaddr,
                      void *dst, void *src, int len),
              void (*reader) (struct vm_area_struct * vma,
                      struct page * page, unsigned long vaddr,
                      void *dst, void *src, int len))
{

...
      ret = get_user_pages (tsk, mm, addr, 1, write, 1, &page, &vma);

and kernel we're running in, with function prototypes declared at:

"/usr/src/linux-headers-4.8.0-48/include/linux/mm.h" as:

long get_user_pages(unsigned long start, unsigned long nr_pages,
                int write, int force, struct page **pages,
                struct vm_area_struct **vmas);
----

IMO, HWE kernels should be followed with new systemtap userland packages, for every HWE kernel release, specially for LTS versions.

By backporting yakkety systemtap I got everything working:

ii systemtap 3.0-7 amd64 instrumentation system for Linux
ii systemtap-common 3.0-7 all instrumentation system for Linux (common component)
ii systemtap-doc 3.0-7 all documentation and examples for SystemTap
ii systemtap-runtime 3.0-7 amd64 instrumentation system for Linux (runtime component)

http://pastebin.ubuntu.com/24414866/

[Rafael David Tinoco (rafaeldtinoco)]

Similar (cause/effect) to this:

https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1545330

[Rafael David Tinoco (rafaeldtinoco)]

Trusty is also broken for HWE kernels:

http://pastebin.ubuntu.com/24415883/

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andrea Righi (arighi)]

This particular problem has been addressed and fixed in bug #1830226, marking this one as a duplicate.

There's a separate problem with hwe kernels, I'll create a separate bug for that.