ubuntu-14.04.04: fail to run systemtap test suites

[bugproxy]

---Problem Description---
ubuntu-14.04.04: fail to run systemtap test suites

---uname output---
Linux u140404 4.2.0-21-generic #25~14.04.1-Ubuntu SMP Thu Dec 3 13:55:42 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = power8

As per launchpad feature systemtap now supported and enabled on 14.04.04
- https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1511347

Here are the steps followed to test systemtap -
1. Get 14.04.04 guest VM

2. apt-get update; apt-get install dpkg-dev; apt-get build-dep systemtap

3. setup repo
codename=$(lsb_release -c | awk '{print $2}')
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename} main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

4. apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01 5E0577F2

5. apt-get update

6. devel packages
apt-get install linux-image-`uname -r`-dbgsym
apt-get install linux-headers-4.2.0-24-generic

7. Install required dependencies
 - dejagnu, elfutils-*

8. apt-get source systemtap

9. appy systemtap fixes
see BZ - https://bugzilla.linux.ibm.com/show_bug.cgi?id=131220

root@u140404:~/patch# ll
total 32
drwxr-xr-x 2 root root 4096 Jan 7 03:36 ./
drwx------ 8 root root 4096 Jan 7 03:55 ../
-rw-r--r-- 1 root root 6763 Jan 7 03:36 systemtap-1
-rw-r--r-- 1 root root 3094 Jan 7 03:35 systemtap-1511347.diff
-rw-r--r-- 1 root root 3472 Jan 7 03:36 systemtap-2
-rw-r--r-- 1 root root 5864 Jan 7 03:36 systemtap-3

cd /root/systemtap-2.3
apply above patch in below order
patch -p1 < /root/patch/systemtap-1511347.diff
patch -p1 < /root/patch/systemtap-1
patch -p1 < /root/patch/systemtap-2
patch -p1 < /root/patch/systemtap-3

10. ./configure; make -j10; make install

11. make check -------------------------------------------> failed
root@u140404:~/systemtap-2.3# make check
/bin/bash ./git_version.sh -k -s . -o git_version.h
make check-recursive
make[1]: Entering directory `/root/systemtap-2.3'
Making check in .
make[2]: Entering directory `/root/systemtap-2.3'
  CXX stap-session.o
  CXXLD stap
make check-local
make[3]: Entering directory `/root/systemtap-2.3'
SRCDIR=`cd .; pwd`; \
        PWD=`pwd`; \
 make -C testsuite check SYSTEMTAP_RUNTIME=$SRCDIR/runtime SYSTEMTAP_TAPSET=$SRCDIR/tapset LD_LIBRARY_PATH=$LD_LIBRARY_PATH${LD_LIBRARY_PATH:+:}$PWD/lib-elfutils:$PWD/lib-elfutils/systemtap SYSTEMTAP_PATH=$PWD SYSTEMTAP_INCLUDES=$PWD/includes RUNTESTFLAGS="" PKGLIBDIR="/usr/local/libexec/systemtap";
make[4]: Entering directory `/root/systemtap-2.3/testsuite'
Run "make check" or "make installcheck".
make check-DEJAGNU check-local
make[5]: Entering directory `/root/systemtap-2.3/testsuite'
srcdir='.'; export srcdir; \
 EXPECT=expect; export EXPECT; \
 if /bin/bash -c ""env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/root/systemtap-2.3/runtime SYSTEMTAP_TAPSET=/root/systemtap-2.3/tapset LD_LIBRARY_PATH=/root/systemtap-2.3/lib-elfutils:/root/systemtap-2.3/lib-elfutils/systemtap CRASH_LIBDIR=/usr/local/lib/systemtap PATH=/root/systemtap-2.3:$PATH SYSTEMTAP_PATH=/root/systemtap-2.3 SYSTEMTAP_INCLUDES=/root/systemtap-2.3/includes PKGLIBDIR=/usr/local/libexec/systemtap ./execrc runtest" --version" > /dev/null 2>&1; then \
   exit_status=0; l='systemtap'; for tool in $l; do \
     if "env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/root/systemtap-2.3/runtime SYSTEMTAP_TAPSET=/root/systemtap-2.3/tapset LD_LIBRARY_PATH=/root/systemtap-2.3/lib-elfutils:/root/systemtap-2.3/lib-elfutils/systemtap CRASH_LIBDIR=/usr/local/lib/systemtap PATH=/root/systemtap-2.3:$PATH SYSTEMTAP_PATH=/root/systemtap-2.3 SYSTEMTAP_INCLUDES=/root/systemtap-2.3/includes PKGLIBDIR=/usr/local/libexec/systemtap ./execrc runtest" --tool $tool --tool_opts \'\' --srcdir $srcdir ; \
     then :; else exit_status=1; fi; \
   done; \
 else echo "WARNING: could not find '"env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/root/systemtap-2.3/runtime SYSTEMTAP_TAPSET=/root/systemtap-2.3/tapset LD_LIBRARY_PATH=/root/systemtap-2.3/lib-elfutils:/root/systemtap-2.3/lib-elfutils/systemtap CRASH_LIBDIR=/usr/local/lib/systemtap PATH=/root/systemtap-2.3:$PATH SYSTEMTAP_PATH=/root/systemtap-2.3 SYSTEMTAP_INCLUDES=/root/systemtap-2.3/includes PKGLIBDIR=/usr/local/libexec/systemtap ./execrc runtest"'" 1>&2; :;\
 fi; \
 exit $exit_status
/bin/bash: line 4: env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/root/systemtap-2.3/runtime SYSTEMTAP_TAPSET=/root/systemtap-2.3/tapset LD_LIBRARY_PATH=/root/systemtap-2.3/lib-elfutils:/root/systemtap-2.3/lib-elfutils/systemtap CRASH_LIBDIR=/usr/local/lib/systemtap PATH=/root/systemtap-2.3:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games SYSTEMTAP_PATH=/root/systemtap-2.3 SYSTEMTAP_INCLUDES=/root/systemtap-2.3/includes PKGLIBDIR=/usr/local/libexec/systemtap ./execrc runtest: No such file or directory
make[5]: *** [check-DEJAGNU] Error 1
make[5]: Leaving directory `/root/systemtap-2.3/testsuite'
make[4]: *** [check-am] Error 2
make[4]: Leaving directory `/root/systemtap-2.3/testsuite'
make[3]: *** [check-local] Error 2
make[3]: Leaving directory `/root/systemtap-2.3'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/root/systemtap-2.3'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/root/systemtap-2.3'
make: *** [check] Error 2
root@u140404:~/systemtap-2.3#

Manually ran cmd instead of make check -
cd testsuite
root@u140404:~/systemtap-2.3/testsuite# cat x
env XDG_DATA_DIRS= SYSTEMTAP_SYNC=1 LANG=C SYSTEMTAP_TESTREMOTES= SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/root/systemtap-2.3/runtime SYSTEMTAP_TAPSET=/root/systemtap-2.3/tapset LD_LIBRARY_PATH=/root/systemtap-2.3/lib-elfutils:/root/systemtap-2.3/lib-elfutils/systemtap CRASH_LIBDIR=/usr/local/lib/systemtap PATH=/root/systemtap-2.3:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games SYSTEMTAP_PATH=/root/systemtap-2.3 SYSTEMTAP_INCLUDES=/root/systemtap-2.3/includes PKGLIBDIR=/usr/local/libexec/systemtap ./execrc runtest

root@u140404:~/systemtap-2.3/testsuite# ./x
WARNING: No tool specified
Test Run By root on Thu Jan 21 13:56:33 2016
Native configuration is powerpc64le-unknown-linux-gnu

  === tests ===

Schedule of variations:
    unix

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
Using ./config/unix.exp as tool-and-target-specific interface file.
kernel location: /boot/vmlinux-4.2.0-21-generic
kernel version: 4.2.0-21-generic
systemtap location: /root/systemtap-2.3/stap
systemtap version: version 2.3/0.158, Debian version 2.3-1ubuntu1.2 (trusty
gcc location: /usr/bin/gcc
gcc version: gcc (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4

**** failed systemtap kernel-devel smoke test:

In file included from /root/systemtap-2.3/runtime/linux/task_finder.c:17:0,
                 from /root/systemtap-2.3/runtime/linux/runtime.h:188,
                 from /root/systemtap-2.3/runtime/runtime.h:24,
                 from /tmp/stapoLzh94/stap_79fa3140436d1423447aa39801d1b0c9_872_src.c:24:
/root/systemtap-2.3/runtime/linux/task_finder2.c: In function '__stp_call_mmap_callbacks_with_addr':
/root/systemtap-2.3/runtime/linux/task_finder2.c:703:24: error: 'struct file' has no member named 'f_dentry'
   dentry = vma->vm_file->f_dentry;
                        ^
/root/systemtap-2.3/runtime/linux/task_finder2.c: In function '__stp_utrace_attach_match_filename':
/root/systemtap-2.3/runtime/linux/task_finder2.c:821:11: error: incompatible types when assigning to type 'uid_t' from type 'kuid_t'
  tsk_euid = task_euid(tsk);
           ^
/root/systemtap-2.3/runtime/linux/task_finder2.c: In function '__stp_call_mmap_callbacks_for_task':
/root/systemtap-2.3/runtime/linux/task_finder2.c:1206:42: error: 'struct file' has no member named 'f_dentry'
        vma_cache_p->dentry = vma->vm_file->f_dentry;
                                          ^
/root/systemtap-2.3/runtime/linux/task_finder2.c: In function 'stap_start_task_finder':
/root/systemtap-2.3/runtime/linux/task_finder2.c:1711:12: error: incompatible types when assigning to type 'uid_t' from type 'kuid_t'
   tsk_euid = task_euid(tsk);
            ^
In file included from /root/systemtap-2.3/runtime/linux/runtime.h:194:0,
                 from /root/systemtap-2.3/runtime/runtime.h:24,
                 from /tmp/stapoLzh94/stap_79fa3140436d1423447aa39801d1b0c9_872_src.c:24:
/root/systemtap-2.3/runtime/sym.c: In function '_stp_snprint_addr':
/root/systemtap-2.3/runtime/sym.c:567:4: error: implicit declaration of function 'preempt_enable_no_resched' [-Werror=implicit-function-declaration]
    preempt_enable_no_resched();
    ^
In file included from include/linux/sched.h:56:0,
                 from ./arch/powerpc/include/asm/elf.h:12,
                 from include/linux/elf.h:4,
                 from include/linux/module.h:15,
                 from /root/systemtap-2.3/runtime/linux/runtime.h:14,
                 from /root/systemtap-2.3/runtime/runtime.h:24,
                 from /tmp/stapoLzh94/stap_79fa3140436d1423447aa39801d1b0c9_872_src.c:24:
/root/systemtap-2.3/runtime/transport/control.c: In function '_stp_ctl_write_cmd':
include/linux/cred.h:358:25: error: incompatible types when initializing type 'uid_t' using type 'kuid_t'
 #define current_euid() (current_cred_xxx(euid))
                         ^
/root/systemtap-2.3/runtime/transport/control.c:41:15: note: in expansion of macro 'current_euid'
  uid_t euid = current_euid();
               ^
In file included from /root/systemtap-2.3/runtime/linux/print.c:17:0,
                 from /root/systemtap-2.3/runtime/print.c:17,
                 from /root/systemtap-2.3/runtime/runtime_context.h:22,
                 from /tmp/stapoLzh94/stap_79fa3140436d1423447aa39801d1b0c9_872_src.c:46:
/root/systemtap-2.3/runtime/transport/transport.c: In function '_stp_transport_init':
/root/systemtap-2.3/runtime/transport/transport.c:344:11: error: incompatible types when assigning to type 'uid_t' from type 'kuid_t'
  _stp_uid = current_uid();
           ^
/root/systemtap-2.3/runtime/transport/transport.c:345:11: error: incompatible types when assigning to type 'gid_t' from type 'kgid_t'
  _stp_gid = current_gid();
           ^
cc1: all warnings being treated as errors
make[1]: *** [/tmp/stapoLzh94/stap_79fa3140436d1423447aa39801d1b0c9_872_src.o] Error 1
make: *** [_module_/tmp/stapoLzh94] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

**** aborting testing.

root@u140404:~/systemtap-2.3/testsuite#

== Comment: #2 - Kalpana Shetty <email address hidden> - 2016-01-21 13:59:42 ==
few other cmds as well failing -
root@u140404:~# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 95 library script(s) using 42752virt/32768res/5952shr/26048data kb, in 160usr/0sys/170real ms.
WARNING: cannot find module kernel debuginfo: No DWARF information found [man warning::debuginfo]
semantic error: while resolving probe point: identifier 'kernel' at /usr/local/share/systemtap/tapset/linux/vfs.stp:768:18
        source: probe vfs.read = kernel.function("vfs_read")
                                 ^

semantic error: no match
semantic error: while resolving probe point: identifier 'vfs' at <input>:1:7
        source: probe vfs.read {printf("read performed\n"); exit()}
                      ^

Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 65536virt/35776res/7552shr/27008data kb, in 10usr/0sys/6real ms.
Pass 2: analysis failed. [man error::pass2]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
root@u140404:~#

root@u140404:~# stap -e 'probe oneshot { println("hello world") }'
semantic error: while resolving probe point: identifier 'oneshot' at <input>:1:7
        source: probe oneshot { println("hello world") }
                      ^

semantic error: probe point mismatch (alternatives: __nd_syscall __nfs __scheduler __signal __tcpmib __vm _linuxmib _nd_syscall _signal _sunrpc _syscall _vfs begin begin(number) end end(number) error error(number) generic ioblock ioblock_trace ioscheduler ioscheduler_trace ipmib irq_handler kernel kprobe kprocess linuxmib module(string) nd_syscall netdev netfilter never nfs nfsd perf process process(number) process(string) procfs procfs(string) scheduler scsi signal socket softirq stap staprun sunrpc syscall tcp tcpmib timer tty udp vfs vm workqueue): identifier 'oneshot' at :1:7
        source: probe oneshot { println("hello world") }
                      ^

Pass 2: analysis failed. [man error::pass2]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

== Comment: #4 - Hemant K. Shaw <email address hidden> - 2016-01-22 06:50:17 ==
This fails on x86 as well. Here is the o/p of a simple kprobe test :

The systemtap version is :
 Systemtap translator/driver (version 2.3/0.158, Debian version 2.3-1ubuntu1.3 (trusty))

$ sudo stap -ve 'probe kernel.function("_do_fork") { printf("hit\n"); }'
Pass 1: parsed user script and 95 library script(s) using 26904virt/19216res/4640shr/15180data kb, in 90usr/0sys/97real ms.
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 268420virt/167580res/126684shr/41468data kb, in 760usr/90sys/857real ms.
Pass 3: translated to C into "/tmp/stapwvyY7W/stap_96747bc38104f19f4e295204b6a5d176_873_src.c" using 268420virt/167704res/126808shr/41468data kb, in 10usr/0sys/6real ms.
In file included from /usr/share/systemtap/runtime/linux/task_finder.c:17:0,
                 from /usr/share/systemtap/runtime/linux/runtime.h:188,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapwvyY7W/stap_96747bc38104f19f4e295204b6a5d176_873_src.c:24:
/usr/share/systemtap/runtime/linux/task_finder2.c: In function ?__stp_call_mmap_callbacks_with_addr?:
/usr/share/systemtap/runtime/linux/task_finder2.c:703:24: error: ?struct file? has no member named ?f_dentry?
   dentry = vma->vm_file->f_dentry;
                        ^
/usr/share/systemtap/runtime/linux/task_finder2.c: In function ?__stp_utrace_attach_match_filename?:
/usr/share/systemtap/runtime/linux/task_finder2.c:821:11: error: incompatible types when assigning to type ?uid_t? from type ?kuid_t?
  tsk_euid = task_euid(tsk);
           ^
/usr/share/systemtap/runtime/linux/task_finder2.c: In function ?__stp_call_mmap_callbacks_for_task?:
/usr/share/systemtap/runtime/linux/task_finder2.c:1206:42: error: ?struct file? has no member named ?f_dentry?
        vma_cache_p->dentry = vma->vm_file->f_dentry;
                                          ^
/usr/share/systemtap/runtime/linux/task_finder2.c: In function ?stap_start_task_finder?:
/usr/share/systemtap/runtime/linux/task_finder2.c:1711:12: error: incompatible types when assigning to type ?uid_t? from type ?kuid_t?
   tsk_euid = task_euid(tsk);
            ^
In file included from /usr/share/systemtap/runtime/linux/runtime.h:194:0,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapwvyY7W/stap_96747bc38104f19f4e295204b6a5d176_873_src.c:24:
/usr/share/systemtap/runtime/sym.c: In function ?_stp_snprint_addr?:
/usr/share/systemtap/runtime/sym.c:567:4: error: implicit declaration of function ?preempt_enable_no_resched? [-Werror=implicit-function-declaration]
    preempt_enable_no_resched();
    ^
In file included from include/linux/sched.h:56:0,
                 from include/linux/ptrace.h:5,
                 from include/linux/ftrace.h:13,
                 from include/linux/kprobes.h:42,
                 from /usr/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapwvyY7W/stap_96747bc38104f19f4e295204b6a5d176_873_src.c:24:
/usr/share/systemtap/runtime/transport/control.c: In function ?_stp_ctl_write_cmd?:
include/linux/cred.h:358:25: error: incompatible types when initializing type ?uid_t? using type ?kuid_t?
 #define current_euid() (current_cred_xxx(euid))
                         ^
/usr/share/systemtap/runtime/transport/control.c:41:15: note: in expansion of macro ?current_euid?
  uid_t euid = current_euid();
               ^
In file included from /usr/share/systemtap/runtime/linux/print.c:17:0,
                 from /usr/share/systemtap/runtime/print.c:17,
                 from /usr/share/systemtap/runtime/runtime_context.h:22,
                 from /tmp/stapwvyY7W/stap_96747bc38104f19f4e295204b6a5d176_873_src.c:43:
/usr/share/systemtap/runtime/transport/transport.c: In function ?_stp_transport_init?:
/usr/share/systemtap/runtime/transport/transport.c:344:11: error: incompatible types when assigning to type ?uid_t? from type ?kuid_t?
  _stp_uid = current_uid();
           ^
/usr/share/systemtap/runtime/transport/transport.c:345:11: error: incompatible types when assigning to type ?gid_t? from type ?kgid_t?
  _stp_gid = current_gid();
           ^
cc1: all warnings being treated as errors
make[1]: *** [/tmp/stapwvyY7W/stap_96747bc38104f19f4e295204b6a5d176_873_src.o] Error 1
make: *** [_module_/tmp/stapwvyY7W] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_96747bc38104f19f4e295204b6a5d176_873.ko" in 4460usr/400sys/5342real ms.
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

Kernel version :
4.2.0-26-generic #31~14.04.1-Ubuntu SMP Wed Jan 20 14:26:00 UTC 2016 i686 i686 i686 GNU/Linux

It looks like there is a mismatch between the kernel and the systemtap version. Need to upgrade systemtap. For ubuntu 14.04.03, systemtap 2.3 worked because it has a older kernel and older systemtap. Its no more the case now. The systemtap version needs to be upgraded.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1537125/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Steve Langasek (vorlon)]

> 9. apply systemtap fixes
> see BZ - https://bugzilla.linux.ibm.com/show_bug.cgi?id=131220

Please test with the actual systemtap package from trusty-proposed (2.3-1ubuntu1.3), instead of manually patching the source.

> make check

This is not being run at build time because it is too intrusive on the build environment, as described at <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=526957>. But it should have been part of the autopkgtests for this package, and those autopkgtests should be run for each new kernel. ("Should", but is not the case today.)

Please note that systemtap is not supported by Canonical except as the systemtap-sdt-dev package which is used as a build-dependency for various other supported packages. While it would be acceptable per the Ubuntu archive policy to include a new upstream version of systemtap into trusty-updates, the regression testing for such an update must include tests of the reverse-dependencies (ust and php5).

So we intend to decline this SRU for the 14.04.4 release. I have also marked bug #1511347 "verification-failed" based on the results of this testing with the 14.04.4 kernel. If IBM believes it useful to proceed with that previous SRU of systemtap, so that it's enabled for compatibility with systems running the (Canonical-supported) 14.04.3 kernel, we can reverse that and still release the SRU.

[Steve Langasek (vorlon)]

> For ubuntu 14.04.03, systemtap 2.3 worked because it has a older kernel and older
> systemtap. Its no more the case now. The systemtap version needs to be upgraded.

I have just tested with the systemtap 2.9 package presently in xenial, and the following test case fails the same way there as with systemtap 2.3 in trusty:

# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 110 library script(s) using 55872virt/47296res/6912shr/37504data kb, in 260usr/0sys/259real ms.
semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/linux/vfs.stp:882:18
        source: probe vfs.read = kernel.function("vfs_read")
                                 ^

semantic error: no match

semantic error: while resolving probe point: identifier 'vfs' at <input>:1:7
        source: probe vfs.read {printf("read performed\n"); exit()}
                      ^

semantic error: no match

Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 57024virt/49792res/8064shr/38656data kb, in 0usr/10sys/8real ms.
Pass 2: analysis failed. [man error::pass2]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

It does not appear that an update to a newer systemtap version is going to be of any use on 14.04.

You also had the following warning in your test case output:

WARNING: cannot find module kernel debuginfo: No DWARF information found
[man warning::debuginfo]

This warning is present when using systemtap 2.3, it is not present when using systemtap 2.8 and 2.9. So *this* problem is fixed in later versions of systemtap, but we would need a test case that actually works correctly with later versions of systemtap before anyone could even entertain backporting.

[Frank Ch. Eigler (fche)]

Hi -

> I have just tested with the systemtap 2.9 package presently in xenial,
> and the following test case fails the same way there as with systemtap
> 2.3 in trusty:
>
> # stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
> [...]
> semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/linux/vfs.stp:882:18
> source: probe vfs.read = kernel.function("vfs_read")
> ^
>
> [...]
> Pass 2: analysis failed. [man error::pass2]
> Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
>
> It does not appear that an update to a newer systemtap version is going
> to be of any use on 14.04.
> [...]

This message indicates not something wrong with systemtap, but
something wrong with the installation of kernel debugging symbols.
(See also the error::pass2 man page and the given README.Debian file.)

- FChE

[Steve Langasek (vorlon)]

On Tue, Jan 26, 2016 at 01:03:48AM -0000, Frank Ch. Eigler wrote:
> Hi -

> > I have just tested with the systemtap 2.9 package presently in xenial,
> > and the following test case fails the same way there as with systemtap
> > 2.3 in trusty:

> > # stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
> > [...]
> > semantic error: while resolving probe point: identifier 'kernel' at /usr/share/systemtap/tapset/linux/vfs.stp:882:18
> > source: probe vfs.read = kernel.function("vfs_read")
> > ^

> > [...]
> > Pass 2: analysis failed. [man error::pass2]
> > Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

> > It does not appear that an update to a newer systemtap version is going
> > to be of any use on 14.04.
> > [...]

> This message indicates not something wrong with systemtap, but
> something wrong with the installation of kernel debugging symbols.
> (See also the error::pass2 man page and the given README.Debian file.)

This is when following the directions in README.Debian.

# stap-prep ; echo $?
0
#

It's possible there's an error with the generation of the kernel debugging
symbols package itself, but this is following the directions for their
installation.

And with --vp 02 as suggested by the manpage:

# stap --vp 02 -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 109 library script(s) using
# 55552virt/43456res/6848shr/37248data kb, in 240usr/10sys/257real ms.
Attempting to extract kernel debuginfo build ID from
# /lib/modules/4.2.0-25-generic/build/vmlinux.id
Attempting to extract kernel debuginfo build ID from /sys/kernel/notes
blacklist regexps:
blfn:
^(atomic_notifier_call_chain|default_do_nmi|__die|die_nmi|do_debug|do_general_protection|do_int3|do_IRQ|do_page_fault|do_sparc64_fault|do_trap|dummy_nmi_callback|flush_icache_range|ia64_bad_break|ia64_do_page_fault|ia64_fault|io_check_error|mem_parity_error|nmi_watchdog_tick|notifier_call_chain|oops_begin|oops_end|program_check_exception|single_step_exception|sync_regs|unhandled_fault|unknown_nmi_error|xen_[gs]et_debugreg|xen_irq_.*|xen_.*_fl_direct.*|check_events|xen_adjust_exception_frame|xen_iret.*|xen_sysret64.*|test_ti_thread_flag.*|inat_get_opcode_attribute|system_call_after_swapgs|HYPERVISOR_[gs]et_debugreg|HYPERVISOR_event_channel_op|hash_64|hash_ptr|native_set_pte|.*raw_.*_lock.*|.*raw_.*_unlock.*|.*raw_.*_trylock.*|.*read_lock.*|.*read_unlock.*|.*read_trylock.*|.*write_lock.*|.*write_unlock.*|.*write_trylock.*|.*write_seqlock.*|.*write_sequnlock.*|.*spin_lock.*|.*spin_unlock.*|.*spin_trylock.*|.*spin_is_locked.*|rwsem_.*lock.*|.*mutex_.*lock.*|raw_.*|atomic_.*|atomic64_.*|get_bh|put_bh|.*apic.*|.*APIC.*|.*softirq.*|.*IRQ.*|.*_intr.*|__delay|.*kernel_text.*|get_current|current_.*|.*exception_tables.*|.*setup_rt_frame.*|.*preempt_count.*|preempt_schedule|special_mapping_.*|.*_pte_.*)$
blfn_ret: ^(do_exit|sys_exit|sys_exit_group)$
blfile: ^(kernel/kprobes\.c|arch/.*/kernel/kprobes\.c|.*/include/asm/io\.h|.*/include/asm/io_64\.h|.*/include/asm/bitops\.h|drivers/ide/ide-iops\.c|arch/.*/kernel/paravirt\.c|.*/include/asm/paravirt\.h|fs/seq_file\.c)$
blsection: ^(\...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-01-29 02:11 EDT-------
(In reply to comment #10)
> Please note that systemtap is not supported by Canonical except as the
> systemtap-sdt-dev package which is used as a build-dependency for various
> other supported packages.

Is that Canonical's support statement for systemtap across all supported architectures?

> While it would be acceptable per the Ubuntu
> archive policy to include a new upstream version of systemtap into
> trusty-updates, the regression testing for such an update must include tests
> of the reverse-dependencies (ust and php5).
>
> So we intend to decline this SRU for the 14.04.4 release. I have also
> marked bug #1511347 "verification-failed" based on the results of this
> testing with the 14.04.4 kernel. If IBM believes it useful to proceed with
> that previous SRU of systemtap, so that it's enabled for compatibility with
> systems running the (Canonical-supported) 14.04.3 kernel, we can reverse
> that and still release the SRU.

Yes, please -- we would want to ensure systemtap v2.3 is enabled for 14.04 GA kernel on powerpc.

[Steve Langasek (vorlon)]

On Fri, Jan 29, 2016 at 07:19:58AM -0000, bugproxy wrote:
> ------- Comment From <email address hidden> 2016-01-29 02:11 EDT-------
> (In reply to comment #10)
> > Please note that systemtap is not supported by Canonical except as the
> > systemtap-sdt-dev package which is used as a build-dependency for various
> > other supported packages.

> Is that Canonical's support statement for systemtap across all supported
> architectures?

Yes.

> > While it would be acceptable per the Ubuntu
> > archive policy to include a new upstream version of systemtap into
> > trusty-updates, the regression testing for such an update must include tests
> > of the reverse-dependencies (ust and php5).

> > So we intend to decline this SRU for the 14.04.4 release. I have also
> > marked bug #1511347 "verification-failed" based on the results of this
> > testing with the 14.04.4 kernel. If IBM believes it useful to proceed with
> > that previous SRU of systemtap, so that it's enabled for compatibility with
> > systems running the (Canonical-supported) 14.04.3 kernel, we can reverse
> > that and still release the SRU.

> Yes, please -- we would want to ensure systemtap v2.3 is enabled for
> 14.04 GA kernel on powerpc.

Ok, we'll push the systemtap 2.3 through.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-01-29 09:40 EDT-------
Test results from Ravi:
----------------------------

Systemtap 2.3 on Ubuntu 14.04.03 with 3.13 kernel and proposed enabled
works fine on both ppc64el and on x86. Though, as mentioned in first
comment, "make check" fails on ppc64el. But we observed that "make check"
is also failing on x86.

DETAILS:
Systemtap 2.3 is working fine on ppc64el but with tweak. systemtap is not
able to find kernel debuginfo even if it's installed. We renamed
/usr/lib/debug/boot/vmlinux-3.13.0-76-generic
to
/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug
to make stap2.3 working on ppc64el.

$ sudo stap -ve 'probe kernel.function("do_fork") { printf("hit\n"); }'
[sudo] password for ravi:
Pass 1: parsed user script and 95 library script(s) using 42112virt/32064res/5952shr/25344data kb, in 180usr/10sys/229real ms.
Multiple 'sys_ni_syscall' symbols found.Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 247744virt/171840res/105152shr/65472data kb, in 710usr/110sys/1976real ms.
Pass 3: using cached /home/ravi/.systemtap/cache/31/stap_31fa92e1f4e1c009931ae3b7c69c27c1_934.c
Pass 4: using cached /home/ravi/.systemtap/cache/31/stap_31fa92e1f4e1c009931ae3b7c69c27c1_934.ko
Pass 5: starting run.
$

But "make check" on ppc64el is failing.
$ apt-get source systemtap
$ cd systemtap-2.3/
$ make check ======> This results to same error mentioned in first comment.

ppc64el machine details:

$ cat /etc/issue
Ubuntu 14.04.3 LTS \n \l ========> with -proposed enabled

$ uname -a
Linux ubuntu-140403-qemu-proposed 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:57:34 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

$ stap --version ========> We installed systemtap/trusty-proposed
Systemtap translator/driver (version 2.3/0.158, Debian version 2.3-1ubuntu1.3 (trusty))

We observed that Systemtap 2.3 is working fine on x86 as well.

$ sudo stap -ve 'probe kernel.function("do_fork") { printf("hit\n"); }'
[sudo] password for ravi:
Pass 1: parsed user script and 95 library script(s) using 26540virt/16348res/2128shr/14816data kb, in 90usr/0sys/92real ms.
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 246140virt/151204res/112420shr/39244data kb, in 660usr/600sys/1733real ms.
Pass 3: using cached /home/ravi/.systemtap/cache/81/stap_811d223965dcd0a0a692262dc773f020_879.c
Pass 4: using cached /home/ravi/.systemtap/cache/81/stap_811d223965dcd0a0a692262dc773f020_879.ko
Pass 5: starting run.
$

And, "make check" on x86 is also failing with same error mentioned in first comment.

x86 machine details:

$ cat /etc/issue
Ubuntu 14.04.3 LTS \n \l ========> with -proposed enabled

$ uname -a
Linux ubuntu-140403-qemu 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:58:41 UTC 2016 i686 i686 i686 GNU/Linux

$stap --version ========> We installed systemtap/trusty-proposed
Systemtap translator/driver (version 2.3/0.158, Debian version 2.3-1ubuntu1.3 (trusty))

Regards,
Ravi

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-15 05:55 EDT-------
The issue with ubuntu powerpc is, it stores stripped kernel image
as vmlinu*x* under /boot/; and stap fails to find debuginfo from this
file. Stap uses this files as main elf and tries to find debuginfo
file having ".debug" extension i.e. vmlinux-RELEASE.debug

debuginfo file on ubuntu is installed as vmlinux-RELEASE (without
.debug extension). So stap fails to find the file.

Detailed description of the same is available here:
https://sourceware.org/ml/systemtap/2007-q4/msg00085.html

From link:
"Fedora/RHEL kernel packaging on ppc does use a plain stripped vmlinux
file to boot from, but installs this as /boot/vmlinuz-RELEASE."

Renaming vmlinux to vmlinuz can solve the issue.

- Ravi

[Frank Ch. Eigler (fche)]

Ravi, systemtap can pass through the $SYSTEMTAP_DEBUGINFO_PATH variable to elfutils, which controls where debuginfo files are sought. If you can get a working $SYSTEMTAP_DEBUGINFO_PATH set by hand, we can plop that definition into systemtap proper.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-15 11:06 EDT-------
Hi fche,

I think setting variable won't work in this case. Still I tried with that on ubuntu poweprc. Here is what I got:

===========

# export SYSTEMTAP_DEBUGINFO_PATH=/usr/lib/debug/boot

# echo $SYSTEMTAP_DEBUGINFO_PATH
/usr/lib/debug/boot

# strace -o out -e open ./stap -v probe.stp
Pass 1: parsed user script and 95 library script(s) using 41792virt/32064res/5952shr/25408data kb, in 180usr/10sys/188real ms.
WARNING: cannot find module kernel debuginfo: No DWARF information found [man warning::debuginfo]
semantic error: while resolving probe point: identifier 'kernel' at probe.stp:1:7
source: probe kernel.function("do_fork")
^

semantic error: no match
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 66304virt/35072res/7488shr/26368data kb, in 0usr/0sys/6real ms.
Pass 2: analysis failed. [man error::pass2]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

# cat out | grep vmlinu
open("/boot/vmlinux-3.13.0-76-generic", O_RDONLY) = 3
open("/lib/modules/3.13.0-76-generic/build/vmlinux.id", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/debug/boot/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/debug/boot/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/debug/boot/vmlinux-3.13.0-76-generic.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
#

===========

If stap (which internally uses elfutils) finds /boot/vmlinux file, and debuginfo
is not present in it, it uses it as "main elf" and looks for debuginfo file with ".debug"
extension(i.e. vmlinux-RELEASE.debug). Ubuntu installs debuginfo without any
extension (i.e. vmlinux-RELEASE).

If it does not able to find vmlinux from /boot/ it goes on continue looking for fie on
other paths without ".debug" extension. And it finds proper file. RHEL/Fedora on
powerpc uses /boot/vmlinuz name without actually compressing it. Hence stap
works in these distros.

In short, setting SYSTEMTAP_DEBUGINFO_PATH won't change existing functionality.
It will only change list of paths where to look for debuginfo file.

[Launchpad Janitor (janitor)]

This bug was fixed in the package elfutils - 0.165-3ubuntu1

---------------
elfutils (0.165-3ubuntu1) xenial; urgency=medium

  * Fix finding the debug info for Ubuntu kernels (Mark Wielaard). LP: #1537125.

 -- Matthias Klose <email address hidden> Fri, 19 Feb 2016 16:57:49 +0100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-02-23 01:13 EDT-------
Systemtap (which uses Elfutils internally) was not able to find kernel debuginfo file in Ubuntu ppc even if it's installed properly. I sent a patch that solves this issue. This patch has been accepted and pushed in elfutils master.

Here is the link for the same:
https://git.fedorahosted.org/cgit/elfutils.git/patch/?id=7802e6e57d48189e339b4ab40189eb44d8123559

I'm backporting this patch to Ubuntu 14.04, 15.04 and 16.04.

(Note that, this patch does not solve make check error. This needs to be addressed separately.)

------- Comment From <email address hidden> 2016-02-23 05:48 EDT-------
(In reply to comment #22)

>
> I'm backporting this patch to Ubuntu 14.04, 15.04 and 16.04.
>

Sorry, my bad. I meant Ubuntu 14.04.4, 15.10 and 16.04

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2016-02-23 05:44 EDT-------

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2016-02-23 05:46 EDT-------

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2016-02-23 05:47 EDT-------

[Mark J. Wielaard (3y9m2vcw-ll9d-fkzsxrqg)]

On Wed, Feb 24, 2016 at 07:57:55AM -0000, Launchpad Bug Tracker wrote:
> This bug was fixed in the package elfutils - 0.165-3ubuntu1
>
> ---------------
> elfutils (0.165-3ubuntu1) xenial; urgency=medium
>
> * Fix finding the debug info for Ubuntu kernels (Mark Wielaard). LP:
> #1537125.

Just a note that the patch that went upstream is slightly different.
It is a bit more efficient by removing duplicate searches.

https://git.fedorahosted.org/cgit/elfutils.git/patch/?id=7802e6e57d48189e339b4ab40189eb44d8123559

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-01 03:09 EDT-------
Hi Mark,

> > * Fix finding the debug info for Ubuntu kernels (Mark Wielaard). LP:
> > #1537125.
>
> Just a note that the patch that went upstream is slightly different.
> It is a bit more efficient by removing duplicate searches.
>

This "slightly different" means change find_debuginfo_path such that it will look for both kind of files - without and with .debug suffix instead of calling find_debuginfo_path twice. Right?
Please let me know if you are pointing to something else.

- Ravi

[Mark J. Wielaard (3y9m2vcw-ll9d-fkzsxrqg)]

On Tue, Mar 01, 2016 at 08:09:59AM -0000, bugproxy wrote:
> ------- Comment From <email address hidden> 2016-03-01 03:09 EDT-------
> Hi Mark,
>
> > > * Fix finding the debug info for Ubuntu kernels (Mark Wielaard). LP:
> > > #1537125.
> >
> > Just a note that the patch that went upstream is slightly different.
> > It is a bit more efficient by removing duplicate searches.
> >
>
> This "slightly different" means change find_debuginfo_path such that it will look for both kind of files - without and with .debug suffix instead of calling find_debuginfo_path twice. Right?

Yes. Specifically, this is the patch that got committed
upstream: https://git.fedorahosted.org/cgit/elfutils.git/commit/?id=7802e6e57d48189e339b4ab40189eb44d8123559

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-21 03:51 EDT-------
Canonical,

I can see backported patch is picked up for 16.04. Can you please pick up patches for ubuntu 14.04 and 15.10 as well.

Regards,
Ravi

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-29 01:12 EDT-------
Hi,
I tested systemtap with latest Ubuntu.
Systemtap was not able to find kernel debuginfo ,only this is fixed and the make check error and other failures needs to be fixed yet.

systemtap-2.9# stap -ve 'probe kernel.function("_do_fork") { printf("hit\n"); }'
Pass 1: parsed user script and 110 library script(s) using 60224virt/48320res/7552shr/37504data kb, in 170usr/0sys/170real ms.
Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 100352virt/89408res/8448shr/77632data kb, in 740usr/90sys/852real ms.
Pass 3: translated to C into "/tmp/stapCuLav9/stap_b1fb74cd193f3b5cfd54555602532397_1032_src.c" using 100352virt/89920res/8960shr/77632data kb, in 20usr/90sys/105real ms.
In file included from /usr/share/systemtap/runtime/print.c:17:0,
from /usr/share/systemtap/runtime/runtime_context.h:22,
from /tmp/stapCuLav9/stap_b1fb74cd193f3b5cfd54555602532397_1032_src.c:50:
/usr/share/systemtap/runtime/linux/print.c: In function ?_stp_print_kernel_info?:
/usr/share/systemtap/runtime/linux/print.c:242:20: error: ?struct module? has no member named ?module_core?
THIS_MODULE->module_core,
^
/usr/share/systemtap/runtime/linux/print.c:243:37: error: ?struct module? has no member named ?core_size?
(unsigned long) (THIS_MODULE->core_size - THIS_MODULE->core_text_size)/
^
/usr/share/systemtap/runtime/linux/print.c:243:62: error: ?struct module? has no member named ?core_text_size?
(unsigned long) (THIS_MODULE->core_size - THIS_MODULE->core_text_size)/
^
/usr/share/systemtap/runtime/linux/print.c:244:44: error: ?struct module? has no member named ?core_text_size?
(unsigned long) (THIS_MODULE->core_text_size)/1024,
^
scripts/Makefile.build:258: recipe for target '/tmp/stapCuLav9/stap_b1fb74cd193f3b5cfd54555602532397_1032_src.o' failed
make[1]: *** [/tmp/stapCuLav9/stap_b1fb74cd193f3b5cfd54555602532397_1032_src.o] Error 1
Makefile:1396: recipe for target '_module_/tmp/stapCuLav9' failed
make: *** [_module_/tmp/stapCuLav9] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_b1fb74cd193f3b5cfd54555602532397_1032.ko" in 6380usr/230sys/7666real ms.
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

------- Comment From <email address hidden> 2016-03-29 01:14 EDT-------
- - - -kernel version- - -

root@ubuntu:~/systemtap-2.9# uname -a
Linux ubuntu 4.4.0-15-generic #31-Ubuntu SMP Fri Mar 18 19:06:23 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

[Dan Streetman (ddstreet)]

> /usr/share/systemtap/runtime/linux/print.c:242:20: error: ?struct module? has no member named ?module_core?
> THIS_MODULE->module_core,
...
> root@ubuntu:~/systemtap-2.9# uname -a
> Linux ubuntu 4.4.0-15-generic #31-Ubuntu SMP Fri Mar 18 19:06:23 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

that is being fixed in bug 1557673.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-05 03:19 EDT-------
Hi,
I tested systemtap on latest ubuntu16.04 ,make check error and other failures are not yet fixed.

Any update on this? I see in bug 1557673 ,that this bug was fixed in the package systemtap - 2.9-2ubuntu2.

root@ubuntu:~/systemtap-2.9# dpkg-query -l systemtap
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii systemtap 2.9-2ubuntu2 ppc64el instrumentation system for Linux

root@ubuntu:~/systemtap-2.9# uname -a
Linux ubuntu 4.4.0-16-generic #32-Ubuntu SMP Thu Mar 24 22:31:14 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

root@ubuntu:~/systemtap-2.9# dpkg-query -l systemtap
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii systemtap 2.9-2ubuntu2 ppc64el instrumentation system for Linux
root@ubuntu:~/systemtap-2.9# uname -a
Linux ubuntu 4.4.0-16-generic #32-Ubuntu SMP Thu Mar 24 22:31:14 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
tack(backtrace()); exit()}'# stap --all-modules -e 'probe timer.profile {print_st
In file included from /usr/share/systemtap/runtime/transport/control.c:14:0,
from /usr/share/systemtap/runtime/transport/transport.c:76,
from /usr/share/systemtap/runtime/linux/print.c:17,
from /usr/share/systemtap/runtime/print.c:17,
from /usr/share/systemtap/runtime/runtime_context.h:22,
from /tmp/stapFn1X26/stap_2fb5bd44bc9e9b5ee5e03bdf4b545935_3605_src.c:170:
/usr/share/systemtap/runtime/transport/symbols.c: In function ?_stp_module_update_self?:
/usr/share/systemtap/runtime/transport/symbols.c:243:44: error: ?struct module? has no member named ?symtab?
if (attr->address == (unsigned long) mod->symtab)
^
/usr/share/systemtap/runtime/transport/symbols.c:245:9: error: ?struct module? has no member named ?num_symtab?
mod->num_symtab * sizeof(mod->symtab[0]);
^
/usr/share/systemtap/runtime/transport/symbols.c:245:34: error: ?struct module? has no member named ?symtab?
mod->num_symtab * sizeof(mod->symtab[0]);
^
scripts/Makefile.build:258: recipe for target '/tmp/stapFn1X26/stap_2fb5bd44bc9e9b5ee5e03bdf4b545935_3605_src.o' failed
make[1]: *** [/tmp/stapFn1X26/stap_2fb5bd44bc9e9b5ee5e03bdf4b545935_3605_src.o] Error 1
Makefile:1396: recipe for target '_module_/tmp/stapFn1X26' failed
make: *** [_module_/tmp/stapFn1X26] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

[Dan Streetman (ddstreet)]

> I tested systemtap on latest ubuntu16.04 ,make check error and other failures are not yet fixed.

it works fine for me, although i'm on x86 so you may be seeing a ppc64-specific failure?

# uname -a
Linux systemtap 4.4.0-17-generic #33-Ubuntu SMP Tue Mar 29 17:17:28 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

# dpkg -l systemtap
...
ii systemtap 2.9-2ubuntu2 amd64 instrumentation system for Linux

# pull-lp-source systemtap xenial
...

# ls systemtap*dsc
systemtap_2.9-2ubuntu2.dsc
# cd systemtap-2.9
# ./configure
# make check
...

completes successfully.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-07 02:36 EDT-------
Hi,

Ubuntu14.04.4, debuginfo issue is still not fixed.

root@ubuntu:~/systemtap-2.3-14721/bin# uname -a
Linux ubuntu 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Wed Mar 16 17:53:38 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

root@ubuntu:~/systemtap-2.3-14721/bin# stap --version
Systemtap translator/driver (version 2.3/0.158, Debian version 2.3-1ubuntu1.4 (trusty))
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBSQLITE3 NSS TR1_UNORDERED_MAP NLS

root@ubuntu:~/systemtap-2.3-14721/bin#./stap -ve 'probe kernel.function("do_fork") { printf("hit\n"); }'
Pass 1: parsed user script and 95 library script(s) using 42752virt/32768res/5952shr/26048data kb, in 160usr/0sys/160real ms.
WARNING: cannot find module kernel debuginfo: No DWARF information found [man warning::debuginfo]
semantic error: while resolving probe point: identifier 'kernel' at <input>:1:7
source: probe kernel.function("do_fork") { printf("hit\n"); }
^

semantic error: no match
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 65664virt/35072res/7168shr/27072data kb, in 20usr/80sys/97real ms.
Pass 2: analysis failed. [man error::pass2]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-07 06:41 EDT-------
This bug affects Ubuntu 14.04.4, 15.10 and 16.04. Ravi has posted fixes/backports for all the three releases. I think Canonical has fixed up the patches only for 16.04 at this point and is yet to pick up the fixes for the other releases.

Can you confirm that systemtap works properly on 16.04?

[Tom Jakubowski (tjakubowski)]

I see that symtab compile error when building a systemtap module which uses userspace stack traces. A minimal example:

    probe process.syscall {
      print_usyms(ubacktrace());
    }

Yields the following results when building:

username@mysecrethost:~/stap$ sudo stap -v -c /bin/ls clock.stp
Pass 1: parsed user script and 111 library script(s) using 116848virt/44096res/6504shr/37808data kb, in 110usr/20sys/131real ms.
Pass 2: analyzed script: 1 probe(s), 5 function(s), 1 embed(s), 0 global(s) using 117640virt/45008res/6632shr/38600data kb, in 0usr/0sys/3real ms.
Pass 3: translated to C into "/tmp/stapxZlvnX/stap_ff2e2515c14e3a3266f293b9620dcd18_2586_src.c" using 117904virt/46404res/7584shr/38864data kb, in 10usr/70sys/80real ms.
In file included from /usr/share/systemtap/runtime/transport/control.c:14:0,
                 from /usr/share/systemtap/runtime/transport/transport.c:76,
                 from /usr/share/systemtap/runtime/linux/print.c:17,
                 from /usr/share/systemtap/runtime/print.c:17,
                 from /usr/share/systemtap/runtime/runtime_context.h:22,
                 from /tmp/stapxZlvnX/stap_ff2e2515c14e3a3266f293b9620dcd18_2586_src.c:120:
/usr/share/systemtap/runtime/transport/symbols.c: In function ‘_stp_module_update_self’:
/usr/share/systemtap/runtime/transport/symbols.c:243:44: error: ‘struct module’ has no member named ‘symtab’
    if (attr->address == (unsigned long) mod->symtab)
                                            ^
/usr/share/systemtap/runtime/transport/symbols.c:245:9: error: ‘struct module’ has no member named ‘num_symtab’
      mod->num_symtab * sizeof(mod->symtab[0]);
         ^
/usr/share/systemtap/runtime/transport/symbols.c:245:34: error: ‘struct module’ has no member named ‘symtab’
      mod->num_symtab * sizeof(mod->symtab[0]);
                                  ^
scripts/Makefile.build:258: recipe for target '/tmp/stapxZlvnX/stap_ff2e2515c14e3a3266f293b9620dcd18_2586_src.o' failed
make[1]: *** [/tmp/stapxZlvnX/stap_ff2e2515c14e3a3266f293b9620dcd18_2586_src.o] Error 1
Makefile:1403: recipe for target '_module_/tmp/stapxZlvnX' failed
make: *** [_module_/tmp/stapxZlvnX] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_ff2e2515c14e3a3266f293b9620dcd18_2586.ko" in 440usr/100sys/882real ms.
Pass 4: compilation failed. [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

[Tom Jakubowski (tjakubowski)]

That's on xenial, 16.04, systemtap 2.9-2ubuntu2 and uname -a:

Linux mysecrethost 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-08-08 02:35 EDT-------
This is a duplicate of:
https://bugs.launchpad.net/ubuntu/+bug/1568689

Canonical is yet to pick up the required patches for systemtap.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.