systemtap test suite failed to build due to missing Kernel configs

[Naresh Kamboju]

systemtap test suite failed to build due to missing kernel configs.

with respect to previous bug, the investigation needs to be done.
systemtap fails to build for arndale-uprobe testing
https://bugs.launchpad.net/linaro-ci/+bug/1303785

However, it fails later on (using the script provided by Naresh):

kernel location:
kernel version: 3.14.0-linaro-arndale
systemtap location: /usr/local/bin/stap
systemtap version: version 2.5/0.157, non-git sources
gcc location: /usr/bin/gcc
gcc version: gcc (Ubuntu/Linaro 4.8.1-10ubuntu8) 4.8.1

**** failed systemtap kernel-devel smoke test:

Makefile:622: Cannot use CONFIG_CC_STACKPROTECTOR_REGULAR: -fstack-protector not supported by compiler
In file included from include/linux/kobject.h:21:0,
                 from include/linux/module.h:16,
                 from /usr/local/share/systemtap/runtime/linux/runtime.h:14,
                 from /usr/local/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapAd5hoM/stap_b8bd15d808f759433855a4956409904d_878_src.c:24:
include/linux/sysfs.h: In function 'sysfs_get_dirent':
include/linux/sysfs.h:449:2: error: pointer targets in passing argument 2 of 'kernfs_find_and_get' differ in signedness [-Werror=pointer-sign]
  return kernfs_find_and_get(parent, name);
  ^
In file included from include/linux/sysfs.h:15:0,
                 from include/linux/kobject.h:21,
                 from include/linux/module.h:16,
                 from /usr/local/share/systemtap/runtime/linux/runtime.h:14,
                 from /usr/local/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapAd5hoM/stap_b8bd15d808f759433855a4956409904d_878_src.c:24:
include/linux/kernfs.h:331:1: note: expected 'const char *' but argument is of type 'const unsigned char *'
 kernfs_find_and_get(struct kernfs_node *kn, const char *name)
 ^
In file included from /usr/local/share/systemtap/runtime/linux/task_finder.c:17:0,
                 from /usr/local/share/systemtap/runtime/linux/runtime.h:192,
                 from /usr/local/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapAd5hoM/stap_b8bd15d808f759433855a4956409904d_878_src.c:24:
/usr/local/share/systemtap/runtime/linux/task_finder2.c: In function '__stp_utrace_attach_match_filename':
/usr/local/share/systemtap/runtime/linux/task_finder2.c:821:11: error: incompatible types when assigning to type 'uid_t' from type 'kuid_t'
  tsk_euid = task_euid(tsk);
           ^
/usr/local/share/systemtap/runtime/linux/task_finder2.c: In function 'stap_start_task_finder':
/usr/local/share/systemtap/runtime/linux/task_finder2.c:1711:12: error: incompatible types when assigning to type 'uid_t' from type 'kuid_t'
   tsk_euid = task_euid(tsk);
            ^
In file included from /usr/local/share/systemtap/runtime/linux/runtime.h:198:0,
                 from /usr/local/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapAd5hoM/stap_b8bd15d808f759433855a4956409904d_878_src.c:24:
/usr/local/share/systemtap/runtime/sym.c: In function '_stp_snprint_addr':
/usr/local/share/systemtap/runtime/sym.c:567:4: error: implicit declaration of function 'preempt_enable_no_resched' [-Werror=implicit-function-declaration]
    preempt_enable_no_resched();
    ^
In file included from include/linux/sched.h:53:0,
                 from include/linux/ptrace.h:5,
                 from include/linux/ftrace.h:13,
                 from include/linux/kprobes.h:42,
                 from /usr/local/share/systemtap/runtime/linux/runtime.h:21,
                 from /usr/local/share/systemtap/runtime/runtime.h:24,
                 from /tmp/stapAd5hoM/stap_b8bd15d808f759433855a4956409904d_878_src.c:24:
/usr/local/share/systemtap/runtime/transport/control.c: In function '_stp_ctl_write_cmd':
include/linux/cred.h:333:25: error: incompatible types when initializing type 'uid_t' using type 'kuid_t'
 #define current_euid() (current_cred_xxx(euid))
                         ^
/usr/local/share/systemtap/runtime/transport/control.c:41:15: note: in expansion of macro 'current_euid'
  uid_t euid = current_euid();
               ^
In file included from /usr/local/share/systemtap/runtime/linux/print.c:17:0,
                 from /usr/local/share/systemtap/runtime/print.c:17,
                 from /usr/local/share/systemtap/runtime/runtime_context.h:22,
                 from /tmp/stapAd5hoM/stap_b8bd15d808f759433855a4956409904d_878_src.c:46:
/usr/local/share/systemtap/runtime/transport/transport.c: In function '_stp_transport_init':
/usr/local/share/systemtap/runtime/transport/transport.c:344:11: error: incompatible types when assigning to type 'uid_t' from type 'kuid_t'
  _stp_uid = current_uid();
           ^
/usr/local/share/systemtap/runtime/transport/transport.c:345:11: error: incompatible types when assigning to type 'gid_t' from type 'kgid_t'
  _stp_gid = current_gid();
           ^
cc1: all warnings being treated as errors
make[4]: *** [/tmp/stapAd5hoM/stap_b8bd15d808f759433855a4956409904d_878_src.o] Error 1
make[3]: *** [_module_/tmp/stapAd5hoM] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compilation failed. [man error::pass4]

**** aborting testing.

It seems related to kernel namespaces (CONFIG_USER_NS).

@Naresh: please investigate and fill another bug if needed.

[David Long (dave-long)]

CONFIG_USER_NS is not set by default in exynos_defconfig. It needs to be set. Problems can also arise if using an old version of systemtap.

Also note that exynos_defconfig also does not set kprobes or uprobes configs by default (unlike panda which does select kprobes by default). CONFIG_RELAY also still has to be added.

[Andrey Konovalov (andrey-konovalov)]

Should I add "CONFIG_USER_NS=y" to the linaro/configs/uprobes.conf fragment then?
Currently linaro/configs/uprobes.conf is:
CONFIG_RELAY=y
CONFIG_KPROBES=y
CONFIG_UPROBE_EVENT=y
CONFIG_KPROBES_SANITY_TEST=y
CONFIG_ARM_KPROBES_TEST=m

[David Long (dave-long)]

On 05/13/14 11:33, Andrey Konovalov wrote:
> Should I add "CONFIG_USER_NS=y" to the linaro/configs/uprobes.conf fragment then?
> Currently linaro/configs/uprobes.conf is:
> CONFIG_RELAY=y
> CONFIG_KPROBES=y
> CONFIG_UPROBE_EVENT=y
> CONFIG_KPROBES_SANITY_TEST=y
> CONFIG_ARM_KPROBES_TEST=m
>

Yes. I don't know how this became a dependency but something changed at
some point.

-dl

[David Long (dave-long)]

Using recent upstream kernel sources *and* systemtap sources I am able to build and run my systemtap test case without modifications. If you could send me your final config file, and tell me the repo/branch the source came from I'll repeat my experiment. That said, I have had non-repeatable failures with CONFIG_STACK_PROTECTOR_REGULAR set. Not sure what's going on there.

[Naresh Kamboju (naresh-kamboju)]

The below patch fix this error.

**** failed systemtap kernel-devel smoke test:
Makefile:622: Cannot use CONFIG_CC_STACKPROTECTOR_REGULAR:
-fstack-protector not supported by compiler

linux-linaro-tracking$ git diff linaro/configs/uprobes.conf
diff --git a/linaro/configs/uprobes.conf b/linaro/configs/uprobes.conf
index 7681683..75120db 100644
--- a/linaro/configs/uprobes.conf
+++ b/linaro/configs/uprobes.conf
@@ -3,3 +3,14 @@ CONFIG_KPROBES=y
 CONFIG_UPROBE_EVENT=y
 CONFIG_KPROBES_SANITY_TEST=y
 CONFIG_ARM_KPROBES_TEST=m
+CONFIG_NAMESPACES=y
+CONFIG_UTS_NS=y
+CONFIG_IPC_NS=y
+CONFIG_USER_NS=y
+CONFIG_PID_NS=y
+CONFIG_NET_NS=y
+CONFIG_HAVE_CC_STACKPROTECTOR=y
+# CONFIG_CC_STACKPROTECTOR is not set
+CONFIG_CC_STACKPROTECTOR_NONE=y
+# CONFIG_CC_STACKPROTECTOR_REGULAR is not set
+# CONFIG_CC_STACKPROTECTOR_STRONG is not set

[Naresh Kamboju (naresh-kamboju)]

Tree: linux-linaro-tracking
Branch : linux-linaro
Config:
linaro/configs/linaro-base.conf linaro/configs/distribution.conf linaro/configs/kvm-host.conf linaro/configs/xen.conf linaro/configs/uprobes.conf linaro/configs/arndale.conf linaro/configs/lt-arndale.conf

This config is same as kernel CI for arndale-uprobes
https://ci.linaro.org/jenkins/job/linux-linaro-tracking-ll/hwpack=arndale-uprobes,label=kernel_cloud/

Please suggest the missing parts linux-linaro-tracking tree if mainline kernel works for systemtap.
Please share your test results and kernel config for my better understanding.

[David Long (dave-long)]

On 05/21/14 05:12, Naresh Kamboju wrote:
> Tree: linux-linaro-tracking
> Branch : linux-linaro
> Config:
> linaro/configs/linaro-base.conf linaro/configs/distribution.conf linaro/configs/kvm-host.conf linaro/configs/xen.conf linaro/configs/uprobes.conf linaro/configs/arndale.conf linaro/configs/lt-arndale.conf
>
> This config is same as kernel CI for arndale-uprobes
> https://ci.linaro.org/jenkins/job/linux-linaro-tracking-ll/hwpack=arndale-uprobes,label=kernel_cloud/
>
> Please suggest the missing parts linux-linaro-tracking tree if mainline kernel works for systemtap.
> Please share your test results and kernel config for my better understanding.
>

I have been running a specific scenario in systemtap, provided to me a
long time ago by William Cohen:

> $ git clone git://sourceware.org/git/systemtap.git
> $ cd systemtap
> $ ./configure --prefix=$HOME/install
> $ make
> $ make install
> $ cd testsuite
> $ gcc ./systemtap.base/at_var.c -O2 -g -marm -lm -o at_var
> $ $HOME/install/bin/stap systemtap.base/at_var.stp -c ./at_var

This works with the upstream kernel plus Tixy's and Victor's patches.
Even without those patches it at least builds and starts to run.

I'm having trouble duplicating the results you showed in pastebin. In
particular the "runtest" command is not found. I found a "runtest.sh".
  If I put it in the install/bin directory as "runtest" it runs but dies
trying to process "--tool" as an argument to dirname and basename. It's
as if there must be another runtest command somewhere that processes
this option.

I'm attaching the config file I have been using, based on upstream
exynos_defconfig plus the necessary modifications for [ku]probes, and
including (for now) STACKPROTECTOR_REGULAR.

We should also probably be paying more attention to which version of the
compiler we're using too, especially with this STACKPROTECTOR stuff.
I'm using gcc 4.7.3.

-dl

[Andrey Konovalov (andrey-konovalov)]

On 05/21/2014 01:03 PM, Naresh Kamboju wrote:
> The below patch fix this error.
>
> **** failed systemtap kernel-devel smoke test:
> Makefile:622: Cannot use CONFIG_CC_STACKPROTECTOR_REGULAR:
> -fstack-protector not supported by compiler
>
>
> linux-linaro-tracking$ git diff linaro/configs/uprobes.conf
> diff --git a/linaro/configs/uprobes.conf b/linaro/configs/uprobes.conf
> index 7681683..75120db 100644
> --- a/linaro/configs/uprobes.conf
> +++ b/linaro/configs/uprobes.conf
> @@ -3,3 +3,14 @@ CONFIG_KPROBES=y
> CONFIG_UPROBE_EVENT=y
> CONFIG_KPROBES_SANITY_TEST=y
> CONFIG_ARM_KPROBES_TEST=m
> +CONFIG_NAMESPACES=y
> +CONFIG_UTS_NS=y
> +CONFIG_IPC_NS=y
> +CONFIG_USER_NS=y
> +CONFIG_PID_NS=y
> +CONFIG_NET_NS=y

- this has been addressed by commit:
https://git.linaro.org/kernel/configs.git/commitdiff/bf8bdf9ab4fed4adb2e8ad035d7bec44078b1ffa
(UTS_NS, IPC_NS, PID_NS, and NET_NS are enabled by default if CONFIG_NAMESPACES=y)

It has also been merged into linux-linaro tree.

> +CONFIG_HAVE_CC_STACKPROTECTOR=y
> +# CONFIG_CC_STACKPROTECTOR is not set
> +CONFIG_CC_STACKPROTECTOR_NONE=y
> +# CONFIG_CC_STACKPROTECTOR_REGULAR is not set
> +# CONFIG_CC_STACKPROTECTOR_STRONG is not set

- this is essentially selecting CC_STACKPROTECTOR_NONE instead of CC_STACKPROTECTOR_REGULAR.
I believe that this change is not needed: I've tried building and installing the current linux-linaro
kernel natively (on Arndale) using the current set of config fragments (with CONFIG_CC_STACKPROTECTOR_REGULAR=y),
and 'make installcheck' seems to be going OK (not completed atm; see the attachment).

[Andrey Konovalov (andrey-konovalov)]

When trying the "cross build the kernel deb packages, replace the packages in the hwpack with the ones just built, create the sdcard using l-m-c" method (same current linux-linaro tree and the same set of config fragments) I got an error related to scripts/recordmcount contents. What surprized me, is that in the cross build this file is:

../out/scripts/recordmcount: file format elf64-x86-64
architecture: i386:x86-64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000402648

- so it seems natural that systemtap had troubles reading this file.

In the native build it (of course) is:

# objdump -f /root/linux-linaro-tracking/scripts/recordmcount
/root/linux-linaro-tracking/scripts/recordmcount: file format elf32-littlearm
architecture: arm, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x00009bc1

Does it ring a bell?

[Andrey Konovalov (andrey-konovalov)]

> I got an error related to scripts/recordmcount contents
I ment the error when running 'make installcheck' in the systemtap directory on the target

[David Long (dave-long)]

On 05/21/14 15:47, Andrey Konovalov wrote:
> When trying the "cross build the kernel deb packages, replace the
> packages in the hwpack with the ones just built, create the sdcard using
> l-m-c" method (same current linux-linaro tree and the same set of config
> fragments) I got an error related to scripts/recordmcount contents. What
> surprized me, is that in the cross build this file is:
>
> ../out/scripts/recordmcount: file format elf64-x86-64
> architecture: i386:x86-64, flags 0x00000112:
> EXEC_P, HAS_SYMS, D_PAGED
> start address 0x0000000000402648
>
> - so it seems natural that systemtap had troubles reading this file.
>
> In the native build it (of course) is:
>
> # objdump -f /root/linux-linaro-tracking/scripts/recordmcount
> /root/linux-linaro-tracking/scripts/recordmcount: file format elf32-littlearm
> architecture: arm, flags 0x00000112:
> EXEC_P, HAS_SYMS, D_PAGED
> start address 0x00009bc1
>
> Does it ring a bell?
>

The scripts programs have to run on the host system, so they're going to
be built for whatever the host is. After cross-building I had to do a
"make scripts" on the target system to rebuild those for the native
architecture, prior to letting systemtap do module builds. I'm actually
not positive this is the way this is meant to be done, but it worked for me.

[Naresh Kamboju (naresh-kamboju)]

These are steps i follow to build/install/test systemtap on target.

# Get linaro kernel on target

# Build and install kernel having uprobes.conf

# make LOADADDR=XXXXXXX uImage
# make modules
# make install
# make headers_install
# make modules_install
#
# Boot with latest built image.

###############################################################################
# Install dep
apt-get install build-essential m4 dejagnu bzip2 wget gettext

# Download elfutils need for systemtap build.
wget --no-check-certificate https://fedorahosted.org/releases/e/l/elfutils/0.158/elfutils-0.158.tar.bz2
tar -xvf elfutils-0.158.tar.bz2

# Download systemtap
 wget http://people.linaro.org/~naresh.kamboju/systemtap.tar.gz
 tar -xvf systemtap.tar.gz
# git clone http://sourceware.org/git/systemtap.git
cd systemtap

# runtest.exp is need by testsuite
cp /usr/share/dejagnu/* . -a

# below tests are not running on ARM. so lets skip
mv testsuite/systemtap.base/poll_map.exp testsuite/systemtap.base/poll_map.exp.back
mv testsuite/systemtap.base/pr14546.exp testsuite/systemtap.base/pr14546.exp.back
mv testsuite/systemtap.base/pr10854.exp testsuite/systemtap.base/pr10854.exp.back

# Configure systemtap
./configure --prefix=/usr/local/ --with-elfutils=../elfutils-0.158

# Build and install systemtap
make all
make install

# Build and run systemtap testsuite.
make installcheck

# print each test case pass/fail status
cat testsuite/systemtap.sum

[Andrey Konovalov (andrey-konovalov)]

Below are the steps to make 'make installcheck' to run (most of) its tests - w/o rebuilding the kernel on the target.

In short:
1) 'make scripts' must be run on target to get the arm version of the recordmcount
2) the linux-image debug deb package needs to be installed on the target (for arndale the name is linux-image-<kernel version>-linaro-arndale-dbg_<kernel version>-linaro-arndale-<n>_armhf.deb).

Due to its size the _dbg package is not included into the hwpack. And the _dbg package is not uploaded to elsewhere. So I had to rebuild the deb packages myself to get all the deb packages I need and the corresponding uImage (couldn't use the hwpacks from snapshots.l.o).

The steps:
* cross build the packages and the kernel as usual (make uImage, make deb-pkg). Used the same config fragments as the linux-linaro CI job. Copy the uImage and the deb packages to the sd card preprogrammed for the board (used l-m-c and the most recent hwpack and the trusty developer rootfs from snapshots.l.o). Insert the card into the board, power it on.
* After getting to the boot propmt replace the uImage and install the kernel deb packages (I put the new uImage into the boot partition, and the deb packages under /root/deb/; the board is Arndale - hence the *.deb names):
    # install the kernel packages:
    dpkg -i --force-all deb/linux-headers-3.15-rc5-ll-linaro-arndale_3.15-rc5-ll-linaro-arndale-2_armhf.deb
    dpkg -i --force-all deb/linux-image-3.15-rc5-ll-linaro-arndale_3.15-rc5-ll-linaro-arndale-2_armhf.deb
    dpkg -i --force-all deb/linux-image-3.15-rc5-ll-linaro-arndale-dbg_3.15-rc5-ll-linaro-arndale-2_armhf.deb
    # mount the boot partition and replace the uImage:
    mount -t vfat /dev/mmcblk1p2 /mnt
    cp /mnt/uImage.new /mnt/uImage
    # reboot to the new kernel:
    shutdown -r now
* After reboot:
    # I had to set the date as the developer rootfs doesn't do that on its own:
    date 052314142014
    apt-get install git build-essential m4 dejagnu bzip2 wget gettext
    cd /usr/src/linux-headers-3.15-rc5-ll-linaro-arndale/
    make scripts
    # 'make scripts' fails, but scripts/recordmcount is built, and this is enough to make systemtap's 'make installcheck' happy
    cd ~
    wget --no-check-certificate https://fedorahosted.org/releases/e/l/elfutils/0.157/elfutils-0.157.tar.bz2
    tar -xvf elfutils-0.157.tar.bz2
    git clone --depth 1 -b release-2.5 git://sourceware.org/git/systemtap.git
    cd systemtap
    # then just follow the script by Naresh (see comment #12):
    cp /usr/share/dejagnu/* . -a
    mv testsuite/systemtap.base/poll_map.exp testsuite/systemtap.base/poll_map.exp.back
    mv testsuite/systemtap.base/pr14546.exp testsuite/systemtap.base/pr14546.exp.back
    mv testsuite/systemtap.base/pr10854.exp testsuite/systemtap.base/pr10854.exp.back
    ./configure --prefix=/usr/local/ --with-elfutils=../elfutils-0.157
    make all
    make install
    make installcheck

I got to "Running ./systemtap.base/target_set_thread.exp ...", and terminated the test here after waiting for quite a while.
Here is what I got on the console:
-----8<-----
Running ./systemtap.base/target_set.exp ...
FAIL: target_set - timeout (expect_target_set_pids)
Run...

[Andrey Konovalov (andrey-konovalov)]

Again, as the summary for comment #13
The first two things which needs to be done to make systemtap test suite to run on the target w/o rebuilding the kernel on the target are:
* the native versions of the scripts must be installed on the target
* the debug information (the linux-image-<kernel version>-linaro-<board name>-dbg_<kernel version>-linaro-<board name>-<n>_armhf.deb ) must be installed on the target

[Fathi Boudra (fboudra)]

I'll add the debug package in the hwpack. Naresh will update the test definition to run make scripts.

[Naresh Kamboju (naresh-kamboju)]

I have update my script to run " make scripts"

[David Long (dave-long)]

The problem I had booting turned out to be because I needed to remake the board.dtb file to go with my updated kernel.

The runtest problem I saw was fixed with the procedures recommended in #12 above.

I am now duplicating the original "signedness" part of the problem reported in the bug description. I am trying to understand why we don't see a warning for this in the normal kernel build.

[David Long (dave-long)]

The "signedness" error comes about because systemtap includes kernel header files which depend on the -Wno-pointer-sign compiler option. This is the default for the compiler, but is overidden by -Wall. In the kernel build both those options are used with the effect of restoring the default behavior. It looks like systemtap builds with only -Wall. I have temporarily hacked my way past this by changing the problematic kernel header file thusly:

diff --git a/include/linux/sysfs.h b/include/linux/sysfs.h
index 5ffaa34..a332b3d 100644
--- a/include/linux/sysfs.h
+++ b/include/linux/sysfs.h
@@ -454,7 +454,7 @@ static inline void sysfs_notify_dirent(struct kernfs_node *kn)
 static inline struct kernfs_node *sysfs_get_dirent(struct kernfs_node *parent,
                                                   const unsigned char *name)
 {
- return kernfs_find_and_get(parent, name);
+ return kernfs_find_and_get(parent, (char *) name);
 }

The proper fix needs to be in the systemtap config and/or makefiles instead.

I've run into a problem with scripts/basic/fixdep not getting rebuilt for ARM with a "make scripts". I had to go into that subdir and make it myself, after the "make scripts". This may only be because of the fact I've just been copying over the cross-built tree. There may be a better way to do that. Certainly a native build should be free from this problem (and other /scripts problems).

I'm currently having trouble with systemtap failing to find __stack_chk_fail and __stack_chk_guard. Since I see these global kernel symbols defined even without --stack-protect, and since I configured systemtap with --disable-ssp this seems odd.

[David Long (dave-long)]

After a lot of experimenting I've come to the conclusion the 4.8 gcc that's part of linaro-saucy-server-20140410-652.tar.gz does not compile the current linux-linaro-tracking kernel. Installing the 4.7 compiler on the target system made the native build work. I'd be curious if my experience is unique.

I'm getting the tests to run now, although I have many failures. I'm not certain yet if I've gotten the tests to compile with "-marm". If not that would explain failures in tests using the current uprobes.

[Naresh Kamboju (naresh-kamboju)]

After resolving make script issues.
systemtap test execution reported "missing arm kernel/module debuginfo"

kernel location:
kernel version: 3.15.0-rc5-linaro-arndale
systemtap location: /usr/local/bin/stap
systemtap version: version 2.6/0.158, non-git sources
gcc location: /usr/bin/gcc
gcc version: gcc (Ubuntu/Linaro 4.8.2-19ubuntu1) 4.8.2

**** failed systemtap kernel-debuginfo smoke test:

semantic error: while resolving probe point: identifier 'kernel' at /usr/local/share/systemtap/tapset/linux/syscalls2.stp:143:24
        source: probe __syscall.open = kernel.function("sys_open").call
                                       ^

semantic error: missing arm kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/3.15.0-rc5-linaro-arndale/build'
semantic error: while resolving probe point: identifier '__syscall' at :123:47
        source: probe syscall.open = __syscall.compat_open ?, __syscall.open
                                                              ^

semantic error: no match
Pass 2: analysis failed. [man error::pass2]
Number of similar error messages suppressed: 1.
Rerun with -v to see them.

**** aborting testing.

make[2]: Leaving directory `/root/systemtap/testsuite'

[Naresh Kamboju (naresh-kamboju)]

I have tried to install debug package and found below packages. The version of package is not matching with `uname -r`.

apt-cache search linux- | grep arndale | grep dbg
linux-image-3.14.0-1-linaro-arndale-dbgsym - Linux kernel debug image for version 3.14.0 on Samsung ARNDALE-based systems
linux-image-3.14.0-1-linaro-arndale-octa-dbgsym - Linux kernel debug image for version 3.14.0 on Samsung ARNDALE-based systems
linux-image-3.15.0-1-linaro-arndale-octa-dbgsym - Linux kernel debug image for version 3.15.0 on Samsung ARNDALE-based systems
linux-image-3.15.0-1-linaro-arndale-dbgsym - Linux kernel debug image for version 3.15.0 on Samsung ARNDALE-based systems
root@linaro-server:~# uname -r
3.15.0-rc5-linaro-arndale

[David Long (dave-long)]

After installing a couple packages (e.g.: "xz-utils"), and adding a lot of "-marm"s to test program compiles, I currently have the following summary from a test run (using systemtap from June 5):

                === systemtap Summary ===

# of expected passes 3738
# of unexpected failures 349
# of expected failures 287
# of unknown successes 1
# of known failures 51
# of untested testcases 204
# of unsupported tests 7

I still have a half dozen or so tests switched off to avoid hangs.

Tests pertaining to uprobes can be identified by the presence of at least one "process" keyword in the corresponding *.stp file. Based on this there are 181 test cases that use uprobes. Of these I currently see 29 experiencing failures (the actual reported numbers will differ as one *.stp file can report multiple failures). I also see many failures in tests not related to uprobes. In at least a couple cases these are due to hardcoded Intel assembly code in "asm()"s. There are undoubtably other examples of porting issues, and hopefully that repesents the vast majority of the issues. I am only focusing on uprobes-specific failures for now.

I think I see a way to conditionally add "-marm" in the *.exp files when the target is identified as ARM. This would hopefully make this change upstreamable (later to be enhanced when uprobes works for Thumb code and on 64-bit ARM).

[Naresh Kamboju (naresh-kamboju)]

Davelong tested results uploaded on dejazilla.
which provides the comparison of results

- https://web.elastic.org/~dejazilla/viewsummary.php?summary=%3D%27%3CCA%2BG9fYu1eXk%2B%2B9FiP1%2B7sKREA6jGZ3WFOZZ%2BTEfMuAH6oOELEg%40mail.gmail.com%3E%27
- https://web.elastic.org/~dejazilla/viewsummary.php?_offset=0&_limit=20&_sort=1A&summary=&age=&rg=&tool=&variant=&versions=&pass=&fail=&kpass=&kfail=&xpass=&xfail=&untested=&unresolved=&unsupported=&warning=&error=

[David Long (dave-long)]

We're down from 29 *.exp files using uprobes and generating failures, to 24. I think I've picked all the "low-hanging-fruit". The remaining problems are likely to take time to address. It looks like some tests are experiencing problems because they try to uprobe system library routines that were not built with -marm. Those can only be fixed by adding thumb support to uprobes.

[Chase Qi (chase-qi)]

Migrated to bugzilla: https://bugs.linaro.org/show_bug.cgi?id=525