stap fails with 3.5/3.8 kernels in precise

[Chris J Arges]

[Impact]
* When using systemtap on Precise with a hwe kernel (3.5/3.8) systemtap probes runtime compiles fail.

[Test Case]
# echo 'probe timer.s(1) {exit();}' | stap -

* Expected Result:

script compiles and starts running. exits with 0.

* Actual Result:

In file included from /usr/share/systemtap/runtime/transport/transport.c:53:0,
                 from /usr/share/systemtap/runtime/print.c:18,
                 from /usr/share/systemtap/runtime/runtime.h:128,
                 from /tmp/stapEhR7Am/stap_17350b932f2d616d597e792fa8968eb5_720.c:59:
/usr/share/systemtap/runtime/transport/relay_v2.c:241:2: error: initialization from incompatible pointer type [-Werror]
/usr/share/systemtap/runtime/transport/relay_v2.c:241:2: error: (near initialization for ‘__stp_relay_callbacks.create_buf_file’) [-Werror]
In file included from /usr/share/systemtap/runtime/print.c:18:0,
                 from /usr/share/systemtap/runtime/runtime.h:128,
                 from /tmp/stapEhR7Am/stap_17350b932f2d616d597e792fa8968eb5_720.c:59:
/usr/share/systemtap/runtime/transport/transport.c: In function ‘_stp_get_root_dir’:
/usr/share/systemtap/runtime/transport/transport.c:386:8: error: ‘struct hlist_head’ has no member named ‘next’
In file included from /tmp/stapEhR7Am/stap_17350b932f2d616d597e792fa8968eb5_720.c:61:0:
/usr/share/systemtap/runtime/stat.c: In function ‘_stp_stat_get’:
/usr/share/systemtap/runtime/stat.c:213:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
/usr/share/systemtap/runtime/stat.c:213:2: note: each undeclared identifier is reported only once for each function it appears in
/usr/share/systemtap/runtime/stat.c: In function ‘_stp_stat_clear’:
/usr/share/systemtap/runtime/stat.c:248:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
cc1: all warnings being treated as errors
make[1]: *** [/tmp/stapEhR7Am/stap_17350b932f2d616d597e792fa8968eb5_720.o] Error 1
make: *** [_module_/tmp/stapEhR7Am] Error 2
Warning: make exited with status: 2
Pass 4: compilation failed. Try again with another '--vp 0001' option.

[Regression Potential]
* All three patches used to fix this: e14ac0e2, 52dea427, 56111fdc are in upsteam systemtap.
* I've tested this package on 3.5/3.8 and again on 3.2 to ensure it didn't break.

--

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: systemtap 1.6-1ubuntu1
ProcVersionSignature: Ubuntu 3.8.0-31.46~precise1-generic 3.8.13.8
Uname: Linux 3.8.0-31-generic x86_64
ApportVersion: 2.0.1-0ubuntu17.4
Architecture: amd64
Date: Wed Oct 9 07:55:33 2013
InstallationMedia: Ubuntu-Server 12.04.3 LTS "Precise Pangolin" - Release amd64 (20130820.2)
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: systemtap
UpgradeStatus: No upgrade log present (probably fresh install)

[Frank Ch. Eigler (fche)]

This problem was fixed in upstream systemtap in commit e14ac0e274c back in Apr. 2012, and included in subsequent releases 1.8, 2.0, 2.1, 2.2, and 2.3.

[Chris J Arges (arges)]

So this actually requires three patches to be backported to fix all issues:

e14ac0e2
/usr/share/systemtap/runtime/stat.c:213:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
/usr/share/systemtap/runtime/stat.c:248:2: error: ‘cpu_possible_map’ undeclared (first use in this function)

52dea427
/usr/share/systemtap/runtime/transport/transport.c:386:8: error: ‘struct hlist_head’ has no member named ‘next’

56111fdc
/usr/share/systemtap/runtime/transport/relay_v2.c:241:2: error: (near initialization for ‘__stp_relay_callbacks.create_buf_file’)

Going to do some more testing, and see how I can backport these for 1.6.

[Timo Juhani Lindfors (timo-lindfors)]

I would have backported these but I did not know that it was possible to upload new packages to precise since it has already been released. Can you maybe handle such backports in the future or give some guidance?

[Chris J Arges (arges)]

@timo-lindfors,
Yup I'll handle it. And yes it is possible to update packages through our SRU process:
https://wiki.ubuntu.com/StableReleaseUpdates
I already have a rough-draft backport, so I'll be cleaning that up shortly.

[Chris J Arges (arges)]

I've also tested Quantal/Raring/Saucy and they all seem to work just fine with the 3.5/3.8/3.11 kernels respectively.

[Chris J Arges (arges)]

Test fix here with debdiff:
http://people.canonical.com/~arges/lp1237390/

[Timo Juhani Lindfors (timo-lindfors)]

Chris, thank you! A few comments:

1) I don't know if I have advertised this already but if you care about systemtap in ubuntu I could really use some help. See https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/1144040

2) If new kernels are uploaded to old ubuntu releases would it be possible to also upload new systemtap releases there? Backporting will fix some issues but sometimes new kernels come with fully new features that an old systemtap won't be able to use.

[Brian Murray (brian-murray)]

Hello Chris, or anyone else affected,

Accepted systemtap into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/systemtap/1.6-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[katsu (katsukatsu-deactivatedaccount)]

verify condition: ubuntu 12.04.3(kernel-3.8) in virtualbox.

-------------------
[Test Case]
# echo 'probe timer.s(1) {exit();}' | stap - ; echo $?

* Actual Result:
In file included from /usr/share/systemtap/runtime/transport/transport.c:53:0,
                 from /usr/share/systemtap/runtime/print.c:18,
                 from /usr/share/systemtap/runtime/runtime.h:128,
                 from /tmp/stapTSxLrZ/stap_34d5fb5c9ba75ea41176182eabf9f52f_642.c:59:
/usr/share/systemtap/runtime/transport/relay_v2.c:241:2: error: initialization from incompatible pointer type [-Werror]
/usr/share/systemtap/runtime/transport/relay_v2.c:241:2: error: (near initialization for '__stp_relay_callbacks.create_buf_file') [-Werror]
In file included from /usr/share/systemtap/runtime/print.c:18:0,
                 from /usr/share/systemtap/runtime/runtime.h:128,
                 from /tmp/stapTSxLrZ/stap_34d5fb5c9ba75ea41176182eabf9f52f_642.c:59:
/usr/share/systemtap/runtime/transport/transport.c: In function '_stp_get_root_dir':
/usr/share/systemtap/runtime/transport/transport.c:386:8: error: 'struct hlist_head' has no member named 'next'
In file included from /tmp/stapTSxLrZ/stap_34d5fb5c9ba75ea41176182eabf9f52f_642.c:61:0:
/usr/share/systemtap/runtime/stat.c: In function '_stp_stat_get':
/usr/share/systemtap/runtime/stat.c:213:2: error: 'cpu_possible_map' undeclared (first use in this function)
/usr/share/systemtap/runtime/stat.c:213:2: note: each undeclared identifier is reported only once for each function it appears in
/usr/share/systemtap/runtime/stat.c: In function '_stp_stat_clear':
/usr/share/systemtap/runtime/stat.c:248:2: error: 'cpu_possible_map' undeclared (first use in this function)
cc1: all warnings being treated as errors
make[1]: *** [/tmp/stapTSxLrZ/stap_34d5fb5c9ba75ea41176182eabf9f52f_642.o] Error 1
make: *** [_module_/tmp/stapTSxLrZ] Error 2
Warning: make exited with status: 2
Pass 4: compilation failed. Try again with another '--vp 0001' option.
1
#
-------------------

enable precise-proposed

$ sudo apt-get install -t precise-proposed systemtap

upgrade systemtap-common 1.6-1ubuntu1 1.6-1ubuntu1.1
upgrade systemtap 1.6-1ubuntu1 1.6-1ubuntu1.1
upgrade systemtap-runtime 1.6-1ubuntu1 1.6-1ubuntu1.1

-------------------
[verification case]

$ uname -r
3.8.0-35-generic
$ sudo -i
# echo 'probe timer.s(1) {exit();}' | stap - ; echo $?

* Actual Result:
0
#
(script compiles and starts running. exits with 0.)
-------------------

good works & thank you.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemtap - 1.6-1ubuntu1.1

---------------
systemtap (1.6-1ubuntu1.1) precise; urgency=low

  * Backported the following changes to fix issues when running with the
    hwe kernels (3.5/3.8) (LP: #1237390)
    - fix-compile-problem-by-changing-stp_for_each_cpu-to.patch:
      [e14ac0e2] runtime/runtime.h: Replaced 'stp_for_each_cpu()' with
      'for_each_possible_cpu()'
    - runtime-autoconf-for-fs_supers-hlist-change.patch:
      [52dea427] runtime: autoconf for fs_supers hlist change
    - runtime-autoconf-for-relay-umode_t-change.patch:
      [56111fdc] runtime: autoconf for relay umode_t change
 -- Chris J Arges <email address hidden> Thu, 10 Oct 2013 12:11:53 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for systemtap has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.