Python should be compiled with USDT/dtrace user-space probes

[Trent Lloyd]

Python3.6 and later support being compiled with USDT/dtrace user-space probes, this allows tools such as perf and BPF utilities (bpfcc-tools) to do runtime profiling and tracing of Python execution.

Ubuntu should compile Python with this support to enable better debugging of Python applications in production. perf/bpf tools can be attached to running production processes without having to restart them under a debugger which may destroy the situation you are trying to observe. This would be particularly useful with OpenStack deployments for example.

The current python3.6/python3.7/python3.8 packages do not build with this support, because it requires a build-dependency on systemtap-sdt-dev and the option --with-dtrace to be passed at compile time.

systemtap-sdt-dev is not currently in main, however, it was previously approved for a MIR in Bug #1203590 - it seems the original packages this was required for (Source Package: ust) was demoted to universe at some point and the systemtap-sdt-dev dependency went with it. So I am hopeful it will be re-approved for main inclusion and allow this to be done

[Trent Lloyd (lathiat)]

Attaching debdiff showing how to enable build-time support. debian/control is generated for the python version by debian/control.in so we patch both files, then enable --with-dtrace in the rules.

At this stage this patch is blocked by the MIR for systemtap-sdt-dev, and I will also file this upstream in Debian for the same

[Trent Lloyd (lathiat)]

Adding patch (git format-patch) against upstream Debian cpython-team/python3 git repository (canonical home of debian/ for python3 packages)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1818778-python3.8-enable-usdt.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Trent Lloyd (lathiat)]

You can unsubscribe ubuntu-sponsors from this

[Seth Arnold (seth-arnold)]

The systemtap dependency may be easier to handle this time around than last time: we've slightly changed the rules about build-dependencies:

https://lists.ubuntu.com/archives/ubuntu-devel-announce/2016-April/001179.html

Be sure to pay attention to this point:

> - If you have a build-dependency which does not result in a runtime
> dependency, but *does* result in code being copied into the final
> package, you must ensure that this code copying is declared using the
> Built-Using header[5]. (This is an uncommon case; the vast majority of
> the affected packages are written in go, where our package toolchain
> handles this automatically today.)

Thanks

[Simon Quigley (tsimonq2)]

Unsubscribing the Ubuntu Sponsors Team as per the comment in #4.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.8 - 3.8.0~a4-3

---------------
python3.8 (3.8.0~a4-3) experimental; urgency=medium

  * Update symbols files.

 -- Matthias Klose <email address hidden> Wed, 15 May 2019 00:30:25 +0200

[Trent Lloyd (lathiat)]

Matthias: Thanks for the upload in python3.8; I have verified this is working as expected

can we get the same upload for python3.7 since thats the default and it's quite hard to use python3.8 really.. maybe by f* we'll have python3.8 but at least for e* i'd like to get the same in for python3.7

[Łukasz Zemczak (sil2100)]

Hello Trent, or anyone else affected,

Accepted python3.8 into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.8/3.8.0-1~19.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Trent, or anyone else affected,

Accepted python3.8 into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python3.8/3.8.0-3~19.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python3.8 - 3.8.0-3~19.04

---------------
python3.8 (3.8.0-3~19.04) disco-proposed; urgency=medium

  * SRU: LP: #1835737. Backport the final Python 3.8.0 release.

python3.8 (3.8.0-3) unstable; urgency=medium

  * python3.8-dev: Depend on zlib1g-dev, needed to link as an
    embedded interpreter.

python3.8 (3.8.0-2) unstable; urgency=medium

  * Fix a symlink to the shared debug library.
  * Install the python3.8d-embed pkg-config file.
  * Don't ship the python3-embed pkg-config file.

python3.8 (3.8.0-1) unstable; urgency=medium

  * Python 3.8.0 release.

python3.8 (3.8.0~rc1-3) unstable; urgency=medium

  * Apply proposed patch for issue 38368. LP: #1847036. Addresses: #941650.

python3.8 (3.8.0~rc1-2) unstable; urgency=medium

  * Python 3.8.0 release candidate 1.
  * Refresh patches.
  * Update symbols files.
  * Bump standards version.

python3.8 (3.8.0~b4-1) unstable; urgency=medium

  * Python 3.8.0 beta 4 release.

python3.8 (3.8.0~b3-1) unstable; urgency=medium

  * Python 3.8.0 beta 3 release.
  * Fix FTCBFS: (Helmut Grohne). Closes: #932077.
    + Build-Depends, when cross compiling we need python3.8.
    + Export CC, because dtrace needs it.
    + Honour DEB_BUILD_OPTIONS=nocheck more thoroughly.
    + Fix up sysconfigdata-name.diff.
  * Implement the native thread ids for the Hurd and KFreeBSD (Samuel
    Thibault). Closes: #931624.
  * Enable pgo/lto builds on arm64. Addresses: #934812.
  * Update symbols files.
  * Don't propagate lto flags to the _sysconfigdata module. Addresses: #934771.
  * d/patches/issue35998.diff: Disable TLS1.3 in the client on all platforms
    rather than just reducing the payload size (Michael Hudson-Doyle).

python3.8 (3.8.0~b2-5) unstable; urgency=high

  * Bump standards version.
  * Revert the KFreeBSD and Hurd fix for PyThread_get_thread_native_id,
    and use the implementation from Python3.7. Mark the symbol as linux only.
  * Don't encode the MACHDEP into the _sysconfigdata file name.

python3.8 (3.8.0~b2-4) unstable; urgency=medium

  * Fix the LTO build.

python3.8 (3.8.0~b2-3) unstable; urgency=medium

  * Define PY_HAVE_THREAD_NATIVE_ID for KFreeBSD and Hurd builds.
  * Fix issue #37504, building the texinfo docs (Dmitry Shachnev).
  * Enable LTO/PGO builds on some architectures.

python3.8 (3.8.0~b2-1) experimental; urgency=medium

  * Python 3.8.0 beta 2 release.
  * Refresh patches.
  * Update the symbols files.
  * Don't build the texinfo documentation, fails. See issue #37504.

python3.8 (3.8.0~b1-2) experimental; urgency=medium

  * Fix include links in python3.8-dbg.

python3.8 (3.8.0~b1-1) experimental; urgency=medium

  * Python 3.8.0 beta 1 release.
  * Refresh patches.
  * Add new importlib dependencies to python3-minimal.
  * Update symbols files.

python3.8 (3.8.0~a4-3) experimental; urgency=medium

  * Update symbols files.

python3.8 (3.8.0~a4-2) experimental; urgency=medium

  * Update to 20190514 from the trunk.
  * Fix interpreter name in the -dbg autopkg tests.
  * Enable dtrace/USDT probe support (--with-dtrace) and build-dep on
    systemtap-sdt-dev (Trent Lloyd). LP: #1818778.

python3.8 (3.8.0~a4-1) experimental; urgency=medium...