Version check wrapper scripts for e.g. perf packaged in linux-tools-common prevent usage in a Docker container e.g. on Kubernetes

[Adam Novak]

The `perf` profiling tool, and a few other tools, are part of the Linux project. Ubuntu gets them from a package specific to the running kernel version, and uses wrapper scripts installed as e.g. /usr/bin/perf to dispatch to the appropriate version for the running kernel. This appears to involve not only checking the actual kernel version number (e.g. 4.15.0) but also the patch level/build number and flavor that Ubuntu adds (e.g. 4.15.0-62-generic) if found in the kernel version.

This is perfectly fine on a real system, where Ubuntu's package manager is actually in control of the kernel. But Ubuntu is also fantastically popular as a base for Docker images, and in a Docker container you have to take your lumps and run on whatever kernel version the host happens to be using, even if the host isn't using an official Ubuntu kernel, or running Ubuntu at all.

As currently designed, the linux-tools-common wrapper scripts cannot work reliably in a Docker environment.

When running in my particular Kubernetes environment (where the host is some kind of Red Hat Enterprise Linux derivative), this problem manifests as the following exchange:

root@adamnovak-pod:/vg# perf record ls
WARNING: perf not found for kernel 3.10.0

  You may need to install the following packages for this specific kernel:
    linux-tools-3.10.0

  You may also want to install one of the following packages to keep up to date:
    linux-tools
root@adamnovak-pod:/vg# apt install linux-tools-3.10.0
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-tools-3.10.0
E: Couldn't find any package by glob 'linux-tools-3.10.0'
E: Couldn't find any package by regex 'linux-tools-3.10.0'

However, the perf binary I happen to have installed, when I get it to try to run, at least appears to work, despite the kernel version mismatch. In larger tests, I've gotten perfectly fine profiling data.

root@adamnovak-pod:/vg# /usr/lib/linux-tools/4.15.0-62-generic/perf record ls
bin deps perf.data scripts
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.002 MB perf.data (10 samples) ]
root@adamnovak-pod:/vg#

So it is not clear that the level of specificity enforced by the wrapper scripts is actually necessary.

There are a few potential solutions I can see to this problem:

1. Modify the scripts to treat the lack of the exact right tool version being installed as a warning instead of an error, and have them select the closest version to run instead.

2. Create linux-tools-3.10.0 ish through linux-tools-5.x.x packages, containing builds of perf and co. derived from those official mainline kernels, and a metapackage to install all of them. Then the package that you get prompted to install when running in a container will exist, and you will be able to install it to have a toolset that has a high probability of being compatible with the kernel you are running on. A container that needs to be portable between different hosts can install the metapackage.

3. Create a package that conflicts with linux-tools-common and provides some arbitrary version of the tools. People who want a toolset that can run in a container can install that package instead, and bear the responsibility for what happens if the tools provided don't work with the kernel that happens to be running.

4. Hook into the alternatives system; allow the user to switch between packages providing the tools for different kernel versions, with the default being the dispatch script.

The workaround I am currently using is running this as soon as I get inside the container, to clobber the wrapper script with whatever actual binary I have installed:

cp /usr/lib/linux-tools/*/perf /usr/bin/perf

However, this is a pretty terrible hack.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-tools-common 4.15.0-62.69
ProcVersionSignature: Ubuntu 4.15.0-62.69-generic 4.15.18
Uname: Linux 4.15.0-62-generic x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: [Errno 2] No such file or directory: 'fuser': 'fuser'
CurrentDmesg: Error: command ['dmesg'] failed with exit code 1: dmesg: read kernel buffer failed: Operation not permitted
Date: Tue Sep 17 19:20:29 2019
Lspci: Error: [Errno 2] No such file or directory: 'lspci': 'lspci'
Lsusb: Error: [Errno 2] No such file or directory: 'lsusb': 'lsusb'
MachineType: Gigabyte Technology Co., Ltd. GA-890GPA-UD3H
PackageArchitecture: all
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.15.0-62-generic root=/dev/mapper/hex--vg-root ro splash quiet vt.handoff=1
SourcePackage: linux
UdevDb: Error: [Errno 2] No such file or directory: 'udevadm': 'udevadm'
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 11/24/2010
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: FF
dmi.board.name: GA-890GPA-UD3H
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrFF:bd11/24/2010:svnGigabyteTechnologyCo.,Ltd.:pnGA-890GPA-UD3H:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnGA-890GPA-UD3H:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: GA-890GPA-UD3H
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[Adam Novak (interfect)]

I reported this bug from this container running on an Ubuntu-based host where I built it. When I tried to run ubuntu-bug in the container on the host where I actually encountered the problem, it refused to report the bug due to... a non-Ubuntu kernel!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1844443

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Adam Novak (interfect)]

There's not really an applicable Ubuntu kernel to collect logs for; the problem is only apparent in a container on top of a host that doesn't run a recognized kernel.