ZFS kernel modules lack debug symbols
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| linux (Ubuntu) |
High
|
Mauricio Faria de Oliveira |
Bug Description
The ZFS kernel modules aren't built with debug symbols,
which introduces problems/issues for debugging/support.
Patches are required in:
1) linux kernel packaging, to add infrastructure to
enable/
(this is sufficient with zfs-linux now in Eoan.)
2) zfs-linux and spl-linux, for the stable releases,
which need a few patches to enable debug symbols
(add option './configure --enable-debuginfo' and
'(ZFS|
Initially submitting the kernel patchset for Unstable,
for review/feedback. It backports nicely into B/D/E,
should it be accepted; for X (doesn't use DKMS builds)
a simpler patch for the moment (until it does) works.
The zfs/spl-linux patches are ready, to be submitted
once the approach used by the kernel package settles.
CVE References
Changed in linux (Ubuntu): | |
status: | New → In Progress |
assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
description: | updated |
description: | updated |
Mauricio Faria de Oliveira (mfo) wrote : | #1 |
Mauricio Faria de Oliveira (mfo) wrote : | #2 |
Test Build 2) New behavior if ZFS modules are *not* built with debug symbols
goal: show failsafe/backwards compatible behavior if zfs-dksm doesn't support/build debug symbols
and kernel build log reports missing debug symbols, and extra modules have .gnu_debuglink.
- test packaging
- zfs not built with debug symbols (disabled manually in dkms-build if-check)
- zfs modules not present in debug package
- extra modules have .gnu_debuglink section
- (no regressions)
Test packaging, with debug symbols *not enabled* in zfs-dkms:
The debug symbols are not found (as expected),
and this case is handled without problems:
II: dkms-build installing zfs into /home/ubuntu/
ignoring zavl.ko (missing debug symbols)
stripping zavl.ko
ignoring znvpair.ko (missing debug symbols)
stripping znvpair.ko
ignoring zunicode.ko (missing debug symbols)
stripping zunicode.ko
ignoring zcommon.ko (missing debug symbols)
stripping zcommon.ko
ignoring zfs.ko (missing debug symbols)
stripping zfs.ko
ignoring icp.ko (missing debug symbols)
stripping icp.ko
ignoring zlua.ko (missing debug symbols)
stripping zlua.ko
ignoring spl.ko (missing debug symbols)
stripping spl.ko
II: dkms-build installing zfs into /home/ubuntu/
signing zavl.ko
signing znvpair.ko
signing zunicode.ko
signing zcommon.ko
signing zfs.ko
signing icp.ko
signing zlua.ko
signing spl.ko
II: dkms-build build zfs complete
The debug package contains the ZFS directory, but it's empty:
$ dpkg-deb -x linux-image-
$ ls ddeb-test-
$
The kernel build log documents which modules do not have debug symbols,
now covering modules built with DKMS (zfs and vbox):
$ grep WARNING build.log
echo "WARNING: Missing debug symbols for module '$module'."; \
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
The ZFS modules have no '.gnu_debuglink' section or any other debug section (as expected):
$ dpkg-deb -x l...
Mauricio Faria de Oliveira (mfo) wrote : | #3 |
Test Build 3) New behavior if ZFS modules are built with debug symbols
goal: show zfs debug symbols are correctly built and packaged into non-debug & debug packages.
- test packaging
- zfs built with debug symbols
- zfs modules present in debug package
- extra modules *have* .gnu_debuglink section
Test packaging, debug symbols *enabled* in zfs-dkms:
Modules are built with debug symbols, copied to debug package directory,
and stripped before being copied into strip/non-debug package directory.
II: dkms-build installing zfs into /home/ubuntu/
copying zavl.ko
stripping zavl.ko
copying znvpair.ko
stripping znvpair.ko
copying zunicode.ko
stripping zunicode.ko
copying zcommon.ko
stripping zcommon.ko
copying zfs.ko
stripping zfs.ko
copying icp.ko
stripping icp.ko
copying zlua.ko
stripping zlua.ko
copying spl.ko
stripping spl.ko
II: dkms-build installing zfs into /home/ubuntu/
signing zavl.ko
signing znvpair.ko
signing zunicode.ko
signing zcommon.ko
signing zfs.ko
signing icp.ko
signing zlua.ko
signing spl.ko
II: dkms-build build zfs complete
The ZFS modules are now present in the debug package:
$ dpkg-deb -x linux-image-
$ ls -1 ddeb-test-
icp.ko
spl.ko
zavl.ko
zcommon.ko
zfs.ko
zlua.ko
znvpair.ko
zunicode.ko
And now all modules in 'linux-modules' have the '.gnu_debuglink' section:
$ dpkg-deb -x linux-modules-
$ find deb-modules/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done
$
The build log no longer shows ZFS modules as missing debug symbols:
$ grep WARNING build.log
echo "WARNING: Missing debug symbols for module '$module'."; \
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/
$
The only modules in 'linux-
$ dpkg-deb -x linux-modules-
$ find deb-modules-extra/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done
Module without debug link 'deb-modules-
Module without debug link 'deb-modules-
$
As reflected in the kernel build log.
$ grep WARNING build.log
echo "WARNING: Missing debug symbols for module '$module'."; \
WARNING: Missing debug symbols for module '/lib/modules/
WARNING: Missing debug symbols for module '/lib/modules/5...
Mauricio Faria de Oliveira (mfo) wrote : | #4 |
Test Build 4) All debug symbols disabled
goal: show no zfs debug symbol activity happens either (along w/ other debug symbol stuff)
- test packaging
- nothing built with debug symbols
- no debug package present
- no .gnu_debuglink section at all
- (no regressions)
Test packaging, debug symbols disabled at all (skipdbg=true).
The dkms-build script doesn't do any debug symbol work at all.
II: dkms-build installing zfs into /home/ubuntu/
signing zavl.ko
signing znvpair.ko
signing zunicode.ko
signing zcommon.ko
signing zfs.ko
signing icp.ko
signing zlua.ko
signing spl.ko
II: dkms-build build zfs complete
No debug sections are present in ZFS modules (as expected):
$ objdump -h deb-modules/
$
And the check for modules without debug symbols is not exercised (as expected):
$ grep WARNING build.log
$
$ find deb-modules/ -name '*.ko' | wc -l
1000
$ find deb-modules/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done | wc -l
1000
$ find deb-modules-extra/ -name '*.ko' | wc -l
4508
$ find deb-modules-extra/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done | wc -l
4508
description: | updated |
Mauricio Faria de Oliveira (mfo) wrote : | #5 |
[Unstable][PATCH 0/6] Add support for ZFS debug symbols
https:/
Mauricio Faria de Oliveira (mfo) wrote : | #6 |
Attaching the debdiffs for zfs-linux/spl-linux on X/B/D/E,
for documentation purposes; will send testing/notes later.
Independently of the kernel packaging approach determined
to enable debug symbols on ZFS/SPL modules, these kind of
patches for the userspace packages are be required anyway,
and correctly performed that task when building with DKMS.
So I'll probably move forward with their SRU request soon,
in the benefit of having this available sooner if required
(i.e. so users/engineers in need of debug symbols may just
rebuild with DKMS using this, and be able to investigate.)
tags: | added: sts |
tags: | added: linux |
Changed in linux (Ubuntu): | |
importance: | Undecided → High |
Launchpad Janitor (janitor) wrote : | #7 |
This bug was fixed in the package linux - 5.3.0-17.18
---------------
linux (5.3.0-17.18) eoan; urgency=medium
* eoan/linux: 5.3.0-17.18 -proposed tracker (LP: #1846641)
* CVE-2019-17056
- nfc: enforce CAP_NET_RAW for raw sockets
* CVE-2019-17055
- mISDN: enforce CAP_NET_RAW for raw sockets
* CVE-2019-17054
- appletalk: enforce CAP_NET_RAW for raw sockets
* CVE-2019-17053
- ieee802154: enforce CAP_NET_RAW for raw sockets
* CVE-2019-17052
- ax25: enforce CAP_NET_RAW for raw sockets
* CVE-2019-15098
- ath6kl: fix a NULL-ptr-deref bug in ath6kl_
* xHCI on AMD Stoney Ridge cannot detect USB 2.0 or 1.1 devices.
(LP: #1846470)
- x86/PCI: Avoid AMD FCH XHCI USB PME# from D0 defect
* Re-enable linux-libc-dev build on i386 (LP: #1846508)
- [Packaging] Build only linux-libc-dev for i386
- [Debian] final-checks -- ignore archtictures with no binaries
* arm64: loop on boot after installing linux-generic-
proposed (LP: #1845820)
- [Config] Disable CONFIG_
* Revert ESE DASD discard support (LP: #1846219)
- SAUCE: Revert "s390/dasd: Add discard support for ESE volumes"
* Miscellaneous Ubuntu changes
- update dkms package versions
linux (5.3.0-16.17) eoan; urgency=medium
* eoan/linux: 5.3.0-16.17 -proposed tracker (LP: #1846204)
* zfs fails to build on s390x with debug symbols enabled (LP: #1846143)
- SAUCE: s390: Mark atomic const ops always inline
linux (5.3.0-15.16) eoan; urgency=medium
* eoan/linux: 5.3.0-15.16 -proposed tracker (LP: #1845987)
* Drop i386 build for 19.10 (LP: #1845714)
- [Packaging] Remove x32 arch references from control files
- [Debian] final-checks -- Get arch list from debian/control
* ZFS kernel modules lack debug symbols (LP: #1840704)
- [Debian] Fix conditional for setting zfs debug package path
* Use pyhon3-sphinx instead of python-sphinx for building html docs
(LP: #1845808)
- [Packaging] Update sphinx build dependencies to python3 packages
* Kernel panic with 19.10 beta image (LP: #1845454)
- efi/tpm: Don't access event->count when it isn't mapped.
- efi/tpm: don't traverse an event log with no events
- efi/tpm: only set efi_tpm_
linux (5.3.0-14.15) eoan; urgency=medium
* eoan/linux: 5.3.0-14.15 -proposed tracker (LP: #1845728)
* Drop i386 build for 19.10 (LP: #1845714)
- [Debian] Remove support for producing i386 kernels
- [Debian] Don't use CROSS_COMPILE for i386 configs
* udevadm trigger will fail when trying to add /sys/devices/vio/
(LP: #1845572)
- SAUCE: powerpc/vio: drop bus_type from parent device
* Trying to online dasd drive results in invalid input/output from the kernel
on z/VM (LP: #1845323)
- SAUCE: s390/dasd: Fix error handling during online processing
* intel-lpss driver conflicts with write-combining MTRR region (LP: #1845584)
- SAUCE: mfd: intel-lpss: add quirk for Dell XPS 13 7390 2-in-1
* Support Hi1620 zip hw accelerator (LP: #1845355)
- [Config] Enable HiSilicon QM/ZIP as module...
Changed in linux (Ubuntu): | |
status: | In Progress → Fix Released |
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (linux-gcp-5.3/5.3.0-1008.9~18.04.1) | #8 |
All autopkgtests for the newly accepted linux-gcp-5.3 (5.3.0-
The following regressions have been reported in tests triggered by the package:
linux-gcp-
Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUp
[1] https:/
Thank you!
Test Build 1) Old behavior
goal: show limitations/issues.
- original packaging
- zfs not built with debug symbols
- zfs modules not present in debug package
- extra modules lack .gnu_debuglink section
Original packaging:
There are no ZFS modules in the debug package:
$ dpkg-deb -x linux-image- unsigned- 5.3.0-8- generic- dbgsym_ 5.3.0-8. 9_amd64. ddeb ddeb-orig
$ ls ddeb-orig/ usr/lib/ debug/lib/ modules/ 5.3.0-8- generic/ kernel/ zfs
...: No such file or directory
Accordingly, the ZFS modules are the only modules without '.gnu_debuglink' section in the 'linux-modules' package:
$ dpkg-deb -x linux-modules- 5.3.0-8- generic_ 5.3.0-8. 9_amd64. deb deb-modules
$ find deb-modules/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done lib/modules/ 5.3.0-8- generic/ kernel/ zfs/icp. ko' lib/modules/ 5.3.0-8- generic/ kernel/ zfs/spl. ko' lib/modules/ 5.3.0-8- generic/ kernel/ zfs/zavl. ko' lib/modules/ 5.3.0-8- generic/ kernel/ zfs/zcommon. ko' lib/modules/ 5.3.0-8- generic/ kernel/ zfs/zfs. ko' lib/modules/ 5.3.0-8- generic/ kernel/ zfs/zlua. ko' lib/modules/ 5.3.0-8- generic/ kernel/ zfs/znvpair. ko' lib/modules/ 5.3.0-8- generic/ kernel/ zfs/zunicode. ko'
Module without debug link 'deb-modules/
Module without debug link 'deb-modules/
Module without debug link 'deb-modules/
Module without debug link 'deb-modules/
Module without debug link 'deb-modules/
Module without debug link 'deb-modules/
Module without debug link 'deb-modules/
Module without debug link 'deb-modules/
By the way, this is also the case for *all* modules in the 'linux- modules- extra' package:
(only modules in the 'linux-modules' package have '.gnu_debuglink' sections).
$ dpkg-deb -x linux-modules- extra-5. 3.0-8-generic_ 5.3.0-8. 9_amd64. deb deb-modules-extras
$ find deb-modules-extras/ -name '*.ko' | wc -l
4508
$ find deb-modules-extras/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done | wc -l
4508