ZFS kernel modules lack debug symbols

[Mauricio Faria de Oliveira]

The ZFS kernel modules aren't built with debug symbols,
which introduces problems/issues for debugging/support.

Patches are required in:

1) linux kernel packaging, to add infrastructure to
   enable/build/strip/package debug symbols on DKMS.
   (this is sufficient with zfs-linux now in Eoan.)

2) zfs-linux and spl-linux, for the stable releases,
   which need a few patches to enable debug symbols
   (add option './configure --enable-debuginfo' and
   '(ZFS|SPL)_DKMS_ENABLE_DEBUGINFO' to dkms.conf.)

Initially submitting the kernel patchset for Unstable,
for review/feedback. It backports nicely into B/D/E,
should it be accepted; for X (doesn't use DKMS builds)
a simpler patch for the moment (until it does) works.

The zfs/spl-linux patches are ready, to be submitted
once the approach used by the kernel package settles.

[Mauricio Faria de Oliveira (mfo)]

Test Build 1) Old behavior

goal: show limitations/issues.

- original packaging
- zfs not built with debug symbols
- zfs modules not present in debug package
- extra modules lack .gnu_debuglink section

Original packaging:

There are no ZFS modules in the debug package:

 $ dpkg-deb -x linux-image-unsigned-5.3.0-8-generic-dbgsym_5.3.0-8.9_amd64.ddeb ddeb-orig

 $ ls ddeb-orig/usr/lib/debug/lib/modules/5.3.0-8-generic/kernel/zfs
 ...: No such file or directory

Accordingly, the ZFS modules are the only modules without '.gnu_debuglink' section in the 'linux-modules' package:

 $ dpkg-deb -x linux-modules-5.3.0-8-generic_5.3.0-8.9_amd64.deb deb-modules

 $ find deb-modules/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done
 Module without debug link 'deb-modules/lib/modules/5.3.0-8-generic/kernel/zfs/icp.ko'
 Module without debug link 'deb-modules/lib/modules/5.3.0-8-generic/kernel/zfs/spl.ko'
 Module without debug link 'deb-modules/lib/modules/5.3.0-8-generic/kernel/zfs/zavl.ko'
 Module without debug link 'deb-modules/lib/modules/5.3.0-8-generic/kernel/zfs/zcommon.ko'
 Module without debug link 'deb-modules/lib/modules/5.3.0-8-generic/kernel/zfs/zfs.ko'
 Module without debug link 'deb-modules/lib/modules/5.3.0-8-generic/kernel/zfs/zlua.ko'
 Module without debug link 'deb-modules/lib/modules/5.3.0-8-generic/kernel/zfs/znvpair.ko'
 Module without debug link 'deb-modules/lib/modules/5.3.0-8-generic/kernel/zfs/zunicode.ko'

By the way, this is also the case for *all* modules in the 'linux-modules-extra' package:
(only modules in the 'linux-modules' package have '.gnu_debuglink' sections).

 $ dpkg-deb -x linux-modules-extra-5.3.0-8-generic_5.3.0-8.9_amd64.deb deb-modules-extras

 $ find deb-modules-extras/ -name '*.ko' | wc -l
 4508

 $ find deb-modules-extras/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done | wc -l
 4508

[Mauricio Faria de Oliveira (mfo)]

Test Build 2) New behavior if ZFS modules are *not* built with debug symbols

goal: show failsafe/backwards compatible behavior if zfs-dksm doesn't support/build debug symbols
      and kernel build log reports missing debug symbols, and extra modules have .gnu_debuglink.

- test packaging
- zfs not built with debug symbols (disabled manually in dkms-build if-check)
- zfs modules not present in debug package
- extra modules have .gnu_debuglink section
- (no regressions)

Test packaging, with debug symbols *not enabled* in zfs-dkms:

 The debug symbols are not found (as expected),
 and this case is handled without problems:

 II: dkms-build installing zfs into /home/ubuntu/dbgsym/unstable/debian/linux-image-unsigned-5.3.0-8-generic-dbgsym/usr/lib/debug/lib/modules/5.3.0-8-generic/kernel/zfs (debug symbols)
 ignoring zavl.ko (missing debug symbols)
 stripping zavl.ko
 ignoring znvpair.ko (missing debug symbols)
 stripping znvpair.ko
 ignoring zunicode.ko (missing debug symbols)
 stripping zunicode.ko
 ignoring zcommon.ko (missing debug symbols)
 stripping zcommon.ko
 ignoring zfs.ko (missing debug symbols)
 stripping zfs.ko
 ignoring icp.ko (missing debug symbols)
 stripping icp.ko
 ignoring zlua.ko (missing debug symbols)
 stripping zlua.ko
 ignoring spl.ko (missing debug symbols)
 stripping spl.ko
 II: dkms-build installing zfs into /home/ubuntu/dbgsym/unstable/debian/linux-modules-5.3.0-8-generic/lib/modules/5.3.0-8-generic/kernel/zfs
 signing zavl.ko
 signing znvpair.ko
 signing zunicode.ko
 signing zcommon.ko
 signing zfs.ko
 signing icp.ko
 signing zlua.ko
 signing spl.ko
 II: dkms-build build zfs complete

The debug package contains the ZFS directory, but it's empty:

 $ dpkg-deb -x linux-image-unsigned-5.3.0-8-generic-dbgsym_5.3.0-8.9_amd64.ddeb ddeb-test-disabled
 $ ls ddeb-test-disabled/usr/lib/debug/lib/modules/5.3.0-8-generic/kernel/zfs/
 $

The kernel build log documents which modules do not have debug symbols,
now covering modules built with DKMS (zfs and vbox):

 $ grep WARNING build.log
          echo "WARNING: Missing debug symbols for module '$module'."; \
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/zfs/zavl.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/zfs/znvpair.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/zfs/zunicode.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/zfs/zcommon.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/zfs/zfs.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/zfs/icp.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/zfs/zlua.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/zfs/spl.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/virtualbox-guest/vboxguest.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/virtualbox-guest/vboxsf.ko'.

The ZFS modules have no '.gnu_debuglink' section or any other debug section (as expected):

 $ dpkg-deb -x l...

[Mauricio Faria de Oliveira (mfo)]

Test Build 3) New behavior if ZFS modules are built with debug symbols

goal: show zfs debug symbols are correctly built and packaged into non-debug & debug packages.

- test packaging
- zfs built with debug symbols
- zfs modules present in debug package
- extra modules *have* .gnu_debuglink section

Test packaging, debug symbols *enabled* in zfs-dkms:

Modules are built with debug symbols, copied to debug package directory,
and stripped before being copied into strip/non-debug package directory.

 II: dkms-build installing zfs into /home/ubuntu/dbgsym/unstable/debian/linux-image-unsigned-5.3.0-8-generic-dbgsym/usr/lib/debug/lib/modules/5.3.0-8-generic/kernel/zfs (debug symbols)
 copying zavl.ko
 stripping zavl.ko
 copying znvpair.ko
 stripping znvpair.ko
 copying zunicode.ko
 stripping zunicode.ko
 copying zcommon.ko
 stripping zcommon.ko
 copying zfs.ko
 stripping zfs.ko
 copying icp.ko
 stripping icp.ko
 copying zlua.ko
 stripping zlua.ko
 copying spl.ko
 stripping spl.ko
 II: dkms-build installing zfs into /home/ubuntu/dbgsym/unstable/debian/linux-modules-5.3.0-8-generic/lib/modules/5.3.0-8-generic/kernel/zfs
 signing zavl.ko
 signing znvpair.ko
 signing zunicode.ko
 signing zcommon.ko
 signing zfs.ko
 signing icp.ko
 signing zlua.ko
 signing spl.ko
 II: dkms-build build zfs complete

The ZFS modules are now present in the debug package:

 $ dpkg-deb -x linux-image-unsigned-5.3.0-8-generic-dbgsym_5.3.0-8.9_amd64.ddeb ddeb-test-enabled

 $ ls -1 ddeb-test-enabled/usr/lib/debug/lib/modules/5.3.0-8-generic/kernel/zfs/
 icp.ko
 spl.ko
 zavl.ko
 zcommon.ko
 zfs.ko
 zlua.ko
 znvpair.ko
 zunicode.ko

And now all modules in 'linux-modules' have the '.gnu_debuglink' section:

 $ dpkg-deb -x linux-modules-5.3.0-8-generic_5.3.0-8.9_amd64.deb deb-modules
 $ find deb-modules/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done
 $

The build log no longer shows ZFS modules as missing debug symbols:

 $ grep WARNING build.log
          echo "WARNING: Missing debug symbols for module '$module'."; \
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/virtualbox-guest/vboxguest.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/virtualbox-guest/vboxsf.ko'.
 $

The only modules in 'linux-modules-extra' without that continue to be virtualbox modules:

 $ dpkg-deb -x linux-modules-extra-5.3.0-8-generic_5.3.0-8.9_amd64.deb deb-modules-extra
 $ find deb-modules-extra/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done
 Module without debug link 'deb-modules-extra/lib/modules/5.3.0-8-generic/kernel/virtualbox-guest/vboxguest.ko'
 Module without debug link 'deb-modules-extra/lib/modules/5.3.0-8-generic/kernel/virtualbox-guest/vboxsf.ko'
 $

As reflected in the kernel build log.

 $ grep WARNING build.log
          echo "WARNING: Missing debug symbols for module '$module'."; \
 WARNING: Missing debug symbols for module '/lib/modules/5.3.0-8-generic/kernel/virtualbox-guest/vboxguest.ko'.
 WARNING: Missing debug symbols for module '/lib/modules/5...

[Mauricio Faria de Oliveira (mfo)]

Test Build 4) All debug symbols disabled

goal: show no zfs debug symbol activity happens either (along w/ other debug symbol stuff)

- test packaging
- nothing built with debug symbols
- no debug package present
- no .gnu_debuglink section at all
- (no regressions)

Test packaging, debug symbols disabled at all (skipdbg=true).

The dkms-build script doesn't do any debug symbol work at all.

 II: dkms-build installing zfs into /home/ubuntu/dbgsym/unstable/debian/linux-modules-5.3.0-8-generic/lib/modules/5.3.0-8-generic/kernel/zfs
 signing zavl.ko
 signing znvpair.ko
 signing zunicode.ko
 signing zcommon.ko
 signing zfs.ko
 signing icp.ko
 signing zlua.ko
 signing spl.ko
 II: dkms-build build zfs complete

No debug sections are present in ZFS modules (as expected):

 $ objdump -h deb-modules/lib/modules/5.3.0-8-generic/kernel/zfs/zfs.ko | grep debug
 $

And the check for modules without debug symbols is not exercised (as expected):

 $ grep WARNING build.log
 $

$ find deb-modules/ -name '*.ko' | wc -l
1000

$ find deb-modules/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done | wc -l
1000

$ find deb-modules-extra/ -name '*.ko' | wc -l
4508

$ find deb-modules-extra/ -name '*.ko' | while read ko; do objdump -h -j .gnu_debuglink $ko >/dev/null 2>&1 || echo "Module without debug link '$ko'"; done | wc -l
4508

[Mauricio Faria de Oliveira (mfo)]

[Unstable][PATCH 0/6] Add support for ZFS debug symbols
https://lists.ubuntu.com/archives/kernel-team/2019-August/103425.html

[Mauricio Faria de Oliveira (mfo)]

Attaching the debdiffs for zfs-linux/spl-linux on X/B/D/E,
for documentation purposes; will send testing/notes later.

Independently of the kernel packaging approach determined
to enable debug symbols on ZFS/SPL modules, these kind of
patches for the userspace packages are be required anyway,
and correctly performed that task when building with DKMS.

So I'll probably move forward with their SRU request soon,
in the benefit of having this available sooner if required
(i.e. so users/engineers in need of debug symbols may just
rebuild with DKMS using this, and be able to investigate.)

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.3.0-17.18

---------------
linux (5.3.0-17.18) eoan; urgency=medium

  * eoan/linux: 5.3.0-17.18 -proposed tracker (LP: #1846641)

  * CVE-2019-17056
    - nfc: enforce CAP_NET_RAW for raw sockets

  * CVE-2019-17055
    - mISDN: enforce CAP_NET_RAW for raw sockets

  * CVE-2019-17054
    - appletalk: enforce CAP_NET_RAW for raw sockets

  * CVE-2019-17053
    - ieee802154: enforce CAP_NET_RAW for raw sockets

  * CVE-2019-17052
    - ax25: enforce CAP_NET_RAW for raw sockets

  * CVE-2019-15098
    - ath6kl: fix a NULL-ptr-deref bug in ath6kl_usb_alloc_urb_from_pipe()

  * xHCI on AMD Stoney Ridge cannot detect USB 2.0 or 1.1 devices.
    (LP: #1846470)
    - x86/PCI: Avoid AMD FCH XHCI USB PME# from D0 defect

  * Re-enable linux-libc-dev build on i386 (LP: #1846508)
    - [Packaging] Build only linux-libc-dev for i386
    - [Debian] final-checks -- ignore archtictures with no binaries

  * arm64: loop on boot after installing linux-generic-hwe-18.04-edge/bionic-
    proposed (LP: #1845820)
    - [Config] Disable CONFIG_ARM_SMMU_DISABLE_BYPASS_BY_DEFAULT

  * Revert ESE DASD discard support (LP: #1846219)
    - SAUCE: Revert "s390/dasd: Add discard support for ESE volumes"

  * Miscellaneous Ubuntu changes
    - update dkms package versions

linux (5.3.0-16.17) eoan; urgency=medium

  * eoan/linux: 5.3.0-16.17 -proposed tracker (LP: #1846204)

  * zfs fails to build on s390x with debug symbols enabled (LP: #1846143)
    - SAUCE: s390: Mark atomic const ops always inline

linux (5.3.0-15.16) eoan; urgency=medium

  * eoan/linux: 5.3.0-15.16 -proposed tracker (LP: #1845987)

  * Drop i386 build for 19.10 (LP: #1845714)
    - [Packaging] Remove x32 arch references from control files
    - [Debian] final-checks -- Get arch list from debian/control

  * ZFS kernel modules lack debug symbols (LP: #1840704)
    - [Debian] Fix conditional for setting zfs debug package path

  * Use pyhon3-sphinx instead of python-sphinx for building html docs
    (LP: #1845808)
    - [Packaging] Update sphinx build dependencies to python3 packages

  * Kernel panic with 19.10 beta image (LP: #1845454)
    - efi/tpm: Don't access event->count when it isn't mapped.
    - efi/tpm: don't traverse an event log with no events
    - efi/tpm: only set efi_tpm_final_log_size after successful event log parsing

linux (5.3.0-14.15) eoan; urgency=medium

  * eoan/linux: 5.3.0-14.15 -proposed tracker (LP: #1845728)

  * Drop i386 build for 19.10 (LP: #1845714)
    - [Debian] Remove support for producing i386 kernels
    - [Debian] Don't use CROSS_COMPILE for i386 configs

  * udevadm trigger will fail when trying to add /sys/devices/vio/
    (LP: #1845572)
    - SAUCE: powerpc/vio: drop bus_type from parent device

  * Trying to online dasd drive results in invalid input/output from the kernel
    on z/VM (LP: #1845323)
    - SAUCE: s390/dasd: Fix error handling during online processing

  * intel-lpss driver conflicts with write-combining MTRR region (LP: #1845584)
    - SAUCE: mfd: intel-lpss: add quirk for Dell XPS 13 7390 2-in-1

  * Support Hi1620 zip hw accelerator (LP: #1845355)
    - [Config] Enable HiSilicon QM/ZIP as module...

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted linux-gcp-5.3 (5.3.0-1008.9~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-gcp-5.3/unknown (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-gcp-5.3

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!