Kernels & kernel drivers fail to build with gcc-9 [error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible]

[satmandu]

[SRU Justification for backport-iwlwifi-dkms]

[Impact]
FTBFS against v5.3 kernels on Eoan/Focal.

[Fix]
* Backport fix from upstream commit f47729b94e11 (backport: rename ktime_get_boot_ns for v5.3+).
* Also updates -fcf-protection=none usage for bug
1830961.

[Test Case]
Verified DKMS module build on linux-oem/linux-oem-osp1/linux-generic >= 4.10 kernels on series Xenial/Bionic/Cosmic/Disco/Eoan and Focal.

[Regression Risk]
Low. There is no functional changes in comparison to 7906-0ubuntu1.

======== original bug description ========

Compiling kernels & kernel modules fails due to these errors:

./include/linux/compiler.h:193:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

(This happens with any kernel modules.)

This appears to be due to the changes in 9.1.0-3ubuntu1 enabling -fcf-protection by default on 19.10's gcc-9.

Switching to gcc-8 allows compilation to proceed.

WORKAROUND:

sudo ln -fs gcc-8 /usr/bin/gcc

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steven Barrett (damentz)]

This affects package builds for Eoan with Liquorix in launchpad: https://launchpad.net/~damentz/+archive/ubuntu/liquorix/+build/17232577

[Trent Lloyd (lathiat)]

This prevents compilation of NVIDIA drivers (DKMS) also.

I *suspect* (but am not at all authoritative or knowledgeable enough to say so) that the fix here will actually be to change the build flags used by the kernel set in /usr/src/linux-headers-$(uname -r)/Makefile

[Vidar Braut Haarr (vhaarr+launchpad)]

As a temporary workaround I changed the compile flags in /usr/src/linux-headers-foo/Makefile around line 634 to

RETPOLINE_CFLAGS_GCC := -mindirect-branch-register
RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch-register

and now it compiled with gcc9

[Vidar Braut Haarr (vhaarr+launchpad)]

However, even though it compiled it seemed to be broken, so instead I just changed the default compiler to gcc-8 and that worked fine.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Daniel van Vugt (vanvugt)]

WORKAROUND:

sudo ln -fs gcc-8 /usr/bin/gcc

[Daniel van Vugt (vanvugt)]

https://lists.ubuntu.com/archives/ubuntu-devel/2019-June/040741.html

[satmandu (satadru-umich)]

This also appears to be breaking the Ubuntu mainline kernel builds:

https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2/BUILD.LOG.amd64

[Alberto Milone (albertomilone)]

I uploaded a workaround, and it is in eoan-proposed:
https://launchpad.net/ubuntu/+source/nvidia-graphics-drivers-430/430.26-0ubuntu3

[jean-christophe manciot (manciot-jeanchristophe)]

Same issue with:

Ubuntu 19.10
xtables-addons-dkms 3.2-1ubuntu2
linux kernel 5.0.0-17 and 5.0.0-20
gcc-9: 9.1.0-6ubuntu2

The compilation complains that:
 echo >&2; \
 echo >&2 " ERROR: Kernel configuration is invalid."; \
 echo >&2 " include/generated/autoconf.h or include/config/auto.conf are missing.";\
 echo >&2 " Run 'make oldconfig && make prepare' on kernel src to fix it."; \

but both files are present. Later on, I get a huge number of similar errors like the following:
 ./include/linux/compiler.h:192:1: error: ‘-mindirect-branch’ and ‘-fcf-protection’ are not compatible

Switching to gcc-8 works around this critical issue.

[Doug Smythies (dsmythies)]

As per comment #4, i made the same change:

doug@serv-ee:~/temp-k-git/linux$ git diff
diff --git a/Makefile b/Makefile
index 3e4868a6498b..68f0858484a5 100644
--- a/Makefile
+++ b/Makefile
@@ -630,8 +630,8 @@ ifdef CONFIG_FUNCTION_TRACER
   CC_FLAGS_FTRACE := -pg
 endif

-RETPOLINE_CFLAGS_GCC := -mindirect-branch=thunk-extern -mindirect-branch-register
-RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch=thunk-inline -mindirect-branch-register
+RETPOLINE_CFLAGS_GCC := -mindirect-branch-register
+RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch-register
 RETPOLINE_CFLAGS_CLANG := -mretpoline-external-thunk
 RETPOLINE_VDSO_CFLAGS_CLANG := -mretpoline
 RETPOLINE_CFLAGS := $(call cc-option,$(RETPOLINE_CFLAGS_GCC),$(call cc-option,$(RETPOLINE_CFLAGS_CLANG)))

And while the kernel compile had a constant stream of warnings, the resulting kernel (5.2) works fine.

[satmandu (satadru-umich)]

The origin of this bug is the change in the gcc-9 package on 26 May 2019. Shouldn't the fcf-protection change there just be reverted?

As per the changelog:

gcc-9 (9.1.0-3) experimental; urgency=medium

  * Update to SVN 20190526 (r271629) from the gcc-9-branch.
    - Fix PR libgomp/90527, PR c++/90532, PR libstdc++/90299,
      PR libstdc++/90454, PR debug/90197, PR pch/90326, PR c++/90484,
      PR tree-optimization/90385, PR c++/90383, PR tree-optimization/90303,
      PR tree-optimization/90316, PR tree-optimization/90316,
      PR libstdc++/90220, PR libstdc++/90557, PR sanitizer/90570,
      PR target/90547 (x86), PR libfortran/90038, PR fortran/90498,
      PR libfortran/90038, PR libfortran/90038, PR fortran/54613,
      PR fortran/54613, PR libstdc++/85965, PR target/90530 (PARISC),
      PR c++/90572.
  * Turn on -fstack-clash-protection and -fcf-protection in Ubuntu 19.10 on
    supported architectures.
  * Fix PR bootstrap/87338 on ia64 (James Clarke). Addresses: #927976.
  * Enable LTO builds on 64bit architectures.
  * Update libstdc++ symbols files for gcc-4-compatible builds.
  * Build the nvptx offload compiler on ppc64el.
  * Build the libgomp-hsa plugin.

 -- Matthias Klose <email address hidden> Sun, 26 May 2019 17:59:59 +0200

[Matthias Klose (doko)]

no, the kernel infrastructure should handle that like any other hardening flags before.

[satmandu (satadru-umich)]

It looks like this is being addressed in recent linux packages.

As per https://launchpad.net/ubuntu/+source/linux/5.2.0-8.9

"SAUCE: add -fcf-protection=none to retpoline flags"

[Doug Smythies (dsmythies)]

If I make the same changes to the Makefile as per the link in comment #16, then the kernel compiles fine and without the zillions of warnings I mentioned in comment #13. However I can not find this anywhere "upstream" as the notes seem to indicate.

doug@serv-ee:~/temp-k-git/linux$ git diff
diff --git a/Makefile b/Makefile
index 3e4868a6498b..d8ccc47215be 100644
--- a/Makefile
+++ b/Makefile
@@ -630,8 +630,8 @@ ifdef CONFIG_FUNCTION_TRACER
   CC_FLAGS_FTRACE := -pg
 endif

-RETPOLINE_CFLAGS_GCC := -mindirect-branch=thunk-extern -mindirect-branch-register
-RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch=thunk-inline -mindirect-branch-register
+RETPOLINE_CFLAGS_GCC := -mindirect-branch=thunk-extern -mindirect-branch-register -fcf-protection=none
+RETPOLINE_VDSO_CFLAGS_GCC := -mindirect-branch=thunk-inline -mindirect-branch-register -fcf-protection=none
 RETPOLINE_CFLAGS_CLANG := -mretpoline-external-thunk
 RETPOLINE_VDSO_CFLAGS_CLANG := -mretpoline
 RETPOLINE_CFLAGS := $(call cc-option,$(RETPOLINE_CFLAGS_GCC),$(call cc-option,$(RETPOLINE_CFLAGS_CLANG)))

[jean-christophe manciot (manciot-jeanchristophe)]

As a follow-up tp my previous comment #12 (https://bugs.launchpad.net/ubuntu/+source/xtables-addons/+bug/1830961/comments/12), I have discovered 2 new facts:
- I am not able to build dpdk 19.05 on the same system anymore, (cf. build log), unless I modify the gcc symlink to point to gcc-8 instead of gcc-9
- however, the xtables -addons issue encountered on the previous system has not been seen on 2 other Ubuntu Eoan platforms with the exact same versions of:
  + gcc-9 (and/usr/bin/gcc -> gcc-9)
  + linux kernels
  + xtables-addons-dkms/common

[Launchpad Janitor (janitor)]

This bug was fixed in the package nvidia-graphics-drivers-430 - 430.34-0ubuntu2

---------------
nvidia-graphics-drivers-430 (430.34-0ubuntu2) eoan; urgency=medium

  * debian/dkms_nvidia/patches/disable_fstack-clash-protection_fcf-protection.patch:
    - Refresh for 430.34 (LP: #1830961).

 -- Alberto Milone <email address hidden> Wed, 10 Jul 2019 15:27:36 +0200

[Daniel van Vugt (vanvugt)]

Assigned to Seth per comment #16.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rocko (rockorequin)]

The patch that satmandu mentioned in https://bugs.launchpad.net/ubuntu/+source/virtualbox/+bug/1830961/comments/16 fixes the v5.2.0 mainline kernel build as well.

[Maarten Fonville (maarten-fonville)]

Can the patch that was applied to the kernel in https://launchpad.net/ubuntu/+source/linux/5.2.0-8.9 please also be applied to the daily mainline builds? (possible duplicate bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1836373 )

[Gianfranco Costamagna (costamagnagianfranco)]

This is fixed in the -proposed kernel, so I don't plan to hack virtualbox sources to drop such flag.

[jean-christophe manciot (manciot-jeanchristophe)]

I confirm that on Ubuntu Eoan with:
- linux kernel 5.2.0-8.9
- gcc-9 9.1.0-8ubuntu1

I am able to:
- build DPDK 19.05
- install xtables-addons-dkms.

[Daniel van Vugt (vanvugt)]

linux (5.2.0-8.9) eoan; urgency=medium

  * linux: 5.2.0-8.9 -proposed tracker (LP: #1835700)

  * Miscellaneous Ubuntu changes
    - [Packaging] replace zfs and spl build with zfs 0.8.1-1ubuntu1
    - SAUCE: test_bpf: remove expected fail for Ctx heavy transformations test on
      s390
    - SAUCE: add -fcf-protection=none to retpoline flags
    - SAUCE: usbip: ensure strings copied using strncpy are null-terminated
    - SAUCE: usbip: add -Wno-address-of-packed-member to EXTRA_CFLAGS
    - SAUCE: perf jvmti: ensure strncpy result is null-terminated
    - update dkms package versions
    - add removed zfs modules to modules.ignore

  [ Upstream Kernel Changes ]

  * Rebase to v5.2

 -- Seth Forshee <email address hidden> Mon, 08 Jul 2019 07:13:41 -0500

[Daniel van Vugt (vanvugt)]

If the new kernel works per comment #25 then no specific DKMS packages need fixing.

[satmandu (satadru-umich)]

Any chance someone could get the recipes fixed for the mainline builds, which also need the -fcf-protection=none retpoline flags added since they're being built using "schroot -c eoan-amd64"?

(I'm not sure who needs to be looped in for that to happen.)

[oscarbg (rtfss1)]

+1 for fixing mainline kernel builds.. been broken for almost two weeks!

[oscarbg (rtfss1)]

maybe Seth Forshee <email address hidden> is the guy to ask

[Doug Smythies (dsmythies)]

Several posters are asking for the mainline builds to be fixed. The mainline builds are an exact reflection of the upstream git master repositories. Therefore they have to wait for Seth's patch submission, which has been picked up and accepted by the kbuild team, to propagate it's way to the mainline upstream git branch. Earlier today, a kernel 5.3-rc1 merge request was issued by the kbuild team [1]. Once that branch has been "pulled" into the upstream mainline, the Ubuntu mainline daily compiles should start working the next day. In about 8 days there will be kernel 5.3-rc1, which should have the patch.

[1] https://www.spinics.net/lists/linux-kbuild/msg22526.html

Other references:

https://kernel.ubuntu.com/~kernel-ppa/mainline/daily/
https://www.spinics.net/lists/linux-kbuild/msg22298.html
https://patchwork.kernel.org/patch/11037379/

[satmandu (satadru-umich)]

Thanks all!

Mainline builds of daily and 5.3-rc1 appear to have been generated for amd64 last night.

[semreh (launchpad-via-forwarder)]

It is nice to see this resolved for 5.3-rc1.

Excuse my ignorance of the process, but will this patch be (perhaps automagically) backported so the next iteration of the 5.2 (and maybe 5.1) stable kernels for amd64 and i386 also get generated successfully?

[Daniel van Vugt (vanvugt)]

I think if you need to build any older kernels then the simplest answer is the workaround in comment #8.

[Christoph Reiter (lazka)]

Just for the record, this still breaks upgrades from 19.04 (I updated today) because dkms tries to build things (nvidia and virtualbox here) for the 19.04 kernel and fails.

[Seth Forshee (sforshee)]

Hmm, I guess we should SRU the --fcf-protection=none change to the 19.04 and 18.04 kernels in that case. I will get this done.

[Rik Mills (rikmills)]

Redditer trying an upgrade from disco -> eoan reported broken upgrade due to this, and like in comment #34 I can reproduce this still.

So I think this a fix backported to 19,04 kernel is definitely required before we release 19.10.

[Łukasz Zemczak (sil2100)]

This bug is linked in the backport-iwlwifi-dkms SRU for eoan but it does not provide the required SRU information (Impact, Test Case, Regression Potential [1]). Could you please update the bug description? Thank you!

[1] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

[You-Sheng Yang (vicamo)]

Copied SRU justification from bug 1848922.

[Timo Aaltonen (tjaalton)]

Hello satmandu, or anyone else affected,

Accepted backport-iwlwifi-dkms into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/backport-iwlwifi-dkms/7906-0ubuntu2~19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[You-Sheng Yang (vicamo)]

Verified version 7906-0ubuntu2~19.10.1 against following kernel headers:

  $ dkms status
  backport-iwlwifi, 7906, 4.15.0-1059-oem, x86_64: installed
  backport-iwlwifi, 7906, 5.0.0-1025-oem-osp1, x86_64: installed
  backport-iwlwifi, 7906, 5.3.0-23-generic, x86_64: installed

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for backport-iwlwifi-dkms has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package backport-iwlwifi-dkms - 7906-0ubuntu2~19.10.1

---------------
backport-iwlwifi-dkms (7906-0ubuntu2~19.10.1) eoan; urgency=low

  * debian/patches/0006-NOUPSTREAM-backport-rename-ktime_get_boot_ns-for-v5..patch
    - fix build against v5.3. (LP: #1848922)
  * debian/patches/0005-Makefile.kernel-pass-fno-stack-clash-protection-and-.patch
    - add -fcf-protection=none when using retpoline flags. (LP: #1830961)
    - drop -fno-stack-clash-protection for it's no longer necessary.

 -- You-Sheng Yang <email address hidden> Mon, 21 Oct 2019 20:22:27 +0800

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[bob var (bobvar)]

This fixed my problem. I am reinstalling VirtualBox 6.1.4 and the command "sudo ln -fs gcc-8 /usr/bin/gcc" solve the problem.