Major memory corruption bug in 2.6.27 CONFIG_DYNAMIC_FTRACE

[Anders Kaseorg]

CONFIG_DYNAMIC_FTRACE was recently discovered to be very broken; it has a bug that leads to memory corruption with memory-mapped devices. This bug was the root cause of the e1000e EEPROM corruption. See here for details:

http://lkml.org/lkml/2008/10/15/337

Therefore, CONFIG_DYNAMIC_FTRACE needs to be disabled in the Intrepid kernel. CONFIG_FTRACE should be disabled as well, because it has very poor performance when CONFIG_DYNAMIC_FTRACE is disabled, even when ftrace is not being used. Also, CONFIG_FTRACE prevents Ksplice from building kernel updates (because gcc -pg disables -ffunction-sections).

[Leann Ogasawara (leannogasawara)]

Hi Anders,

This has already been addressed for the Intrepid kernel:

commit 834ffd02649a8a06a6ca4cfbfe671ec796925f12
Author: Tim Gardner <email address hidden>
Date: Thu Oct 16 07:48:42 2008 -0600

    UBUNTU: Update configs for 'disable CONFIG_DYNAMIC_FTRACE'
    Bug: #263555

    Upstream believes this to be the root cause for the e1000e EEPROM
    corruption.

    Signed-off-by: Tim Gardner <email address hidden>

[Anders Kaseorg (andersk)]

Thanks. Has any consideration been given to disabling CONFIG_FTRACE as well, or is that considered more important for Ubuntu than the performance penalty it incurs without CONFIG_DYNAMIC_FTRACE?

[Steve Langasek (vorlon)]

CONFIG_DYNAMIC_FTRACE has been disabled in the linux 2.6.27-7.12 upload. CONFIG_FTRACE is still enabled, though - do we need to get this disabled for release?

[Amit Kucheria (amitk)]

Hi Anders,

What sort of performance penalty are we talking about here?

From Documentation/ftrace.txt and I quote "If CONFIG_DYNAMIC_FTRACE is set, the system will run with virtually no overhead when function tracing is disabled".

Since in our case DYNAMIC_FTRACE is off, do you still think the overhead is significant?

[Tim Gardner (timg-tpi)]

linux (2.6.27-7.12) intrepid; urgency=low

  [ Chuck Short ]

  * xen: Add xen modules to virtual flavours.

  [ Mario Limonciello ]

  * SAUCE: Add back in lost commit for Apple BT Wireless Keyboard
    - LP: #162083

  [ Tim Gardner ]

  * Remove depmod created files from packages.
    - LP: #250511
  * Changed default TCP congestion algorithm to 'cubic' (again)
    - LP: #278801
  * Update configs for 'disable CONFIG_DYNAMIC_FTRACE'
    - LP: #263555

  [ Upstream Kernel Changes ]

  * x86: register a platform RTC device if PNP doesn't describe it
  * disable CONFIG_DYNAMIC_FTRACE due to possible memory corruption on
    module unload

 -- Tim Gardner <email address hidden> Fri, 17 Oct 2008 11:25:39 -0600

[Anders Kaseorg (andersk)]

CONFIG_FTRACE works by having the compiler insert a call to mcount() into the beginning of every kernel function. The author has quantified this overhead at 15–18% even for a completely empty mcount() function <http://lkml.org/lkml/2008/8/7/270>; the actual overhead is greater because the real mcount() function needs to test whether ftrace is enabled before returning.

CONFIG_DYNAMIC_FTRACE reduces this overhead to nearly zero by patching over the mcount() calls with NOPs at runtime, but obviously because of the bug this feature can no longer be used.

(That’s all with ftrace disabled at runtime. When it is enabled, the overhead jumps to about 5000%, which more or less grinds the system to a halt.)

So I would say that it is important to turn CONFIG_FTRACE off for the release, if that’s still possible.