Cannot scale AMD Turion CPU frequency. It runs on lowest 800MHz

[Darek Nawrocki]

http://askubuntu.com/questions/789096/cant-scale-frequencies-of-amd-turion-cpu-it-always-jumps-to-lowest

The CPU does not scale frequency on kernel "4.2.0-41-generic #48~14.04.1-Ubuntu SMP Fri Jun 24 17:09:15". The notebook behaves very slow despite on Windows Vista it works great.

Cpufreq-info program shows that there is a limit on the frequency setting between 800MHz and 800MHz. I tried to impose the frequency by setting the "performance policy" or by setting the limit in /etc/default/cpufrequency, but it does not work.
Next I tried to check the BIOS limits, because there are some suggestions that some battery/power supply problems could cause the BIOS to set the limits on CPU freq but it is not there.

The CPU freq scaling works good on 'linux-image-4.1.26-040126-generic' kernel. It does not work on this kernel "4.2.0-41-generic" and "linux-headers-4.2.0-38-generic". The current workaround is to switch off the ACPI by "acpi=off" boot parameter. Using this setting the CPU frequency scaling works but it makes the system single CPU core only.

Hardware: HP Compaq 6715B notebook with AMD Turion CPU TL-58 and powernow-k8 cpufreq driver. I've heard some posts that on others HP notebooks there are similar problems with the freq scaling.

$ cpufreq-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to <email address hidden>, please.
analyzing CPU 0:
  driver: powernow-k8
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 109 us.
  hardware limits: 800 MHz - 1.90 GHz
  available frequency steps: 1.90 GHz, 1.80 GHz, 1.60 GHz, 800 MHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 800 MHz and 800 MHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz.
  cpufreq stats: 1.90 GHz:0,07%, 1.80 GHz:0,00%, 1.60 GHz:0,00%, 800 MHz:99,93% (1)
analyzing CPU 1:
  driver: powernow-k8
  CPUs which run at the same hardware frequency: 0 1
  CPUs which need to have their frequency coordinated by software: 0 1
  maximum transition latency: 109 us.
  hardware limits: 800 MHz - 1.90 GHz
  available frequency steps: 1.90 GHz, 1.80 GHz, 1.60 GHz, 800 MHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 800 MHz and 800 MHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 800 MHz.
  cpufreq stats: 1.90 GHz:0,07%, 1.80 GHz:0,00%, 1.60 GHz:0,00%, 800 MHz:99,93% (1)

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-4.2.0-41-generic 4.2.0-41.48~14.04.1
ProcVersionSignature: Ubuntu 4.2.0-41.48~14.04.1-generic 4.2.8-ckt11
Uname: Linux 4.2.0-41-generic x86_64
NonfreeKernelModules: wl
ApportVersion: 2.14.1-0ubuntu3.21
Architecture: amd64
CurrentDesktop: KDE
Date: Fri Jul 1 23:49:34 2016
InstallationDate: Installed on 2016-04-08 (83 days ago)
InstallationMedia: Kubuntu 14.04.4 LTS "Trusty Tahr" - Release amd64 (20160217.1)
SourcePackage: linux-lts-wily
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Gronister (christiangronister)]

Same issue here on a HP Compaq 6715s (TL-59) Ubuntu 16.04 LTS "Xenial Xerus" (kernel 4.4.0-34). 14.04 - 14.04.3 was working fine (3.13, 3.16, 3.19 kernels). Upgrade to 14.04.4 initially caused the issue so I rolled back to 3.19 few months back.
Now I thought to give a fresh install of 16.04 a try. Did not work, i.e. precisely the same issue as in 14.04.4 and as described by Darek in #1598312.

[Lichtmacher (fh-j)]

Same Problem:
on HP6715s after upgrade from 14.04 to 16.04 Xubuntu

Additional the fan is turning at max speed.

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

$ uname -a
Linux HP6715S 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 104
model name : AMD Turion(tm) 64 X2 Mobile Technology TL-58
stepping : 2
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good nopl extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch vmmcall lbrv
bugs : apic_c1e fxsave_leak sysret_ss_attrs
bogomips : 1593.21
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 104
model name : AMD Turion(tm) 64 X2 Mobile Technology TL-58
stepping : 2
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good nopl extd_apicid pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch vmmcall lbrv
bugs : apic_c1e fxsave_leak sysret_ss_attrs
bogomips : 1593.21
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc 100mhzsteps

[Vladimir Marko (swelef)]

Same on HP Compaq nx6325 with AMD Turion(tm) 64 X2 Mobile Technology TL-52, including the fan running at full speed. But given the the thermal trip points, I think the CPU frequency scaling works as intended.

When idle, with the old kernel I see

$ uname -r
3.13.0-101-generic
$ cat /sys/devices/virtual/thermal/thermal_zone0/temp
42000
$ cat /sys/devices/virtual/thermal/thermal_zone0/trip_point_*_temp
105000
95000
75000
65000
55000
40000

but with the new kernel it is

$ uname -r
4.4.0-47-generic
$ cat /sys/devices/virtual/thermal/thermal_zone0/temp
32000
$ cat /sys/devices/virtual/thermal/thermal_zone0/trip_point_*_temp
105000
15900
15900
15900
15900
15900

So it's essentially trying to keep the CPU from overheating.

Additionally, shutting down the laptop does not work, it gets the laptop to some weird power state where it looks like powered down but it won't boot after turning back on. Then I have to long-press the power button to shut down properly. Reboot just gets stuck. If I understand correctly, this is all related to ACPI, so I guess I should learn how to debug ACPI issues.

[Vladimir Marko (swelef)]

Trying the different kernel parameters suggested at https://wiki.ubuntu.com/DebuggingACPI I've got these results:

acpi=off: kernel panic (something too do with PCI)
acpi=ht: no change
pci=noacpi: kernel panic (something too do with PCI)
acpi=noirq: WORKS

So this is a bug in ACPI IRQ routing. Recommended workaround is acpi=noirq.
(Now how to configure grub to add that kernel parameter when doing update-grub?)

[Vladimir Marko (swelef)]

Permanently adding kernel parameters to grub config is described at
    http://askubuntu.com/questions/19486/how-do-i-add-a-kernel-boot-parameter
(But it's all or nothing, so you cannot add the parameter only to one of two dual-booting Linuxes.)

[Vladimir Marko (swelef)]

Upgrading the nx6325 BIOS from F.04 to F.07 did not help, the workaround is still necessary.

[Dariusz Nawrocki (dnawrock+launchpad)]

Vladimir, your solution works for me too. :)

I have F.07 BIOS already. Setting acpi=noirq enables the possibilities to scale CPU processor frequency again on newest kernel on my notebook.

However it is workaround not the final solution. It is nice to have well working kernel out of the box without any quirks. So this is an appeal for Ubuntu kernel guys to address the problem.

Thanks Vladimir.

[Vladimir Marko (swelef)]

The workaround makes networking very unstable (both WiFi and wired).

4.2.0-18 is OK.
4.2.0-19 is BROKEN.

No suspicious configuration changes:

$ diff /boot/config-4.2.0-18-generic /boot/config-4.2.0-19-generic
3c3
< # Linux/x86_64 4.2.0-18-generic Kernel Configuration
---
> # Linux/x86_64 4.2.0-19-generic Kernel Configuration
69c69
< CONFIG_VERSION_SIGNATURE="Ubuntu 4.2.0-18.22~14.04.1-generic 4.2.3"
---
> CONFIG_VERSION_SIGNATURE="Ubuntu 4.2.0-19.23~14.04.1-generic 4.2.6"
5334c5334,5335
< # CONFIG_SND_PCM_OSS is not set
---
> CONFIG_SND_PCM_OSS=m
> CONFIG_SND_PCM_OSS_PLUGINS=y
7530c7531,7532
< # CONFIG_AUFS_EXPORT is not set
---
> CONFIG_AUFS_EXPORT=y
> CONFIG_AUFS_INO_T_64=y

I'll get the ubuntu kernel source and try to bisect.

[sntmail (sntmail-linux-kernel-bugs)]

Created attachment 246061
cpuinfo

My laptop is very slow at current kernels. The frequency stucks at 800 MHz.
Latest kernel which is ok is 4.1.x (4.2 has a backlight problem).

acpi=noirq solves the problem but adds other problems...

[sntmail (sntmail-linux-kernel-bugs)]

Created attachment 246071
lspci

[sntmail (sntmail-linux-kernel-bugs)]

Created attachment 246081
dmesg for 4.1.6 kernel

[sntmail (sntmail-linux-kernel-bugs)]

Created attachment 246091
dmesg for 4.8.10 kernel

[sntmail (sntmail-linux-kernel-bugs)]

Created attachment 246101
dmesg for 4.8.10 kernel with acpi=noirq

[Vladimir Marko (swelef)]

3b806e2b94cad37a8809df7c86f7cfdcd3baa719 is the first bad commit
commit 3b806e2b94cad37a8809df7c86f7cfdcd3baa719
Author: Thomas Gleixner <email address hidden>
Date: Mon Sep 14 12:00:55 2015 +0200

    x86/ioapic: Force affinity setting in setup_ioapic_dest()

    BugLink: http://bugs.launchpad.net/bugs/1509886

    commit 4857c91f0d195f05908fff296ba1ec5fca87066c upstream.

    The recent ioapic cleanups changed the affinity setting in
    setup_ioapic_dest() from a direct write to the hardware to the delayed
    affinity setup via irq_set_affinity().

    That results in a warning from chained_irq_exit():
    WARNING: CPU: 0 PID: 5 at kernel/irq/migration.c:32 irq_move_masked_irq
    [<ffffffff810a0a88>] irq_move_masked_irq+0xb8/0xc0
    [<ffffffff8103c161>] ioapic_ack_level+0x111/0x130
    [<ffffffff812bbfe8>] intel_gpio_irq_handler+0x148/0x1c0

    The reason is that irq_set_affinity() does not write directly to the
    hardware. It marks the affinity setting as pending and executes it
    from the next interrupt. The chained handler infrastructure does not
    take the irq descriptor lock for performance reasons because such a
    chained interrupt is not visible to any interfaces. So the delayed
    affinity setting triggers the warning in irq_move_masked_irq().

    Restore the old behaviour by calling the set_affinity function of the
    ioapic chip in setup_ioapic_dest(). This is safe as none of the
    interrupts can be on the fly at this point.

    Fixes: aa5cb97f14a2 'x86/irq: Remove x86_io_apic_ops.set_affinity and related interfaces'
    Reported-and-tested-by: Mika Westerberg <email address hidden>
    Signed-off-by: Thomas Gleixner <email address hidden>
    Cc: Jiang Liu <email address hidden>
    Cc: <email address hidden>
    Signed-off-by: Greg Kroah-Hartman <email address hidden>

    Signed-off-by: Tim Gardner <email address hidden>
    Signed-off-by: Brad Figg <email address hidden>

:040000 040000 0233ddd54b0d7f34ceb513fe09e8df462ab8b777 e811435d773d53cdb9c2a26e68586f0dc2e41bf8 M arch

[lenb (lenb-linux-kernel-bugs)]

acpi=noirq puts the system into PIC mode, while ACPI is able to put it in IOAPIC mode. Unclear why this has any effect on the problem at hand. Do you get the same result with simply "noapic"?

In both cases, powernow_k8 seems to load and print the same messages.

In the working and non-working cases, what do you see with

grep . /sys/devices/system/cpu/cpu0/cpufreq/*

What is the latest working kernel, and what is the first failing kernel? Can you git-bisect to what commit caused this to break?

[Vladimir Marko (swelef)]

The revert of the offending commit does not apply cleanly to the Ubuntu-4.4.0-47.68 kernel. The closest we can get to a revert is this patch:

diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 64f233a..aec995c 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -2523,7 +2523,6 @@ void __init setup_ioapic_dest(void)
        const struct cpumask *mask;
        struct irq_desc *desc;
        struct irq_data *idata;
- struct irq_chip *chip;

        if (skip_ioapic_setup == 1)
                return;
@@ -2538,7 +2537,6 @@ void __init setup_ioapic_dest(void)
                        continue;

                desc = irq_to_desc(irq);
- raw_spin_lock_irq(&desc->lock);
                idata = irq_desc_get_irq_data(desc);

                /*
@@ -2549,11 +2547,7 @@ void __init setup_ioapic_dest(void)
                else
                        mask = apic->target_cpus();

- chip = irq_data_get_irq_chip(idata);
- /* Might be lapic_chip for irq 0 */
- if (chip->irq_set_affinity)
- chip->irq_set_affinity(idata, mask, false);
- raw_spin_unlock_irq(&desc->lock);
+ irq_set_affinity(irq, mask);
        }
 }
 #endif

I tried this and it works. Another option is this smaller patch:

diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 64f233a..f185a1e 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -2523,7 +2523,6 @@ void __init setup_ioapic_dest(void)
        const struct cpumask *mask;
        struct irq_desc *desc;
        struct irq_data *idata;
- struct irq_chip *chip;

        if (skip_ioapic_setup == 1)
                return;
@@ -2549,10 +2548,7 @@ void __init setup_ioapic_dest(void)
                else
                        mask = apic->target_cpus();

- chip = irq_data_get_irq_chip(idata);
- /* Might be lapic_chip for irq 0 */
- if (chip->irq_set_affinity)
- chip->irq_set_affinity(idata, mask, false);
+ irq_set_affinity_locked(idata, mask, false);
                raw_spin_unlock_irq(&desc->lock);
        }
 }

which essentially keeps the raw_spin_lock_irq()/raw_spin_unlock_irq() rather than raw_spin_lock_irqsave()/raw_spin_unlock_irqrestore() used by __irq_set_affinity() in kernel/irq/manage.c .

I tested both patches on top of Ubuntu-4.4.0-47.68 and they work.
They also apply cleanly to Ubuntu-4.2.0-42.49 but I didn't actually check that it builds/works.

I tried manually inlining the irq_set_affinity_locked() (which required further inlining of some helper functions and providing a declaration of irq_do_set_affinity()) and it still worked. However, forcing the "if (irq_can_move_pcntxt())" path that immediately performs "chip->irq_set_affinity(idata, mask, false)" exposes the bug.

This confirms what I expected after reading https://lkml.org/lkml/2012/3/26/392 . Basically, when the hardware says that you cannot do something right away, you really cannot. And the offending commit just ignores this.

[Vladimir Marko (swelef)]

Looks like pasting the patches directly to the editor wasn't a good idea. It messed up the whitespace. Attaching the "revert" patch.

[Vladimir Marko (swelef)]

Attaching the "alternative" patch.

[Vladimir Marko (swelef)]

This patches also apply cleanly to current master in
    git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
with 1 line offset.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.9 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9-rc8

[Vladimir Marko (swelef)]

Installed
    linux-headers-4.9.0-040900rc8_4.9.0-040900rc8.201612051443_all.deb
    linux-headers-4.9.0-040900rc8-generic_4.9.0-040900rc8.201612051443_amd64.deb
    linux-image-4.9.0-040900rc8-generic_4.9.0-040900rc8.201612051443_amd64.deb
Laptop boots, bug still present:

$ uname -r
4.9.0-040900rc8-generic
$ cat /sys/devices/virtual/thermal/thermal_zone0/trip_point_*_temp
105000
15900
15900
15900
15900
15900
$ cpufreq-info | grep -E 'policy|steps'
  available frequency steps: 1.60 GHz, 800 MHz
  current policy: frequency should be within 800 MHz and 800 MHz.
  available frequency steps: 1.60 GHz, 800 MHz
  current policy: frequency should be within 800 MHz and 800 MHz.

[Vladimir Marko (swelef)]

Both patches apply cleanly to the 4.9-rc8 but I cannot build the kernel with the configuration /boot/config-4.9.0-040900rc8-generic on my laptop without upgrading the compiler:

Cannot use CONFIG_CC_STACKPROTECTOR_STRONG: -fstack-protector-strong not supported by compiler

[sntmail (sntmail-linux-kernel-bugs)]

acpi=noapic doesn't help this.

I think the regression was made in 4.2.x or 4.3.x (4.2.x kernels have a backlight issue on my laptop, so I use 4.1.x)

[sntmail (sntmail-linux-kernel-bugs)]

Created attachment 247171
grep . /sys/devices/system/cpu/cpu0/cpufreq/* for 4.1.6 (ok)

[sntmail (sntmail-linux-kernel-bugs)]

Created attachment 247181
grep . /sys/devices/system/cpu/cpu0/cpufreq/* for 4.8.12 (bug)

[sntmail (sntmail-linux-kernel-bugs)]

btw, seems the same bug was described here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1598312

[Vladimir Marko (swelef)]

Sorry, I missed the request in comment #17 to add the tag kernel-bug-exists-upstream. Done now.

[rui.zhang (rui.zhang-linux-kernel-bugs)]

(In reply to cthx from comment #6)
> acpi=noapic doesn't help this.
>
> I think the regression was made in 4.2.x or 4.3.x (4.2.x kernels have a
> backlight issue on my laptop, so I use 4.1.x)

what do you mean by backlight issue?
If the platform is still alive, please run git-bisect to find out which commit introduces the problem.

[rui.zhang (rui.zhang-linux-kernel-bugs)]

this is only one functional change in powernow-k8 driver since 4.1
commit 38c52e6343f0e28abc7daf15cbbcd7e450667202
Author: Bartosz Golaszewski <email address hidden>
Date: Tue May 26 15:11:31 2015 +0200

    powernow-k8: Replace cpu_core_mask() with topology_core_cpumask()

    The former duplicates the functionality of the latter but is
    neither documented nor arch-independent.

    Signed-off-by: Bartosz Golaszewski <email address hidden>
    Acked-by: Viresh Kumar <email address hidden>
    Acked-by: Rafael J. Wysocki <email address hidden>
    Cc: Benoit Cousson <email address hidden>
    Cc: Catalin Marinas <email address hidden>
    Cc: Fenghua Yu <email address hidden>
    Cc: Guenter Roeck <email address hidden>
    Cc: Jean Delvare <email address hidden>
    Cc: Jonathan Corbet <email address hidden>
    Cc: Linus Torvalds <email address hidden>
    Cc: Oleg Drokin <email address hidden>
    Cc: Peter Zijlstra <email address hidden>
    Cc: Russell King <email address hidden>
    Cc: Thomas Gleixner <email address hidden>
    Link: http://<email address hidden>
    Signed-off-by: Ingo Molnar <email address hidden>

can you please check if "git revert 38c52e6343f0e28abc7daf15cbbcd7e450667202" fixes the problem or not?

[Joseph Salisbury (jsalisbury)]

Thank you for providing a patch, and making Ubuntu better.

Can you provide some information on the status of the patch with regards to getting it merged upstream? Has it been sent upstream, what sort of feedback has it received, is it getting applied to a subsystem maintainer's tree, etc?

[Vladimir Marko (swelef)]

I didn't try to send the patches upstream. I was hoping that someone who already knows how to do that would pick it up.

I just searched the kernel bug database for this issue and found https://bugzilla.kernel.org/show_bug.cgi?id=189291 . In comment #9, cthx posted a link to this bug on 9 Dec, i.e. after I have already posted the bisect results and patches. I guess people don't bother following links, I'll have to post there directly.

[swelef (swelef-linux-kernel-bugs)]

If you followed the link in comment #9 (posted 2016-12-09), you would have found the results of the bisect for Ubuntu (posted 2016-11-30) and two possible patches (posted 2016-12-08).

The bisect for Ubuntu yields

3b806e2b94cad37a8809df7c86f7cfdcd3baa719 is the first bad commit
commit 3b806e2b94cad37a8809df7c86f7cfdcd3baa719
Author: Thomas Gleixner <email address hidden>
Date: Mon Sep 14 12:00:55 2015 +0200

    x86/ioapic: Force affinity setting in setup_ioapic_dest()

    BugLink: http://bugs.launchpad.net/bugs/1509886

    commit 4857c91f0d195f05908fff296ba1ec5fca87066c upstream.

    ...

The commit 4857c91f0d195f05908fff296ba1ec5fca87066c does not revert cleanly in the current kernel tree but it's rather trivial to do it manually. The two possible patches I posted on the Ubuntu bug were successfully tested to fix the issue.

[rui.zhang (rui.zhang-linux-kernel-bugs)]

(In reply to Vladimir Marko from comment #12)
> If you followed the link in comment #9 (posted 2016-12-09), you would have
> found the results of the bisect for Ubuntu (posted 2016-11-30) and two
> possible patches (posted 2016-12-08).
>
> The bisect for Ubuntu yields
>
> 3b806e2b94cad37a8809df7c86f7cfdcd3baa719 is the first bad commit
> commit 3b806e2b94cad37a8809df7c86f7cfdcd3baa719
> Author: Thomas Gleixner <email address hidden>
> Date: Mon Sep 14 12:00:55 2015 +0200
>
> x86/ioapic: Force affinity setting in setup_ioapic_dest()
>
> BugLink: http://bugs.launchpad.net/bugs/1509886
>
> commit 4857c91f0d195f05908fff296ba1ec5fca87066c upstream.
>
> ...
>
> The commit 4857c91f0d195f05908fff296ba1ec5fca87066c does not revert cleanly
> in the current kernel tree but it's rather trivial to do it manually. The
> two possible patches I posted on the Ubuntu bug were successfully tested to
> fix the issue.

Cool. Thanks for your effort.
And we have two fixes that have been verified to fix the problem
https://launchpadlibrarian.net/297109286/revert.patch
and
https://launchpadlibrarian.net/297109378/alternative.patch

Thomas,
can you please take a look at this issue?

[tglx (tglx-linux-kernel-bugs)]

On Mon, 16 Jan 2017, <email address hidden> wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=189291
>
> Zhang Rui <email address hidden> changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> Assignee|<email address hidden> |<email address hidden>
>
> --- Comment #13 from Zhang Rui <email address hidden> ---
> (In reply to Vladimir Marko from comment #12)
> > If you followed the link in comment #9 (posted 2016-12-09), you would have
> > found the results of the bisect for Ubuntu (posted 2016-11-30) and two
> > possible patches (posted 2016-12-08).
> >
> > The bisect for Ubuntu yields
> >
> > 3b806e2b94cad37a8809df7c86f7cfdcd3baa719 is the first bad commit
> > commit 3b806e2b94cad37a8809df7c86f7cfdcd3baa719
> > Author: Thomas Gleixner <email address hidden>
> > Date: Mon Sep 14 12:00:55 2015 +0200
> >
> > x86/ioapic: Force affinity setting in setup_ioapic_dest()
> >
> > BugLink: http://bugs.launchpad.net/bugs/1509886
> >
> > commit 4857c91f0d195f05908fff296ba1ec5fca87066c upstream.
> >
> > ...
> >
> > The commit 4857c91f0d195f05908fff296ba1ec5fca87066c does not revert cleanly
> > in the current kernel tree but it's rather trivial to do it manually. The
> > two possible patches I posted on the Ubuntu bug were successfully tested to
> > fix the issue.
>
> Cool. Thanks for your effort.
> And we have two fixes that have been verified to fix the problem
> https://launchpadlibrarian.net/297109286/revert.patch
> and
> https://launchpadlibrarian.net/297109378/alternative.patch

Neither of those two are fixes. They paper over the problem.

What's worse they reintroduce the regression which was fixed with this
commit.

The old code, i.e. before the conversion to the hierarchical irq domain
wrote directly to the io apic to set the destination and that got dropped
during the conversion. The bisected commit restored the original behaviour.

So now the question is why the revert or the other patch 'fixes' the problem.

Can I please get for both the unmodified kernel and the patched kernel the
following information:

  Boot with apic=verbose on the kernel command line

  Gather dmesg output (full boot log)

  Output from 'cat /proc/interrupts'

Thanks,

 tglx

[swelef (swelef-linux-kernel-bugs)]

Created attachment 252711
good and bad dmesg and /proc/interrupts

Adding archive with dmesg and /proc/interrupts for a good and bad kernel.

Bad: Built at commit 4c9eff7af69c61749b9eb09141f18f35edbf2210, with amd64
generic config from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10-rc4/
except for using regular stack protector because
    Cannot use CONFIG_CC_STACKPROTECTOR_STRONG:
    -fstack-protector-strong not supported by compiler

Good: Built with the additional revert.patch from
    https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1598312

[swelef (swelef-linux-kernel-bugs)]

The irq_set_affinity() call in setup_ipapic_dest() that the commit
    4857c91f0d195f05908fff296ba1ec5fca87066c
replaces was introduced in commit
    aa5cb97f14a2dd5aefabed6538c35ebc087d7c24

I built a kernel at aa5cb97f14a2dd5aefabed6538c35ebc087d7c24 with
the cherry-pick 4857c91f0d195f05908fff296ba1ec5fca87066c and it
worked fine. I conclude that the bug was introduced somewhere in
between and aa5cb97f14a2dd5aefabed6538c35ebc087d7c24 exposed it.

I shall try and bisect again, cherry-picking
    4857c91f0d195f05908fff296ba1ec5fca87066c
at every iteration to expose the underlying bug.
That may take a while.

[swelef (swelef-linux-kernel-bugs)]

Bisecting with
    4857c91f0d195f05908fff296ba1ec5fca87066c
applied on top yields

0be275e3a5607b23f5132121bca22a10ee23aa99 is the first bad commit
commit 0be275e3a5607b23f5132121bca22a10ee23aa99
Author: Jiang Liu <email address hidden>
Date: Tue Apr 14 10:29:57 2015 +0800

    x86/irq: Use cached IOAPIC entry instead of reading from hardware

    Use cached IOAPIC entry instead of reading data from IOAPIC hardware
    registers to improve performance.

    Signed-off-by: Jiang Liu <email address hidden>
    Tested-by: Joerg Roedel <email address hidden>
    Cc: Konrad Rzeszutek Wilk <email address hidden>
    Cc: David Cohen <email address hidden>
    Cc: Sander Eikelenboom <email address hidden>
    Cc: David Vrabel <email address hidden>
    Cc: Tony Luck <email address hidden>
    Cc: Joerg Roedel <[e-mail]>
    Cc: Greg Kroah-Hartman <email address hidden>
    Cc: Bjorn Helgaas <email address hidden>
    Cc: Benjamin Herrenschmidt <email address hidden>
    Cc: Rafael J. Wysocki <email address hidden>
    Cc: Randy Dunlap <email address hidden>
    Cc: Yinghai Lu <email address hidden>
    Cc: Borislav Petkov <email address hidden>
    Cc: Dimitri Sivanich <email address hidden>
    Cc: Grant Likely <email address hidden>
    Link: http://<email address hidden>
    Signed-off-by: Thomas Gleixner <email address hidden>

:040000 040000 c3f631e46c7cee67fd8f1d900d307eabe53341b4 fcce52d36b3a12568a7b4e0033c5578e65bb37e6 M arch

[swelef (swelef-linux-kernel-bugs)]

Created attachment 254511
investigation.tar.bz2

Data from investigation on top of the first bad commit.

[swelef (swelef-linux-kernel-bugs)]

Created attachment 254521
investigation.tar.bz2 with the debug patch

[swelef (swelef-linux-kernel-bugs)]

I've investigated the differences between the writes done before and after
    0be275e3a5607b23f5132121bca22a10ee23aa99 ("offending commit")
with the cherry-pick
    4857c91f0d195f05908fff296ba1ec5fca87066c ("exposing commit")
on top. Please see the data in the attachment added in comment #19.
(Comment #18 didn't contain the patch needed to interpret the data.)

You can "grep SWELEF" in the "good" dmesg output to see that the
"offending commit", despite saying that it is using "cached APIC entry",
actually writes very different data to the APIC entry at the beginning,
compared to the previous read-modify-write approach. Compare the "eu.w1"
values against the final "reg" value after the "->".

The funny thing is that the "delayed" dmesg (with the "exposing commit"
reverted, thus delaying the writes from setup_ioapic_dest()) does not
show any discrepancy for ioapic_set_affinity(). Some discrepancies remain
for io_apic_modify_irq() but these are immaterial as I checked by testing
with only the io_apic_modify_irq() #if to the "reverted" state.

Any further interpretation of the data is beyond my abilities.
Please have a look.

[alexey.kv (alexey.kv-linux-kernel-bugs)]

Have exactly the same problem on my 6715s. Thanks to the original bugreporter for bisecting.
If there is something I can test, I'll be more than happy to do it.

[lenb (lenb-linux-kernel-bugs)]

Is this still a problem in Linux-4.10?

[alexey.kv (alexey.kv-linux-kernel-bugs)]

(In reply to Len Brown from comment #22)
> Is this still a problem in Linux-4.10?

Yes. Linux-4.10.1

[swelef (swelef-linux-kernel-bugs)]

Using WARN_ON(.) I have determined that during initialization
  - ioapic entries are read from
    [<ffffffff810524e8>] __ioapic_read_entry+0x88/0xa0
    [<ffffffff81052532>] ioapic_read_entry+0x32/0x60
    [<ffffffff81d67f71>] enable_IO_APIC+0x63/0x10b
    [<ffffffff81d67001>] apic_bsp_setup+0x89/0xae
    [<ffffffff81d64cc4>] native_smp_prepare_cpus+0x2cf/0x30d
    [<ffffffff81d511a6>] kernel_init_freeable+0xa5/0x1ef
      kernel_init_freeable() init/main.c:995
      {inlined} do_pre_smp_initcalls() init/main.c:888
          do_one_initcall(*fn);
  - mp_irqdomain_activate() is called for apic=0, pin=0 (timer irq) from
    [<ffffffff81054aec>] mp_irqdomain_activate+0x9c/0xb0
    [<ffffffff810da501>] irq_domain_activate_irq+0x41/0x50
    [<ffffffff81d6887c>] setup_IO_APIC+0x687/0x830
    [<ffffffff81052c6d>] ? clear_IO_APIC+0x4d/0x70
    [<ffffffff81d6701a>] apic_bsp_setup+0xa2/0xae
    [<ffffffff81d64cc4>] native_smp_prepare_cpus+0x2cf/0x30d
    [<ffffffff81d511a6>] kernel_init_freeable+0xa5/0x1ef
      kernel_init_freeable() init/main.c:995
      {inlined} do_pre_smp_initcalls() init/main.c:888
          do_one_initcall(*fn);
  - ioapic_set_affinity() is called from
    [<ffffffff8105284c>] ioapic_set_affinity+0xdc/0xf0
    [<ffffffff81d68afd>] setup_ioapic_dest+0xd8/0xf6
    [<ffffffff81d64e39>] native_smp_cpus_done+0x10a/0x117
    [<ffffffff81d7633c>] smp_init+0x69/0x88
    [<ffffffff81d511dc>] kernel_init_freeable+0xdb/0x1ef
      kernel_init_freeable() init/main.c:999
          sched_init_smp();
  - the next mp_irqdomain_activate() is for apic=0, pin=21 and called from
    [<ffffffff81054aec>] mp_irqdomain_activate+0x9c/0xb0
    [<ffffffff810da501>] irq_domain_activate_irq+0x41/0x50
    [<ffffffff810d6f4c>] irq_startup+0x2c/0x80
    [<ffffffff810d5801>] __setup_irq+0x511/0x5a0
    [<ffffffff811dbbd2>] ? kmem_cache_alloc_trace+0x1e2/0x220
    [<ffffffff810d59cd>] ? request_threaded_irq+0xad/0x1b0
    [<ffffffff814409e4>] ? acpi_osi_handler+0xa9/0xa9
    [<ffffffff814409e4>] ? acpi_osi_handler+0xa9/0xa9
    [<ffffffff810d5a17>] request_threaded_irq+0xf7/0x1b0
    [<ffffffff814575be>] ? acpi_ev_sci_dispatch+0x65/0x65
    [<ffffffff81d99f47>] ? acpi_sleep_proc_init+0x2a/0x2a
    [<ffffffff81440ef5>] acpi_os_install_interrupt_handler+0x92/0xc6
    [<ffffffff8145762f>] acpi_ev_install_sci_handler+0x23/0x25
    [<ffffffff81454f72>] acpi_ev_install_xrupt_handlers+0x1c/0x6e
    [<ffffffff81d9b4ba>] acpi_enable_subsystem+0x8b/0x90
    [<ffffffff81d99fbc>] acpi_init+0x75/0x267
    [<ffffffff81d99f47>] ? acpi_sleep_proc_init+0x2a/0x2a
    [<ffffffff81002144>] do_one_initcall+0xd4/0x210
    [<ffffffff81098400>] ? parse_args+0x190/0x480
    [<ffffffff810bbe28>] ? __wake_up+0x48/0x60
    [<ffffffff81d51263>] kernel_init_freeable+0x162/0x1ef
      kernel_init_freeable() init/main.c:1001
      {inlined} do_basic_setup() init/main.c:880
      {inlined} do_initcalls() init/main.c:861
      {inlined} do_initcall_level(.) init/main.c:853
          do_one_initcall(*fn);

The timer irq (pin=0) seems to be special (legacy irq?) and its mp_irqdomain_activate() is called early through setup_IO_APIC() and check_timer(). However, the rest of the mp_irqdomain_activa...

[swelef (swelef-linux-kernel-bugs)]

The IO_APIC_route_entry initialized in the mp_setup_entry() has mask=0 even though IRQs have not been enabled at that point in the kernel initialization. And since there is no other field where the kernel would remember whether IRQs have been enabled, the early "chip->irq_set_affinity(...)" from setup_ioapic_dest() actually enables IRQs way before the necessary structures have been initialized.

I've also tried to move the setup_ioapic_dest() call to a different place during the initialization. When it's before acpi_bus_init() in acpi_init(), it's broken. If it's after that, everything is OK.

This bug can be fixed in several ways. In my order of preference:

1. Revert 1f934641294ca2e09016c689862378fbb15da4d4,
      and 0be275e3a5607b23f5132121bca22a10ee23aa99.
   The commit message of the latter does not match reality,
   the so-called "cached" entry is not what's been written,
   so returning to RMW operation is preferable.

2. Initialize the IO_APIC_route_entry with mask=1 and update
   this flag when interrupts are enabled/disabled.

3. Move setup_ioapic_dest() after acpi_bus_init() in acpi_init(),
   or to another appropriate place inside acpi_bus_init().

4. Revert 4857c91f0d195f05908fff296ba1ec5fca87066c.
   (This has already been refused by Thomas Gleixner.)

I can try and prepare a patch but the maintainer (Thomas Gleixner) needs to decide which approach to take. I'm not going to unnecessarily write multiple patches.

[alexey.kv (alexey.kv-linux-kernel-bugs)]

I've compiled and tested #1 approach (revert f934641294ca2e09016c689862378fbb15da4d4,
and 0be275e3a5607b23f5132121bca22a10ee23aa99)
as Vladimir Marko suggested.

It works very good for me so far on 4.9.16

[swelef (swelef-linux-kernel-bugs)]

I've built a 4.11 kernel with Ubuntu config-4.11.0-041100rc8-generic (and stack protector reduced to regular) and verifed that the issue is still present. However, I do not see any way to edit the bug to change either the affected kernel version or the status.

[dnawrock+kernelorg (dnawrock+kernelorg-linux-kernel-bugs)]

I have built a fresh kernel from current master branch - commit "89c9fea3c8034cdb2fd745f551cde0b507fd6893" . The bug still exists - the CPU frequency is limited to 800 MHz.

After I reverted the two commits above on my local master branch ( 1f934641294ca2e09016c689862378fbb15da4d4
 and 0be275e3a5607b23f5132121bca22a10ee23aa99 ) and built the kernel again the notebook works correctly. It can scale frequency within full range up to 1900 MHz.

[marcodv (marcodv-linux-kernel-bugs)]

Linux 4.10.0-22-generic #24-Ubuntu SMP Mon May 22 17:43:20 UTC 2017 x86_64 GNU/Linux

The same issue on the same notebook.

But I wanted to share a bit about kernel parameters. `acpi=noirq` works around the cpu scaling successfully, but the wifi got broken (though I have tried only one boot so I don't know how statistically significant that is). Then I tried `noapic` (It seems like comment #6 tried `acpi=noapic` instead) and both cpu and wlan seem to work.

What's the status of the patch?

[swelef (swelef-linux-kernel-bugs)]

Awaiting reply from Thomas Gleixner (see comment #25) who seems to be ignoring this bug (last comment at 2017-01-19 16:40:17 UTC) and even a direct email (May 1, 2017). I'm wondering if it really takes public "Linus Nvidia rant" to make things move forward.

[marcodv (marcodv-linux-kernel-bugs)]

I don't know if those issues are related (since they seem to be all ACPI related), but the reboot/powerdown are not shutting the notebook properly. Do your proposed patches solve also this issue? Or are they separate?

[swelef (swelef-linux-kernel-bugs)]

Yes, each of the proposed fixes solves the broken powerdown/reboot as well.

[WSmart (wsmart)]

Thanks for all the work everyone put into developing this, Vladimir in particular. The acpi=noirq work around is working here. I wanted to add, along with the scaling and the reboot fail issues, I also was not able to run ubiquity. The installer would crash giving me freedesktop.org errors. I installed xfce4-power-manger to the live environment and was able to use ubiquity and install the system from there, booting to the 800mhz scaling limit.

Thanks all.
With alcohol you get all the confidence and none of the competence.

[manuel (manuel-linux-kernel-bugs)]

Given that this is still an issue, and that the 4.1 kernel is affected by the meltdown/spectre vulnerability, what's the current way of properly resolving this?

[swelef (swelef-linux-kernel-bugs)]

Re comment #33: What does this bug have to do with the meltdown/spectre?
(This was still an issue when I tried 4.14.)

[manuel (manuel-linux-kernel-bugs)]

(In reply to Vladimir Marko from comment #34)
> Re comment #33: What does this bug have to do with the meltdown/spectre?
> (This was still an issue when I tried 4.14.)

The bug itself doesn't have anything to do with meltdown/spectre. But one potential resolution (using the 4.1 kernel) conflicts with it.

[swelef (swelef-linux-kernel-bugs)]

I just tried 4.16 and everything works fine. Looking at the history of arch/x86/kernel/apic/io_apic.c , I would guess the fix was actually

    commit 90ad9e2d91067983f3328e21b306323877e5f48a
    Author: Thomas Gleixner <email address hidden>
    Date: Wed Sep 13 23:29:49 2017 +0200

        x86/io_apic: Reevaluate vector configuration on activate()
        [...]

which was already present in 4.15.

[swelef (swelef-linux-kernel-bugs)]

Reverting the commit 90ad9e2d91067983f3328e21b306323877e5f48a (see comment #36) did not reintroduce the bug. So it remains unknown where between 4.14 and 4.16 this bug was fixed and whether 4.15 is affected.

[Vladimir Marko (swelef)]

Upstream:
Linux 4.14: BAD
Linux 4.16: GOOD

I did not bisect to find out which CL fixed the bug.

[manuel (manuel-linux-kernel-bugs)]

I just tested 4.15 on nixOS 18.03. The fan is quiet and the temperature can be set to all possible values via "cpupower frequency-set -f". Weirdly, the ondemand governor cranks up the frequency to maximum even if not under load, but that's maybe a separate bug, and no big deal.

[alexey.kv (alexey.kv-linux-kernel-bugs)]

Tested on 4.14.73 - works bad.
On 4.18.11 works good for me, including frequency scaling.