Comment 7 for bug 1667416

Revision history for this message
Michael Hohnbaum (hohnbaum) wrote : Re: [Bug 1667416] [NEW] PowerNV: No rate limit for kernel error "KVM can't copy data from"

Patch for the Kernel Team to evaluate.

                      Michael

On 02/23/2017 10:11 AM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> == Comment: #0 - MIKHAIL S. MEDVEDEV <email address hidden> - 2017-02-03 14:40:58 ==
> ---Problem Description---
> I am experiencing a problem with KVM on Power with a symptom of "KVM can't copy data" message being spammed to kern.log at about 100000 messages a second. Apart from original problem that triggers the error, the printk is a problem on its own. System might run out of space for logs, and would have high CPU load caused by journald.
>
> This bug is to address an absence of rate limiter on printk at
> https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268
>
> Maybe wrapping the printk in printk_ratelimit() is all that is required?
>
> if(printk_ratelimit()){
> printk("KVM can't copy");
> }
>
> Contact Information = Mikhail Medvedev / <email address hidden>, Rafael Folco / <email address hidden>
>
> ---uname output---
> 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
>
> Machine Type = KVM guest on P8 8247 21L
>
> ---Debugger---
> A debugger is not configured
>
> ---Steps to Reproduce---
> 1. Deploy OpenStack with devstack
> 2. Run simultaneously between 4 and 6 instances of a script that
> - boots a VM (openstack server create)
> - immediately creates the backup of the VM (openstack server backup create)
>
> This is not a very good way to reproduce it. I am still unable to
> reproduce the problem just by using libvirt and qemu.
>
>
> == Comment: #5 - VIPIN K. PARASHAR <email address hidden> - 2017-02-07 09:42:31 ==
>
> >From kernel logs
> ===========
>
> [852719.001593] KVM can't copy data from 0x3fff86683c80!
> [852719.001614] KVM can't copy data from 0x3fff86683c80!
> [852719.001617] KVM can't copy data from 0x3fff66a83c80!
> [852719.001624] KVM can't copy data from 0x3fff86683c80!
> [852719.001634] KVM can't copy data from 0x3fff86683c80!
> [852719.001637] KVM can't copy data from 0x3fff66a83c80!
> [852719.001645] KVM can't copy data from 0x3fff86683c80!
> [852719.001655] KVM can't copy data from 0x3fff86683c80!
> [852719.001658] KVM can't copy data from 0x3fff66a83c80!
> [852719.001665] KVM can't copy data from 0x3fff86683c80!
> [852719.001668] KVM can't copy data from 0x3fff66a83c80!
> [852719.001678] KVM can't copy data from 0x3fff66a83c80!
> [852719.001686] KVM can't copy data from 0x3fff86683c80!
>
>
> "KVM can't copy data from" error is being logged at very high rate
> by kernel. rate limiting this error should fix this issue.
>
>
> == Comment: #20 - VIPIN K. PARASHAR <email address hidden> - 2017-02-23 12:01:26 ==
> Fix for this is submitted to PowerPC mailing list
> and is available at below location:
>
> https://patchwork.ozlabs.org/patch/728831/
>
> It reads:
>
> "KVM: PPC: Book3S: Ratelimit copy data failure error messages"
>
> ** Affects: ubuntu
> Importance: Undecided
> Assignee: Taco Screen team (taco-screen-team)
> Status: New
>
>
> ** Tags: architecture-ppc64le bugnameltc-151239 severity-medium targetmilestone-inin1704

--
Michael Hohnbaum
OIL Program Manager
Power (ppc64el) Development Project Manager
Canonical, Ltd.