== Comment: #0 - MIKHAIL S. MEDVEDEV <email address hidden> - 2017-02-03 14:40:58 ==
---Problem Description---
I am experiencing a problem with KVM on Power with a symptom of "KVM can't copy data" message being spammed to kern.log at about 100000 messages a second. Apart from original problem that triggers the error, the printk is a problem on its own. System might run out of space for logs, and would have high CPU load caused by journald.

This bug is to address an absence of rate limiter on printk at https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268

Maybe wrapping the printk in printk_ratelimit() is all that is required?

if(printk_ratelimit()){
printk("KVM can't copy");
}

Contact Information = Mikhail Medvedev / <email address hidden>, Rafael Folco / <email address hidden>

---uname output---
4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = KVM guest on P8 8247 21L

---Debugger---
A debugger is not configured

---Steps to Reproduce---
1. Deploy OpenStack with devstack
2. Run simultaneously between 4 and 6 instances of a script that
- boots a VM (openstack server create)
- immediately creates the backup of the VM (openstack server backup create)

This is not a very good way to reproduce it. I am still unable to reproduce the problem just by using libvirt and qemu.

== Comment: #5 - VIPIN K. PARASHAR <email address hidden> - 2017-02-07 09:42:31 ==

From kernel logs
===========

[852719.001593] KVM can't copy data from 0x3fff86683c80!
[852719.001614] KVM can't copy data from 0x3fff86683c80!
[852719.001617] KVM can't copy data from 0x3fff66a83c80!
[852719.001624] KVM can't copy data from 0x3fff86683c80!
[852719.001634] KVM can't copy data from 0x3fff86683c80!
[852719.001637] KVM can't copy data from 0x3fff66a83c80!
[852719.001645] KVM can't copy data from 0x3fff86683c80!
[852719.001655] KVM can't copy data from 0x3fff86683c80!
[852719.001658] KVM can't copy data from 0x3fff66a83c80!
[852719.001665] KVM can't copy data from 0x3fff86683c80!
[852719.001668] KVM can't copy data from 0x3fff66a83c80!
[852719.001678] KVM can't copy data from 0x3fff66a83c80!
[852719.001686] KVM can't copy data from 0x3fff86683c80!

"KVM can't copy data from" error is being logged at very high rate
by kernel. rate limiting this error should fix this issue.

== Comment: #20 - VIPIN K. PARASHAR <email address hidden> - 2017-02-23 12:01:26 ==
Fix for this is submitted to PowerPC mailing list
and is available at below location:

https://patchwork.ozlabs.org/patch/728831/

It reads:

"KVM: PPC: Book3S: Ratelimit copy data failure error messages"

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Patch for the Kernel Team to evaluate.

Michael

On 02/23/2017 10:11 AM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> == Comment: #0 - MIKHAIL S. MEDVEDEV <email address hidden> - 2017-02-03 14:40:58 ==
> ---Problem Description---
> I am experiencing a problem with KVM on Power with a symptom of "KVM can't copy data" message being spammed to kern.log at about 100000 messages a second. Apart from original problem that triggers the error, the printk is a problem on its own. System might run out of space for logs, and would have high CPU load caused by journald.
>
> This bug is to address an absence of rate limiter on printk at
> https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268
>
> Maybe wrapping the printk in printk_ratelimit() is all that is required?
>
>     if(printk_ratelimit()){
>        printk("KVM can't copy");
>     }
>
> Contact Information = Mikhail Medvedev / <email address hidden>, Rafael Folco / <email address hidden>
>
> ---uname output---
> 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
>
> Machine Type = KVM guest on P8 8247 21L
>
> ---Debugger---
> A debugger is not configured
>
> ---Steps to Reproduce---
>  1. Deploy OpenStack with devstack
> 2. Run simultaneously between 4 and 6 instances of a script that
>     - boots a VM (openstack server create)
>     - immediately creates the backup of the VM (openstack server backup create)
>
> This is not a very good way to reproduce it. I am still unable to
> reproduce the problem just by using libvirt and qemu.
>
>
> == Comment: #5 - VIPIN K. PARASHAR <email address hidden> - 2017-02-07 09:42:31 ==
>
> >From kernel logs
> ===========
>
> [852719.001593] KVM can't copy data from 0x3fff86683c80!
> [852719.001614] KVM can't copy data from 0x3fff86683c80!
> [852719.001617] KVM can't copy data from 0x3fff66a83c80!
> [852719.001624] KVM can't copy data from 0x3fff86683c80!
> [852719.001634] KVM can't copy data from 0x3fff86683c80!
> [852719.001637] KVM can't copy data from 0x3fff66a83c80!
> [852719.001645] KVM can't copy data from 0x3fff86683c80!
> [852719.001655] KVM can't copy data from 0x3fff86683c80!
> [852719.001658] KVM can't copy data from 0x3fff66a83c80!
> [852719.001665] KVM can't copy data from 0x3fff86683c80!
> [852719.001668] KVM can't copy data from 0x3fff66a83c80!
> [852719.001678] KVM can't copy data from 0x3fff66a83c80!
> [852719.001686] KVM can't copy data from 0x3fff86683c80!
>
>
> "KVM can't copy data from" error is being logged at very high rate
> by kernel. rate limiting this error should fix this issue.
>
>
> == Comment: #20 - VIPIN K. PARASHAR <email address hidden> - 2017-02-23 12:01:26 ==
> Fix for this is submitted to PowerPC mailing list
> and is available at below location:
>
> https://patchwork.ozlabs.org/patch/728831/
>
> It reads:
>
> "KVM: PPC: Book3S: Ratelimit copy data failure error messages"
>
> ** Affects: ubuntu
>      Importance: Undecided
>      Assignee: Taco Screen team (taco-screen-team)
>          Status: New
>
>
> ** Tags: architecture-ppc64le bugnameltc-151239 severity-medium targetmilestone-inin1704

--
Michael Hohnbaum
OIL Program Manager
Power (ppc64el) Development Project Manager
Canonical, Ltd.

Please include this for Ubuntu 16.10, 16.04.02 along with Ubuntu 17.04 release.
== Comment: #0 - MIKHAIL S. MEDVEDEV <email address hidden> - 2017-02-03 14:40:58 ==
---Problem Description---
I am experiencing a problem with KVM on Power with a symptom of "KVM can't copy data" message being spammed to kern.log at about 100000 messages a second. Apart from original problem that triggers the error, the printk is a problem on its own. System might run out of space for logs, and would have high CPU load caused by journald.

This bug is to address an absence of rate limiter on printk at https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268

Maybe wrapping the printk in printk_ratelimit() is all that is required?

if(printk_ratelimit()){
printk("KVM can't copy");
}

Contact Information = Mikhail Medvedev / <email address hidden>, Rafael Folco / <email address hidden>

---uname output---
4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = KVM guest on P8 8247 21L

---Debugger---
A debugger is not configured

---Steps to Reproduce---
1. Deploy OpenStack with devstack
2. Run simultaneously between 4 and 6 instances of a script that
- boots a VM (openstack server create)
- immediately creates the backup of the VM (openstack server backup create)

This is not a very good way to reproduce it. I am still unable to reproduce the problem just by using libvirt and qemu.

== Comment: #5 - VIPIN K. PARASHAR <email address hidden> - 2017-02-07 09:42:31 ==

From kernel logs
===========

[852719.001593] KVM can't copy data from 0x3fff86683c80!
[852719.001614] KVM can't copy data from 0x3fff86683c80!
[852719.001617] KVM can't copy data from 0x3fff66a83c80!
[852719.001624] KVM can't copy data from 0x3fff86683c80!
[852719.001634] KVM can't copy data from 0x3fff86683c80!
[852719.001637] KVM can't copy data from 0x3fff66a83c80!
[852719.001645] KVM can't copy data from 0x3fff86683c80!
[852719.001655] KVM can't copy data from 0x3fff86683c80!
[852719.001658] KVM can't copy data from 0x3fff66a83c80!
[852719.001665] KVM can't copy data from 0x3fff86683c80!
[852719.001668] KVM can't copy data from 0x3fff66a83c80!
[852719.001678] KVM can't copy data from 0x3fff66a83c80!
[852719.001686] KVM can't copy data from 0x3fff86683c80!

"KVM can't copy data from" error is being logged at very high rate
by kernel. rate limiting this error should fix this issue.

== Comment: #20 - VIPIN K. PARASHAR <email address hidden> - 2017-02-23 12:01:26 ==
Fix for this is submitted to PowerPC mailing list
and is available at below location:

https://patchwork.ozlabs.org/patch/728831/

It reads:

"KVM: PPC: Book3S: Ratelimit copy data failure error messages"

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Patch for the Kernel Team to evaluate.

Michael

On 02/23/2017 10:11 AM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> == Comment: #0 - MIKHAIL S. MEDVEDEV <email address hidden> - 2017-02-03 14:40:58 ==
> ---Problem Description---
> I am experiencing a problem with KVM on Power with a symptom of "KVM can't copy data" message being spammed to kern.log at about 100000 messages a second. Apart from original problem that triggers the error, the printk is a problem on its own. System might run out of space for logs, and would have high CPU load caused by journald.
>
> This bug is to address an absence of rate limiter on printk at
> https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268
>
> Maybe wrapping the printk in printk_ratelimit() is all that is required?
>
>     if(printk_ratelimit()){
>        printk("KVM can't copy");
>     }
>
> Contact Information = Mikhail Medvedev / <email address hidden>, Rafael Folco / <email address hidden>
>
> ---uname output---
> 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
>
> Machine Type = KVM guest on P8 8247 21L
>
> ---Debugger---
> A debugger is not configured
>
> ---Steps to Reproduce---
>  1. Deploy OpenStack with devstack
> 2. Run simultaneously between 4 and 6 instances of a script that
>     - boots a VM (openstack server create)
>     - immediately creates the backup of the VM (openstack server backup create)
>
> This is not a very good way to reproduce it. I am still unable to
> reproduce the problem just by using libvirt and qemu.
>
>
> == Comment: #5 - VIPIN K. PARASHAR <email address hidden> - 2017-02-07 09:42:31 ==
>
> >From kernel logs
> ===========
>
> [852719.001593] KVM can't copy data from 0x3fff86683c80!
> [852719.001614] KVM can't copy data from 0x3fff86683c80!
> [852719.001617] KVM can't copy data from 0x3fff66a83c80!
> [852719.001624] KVM can't copy data from 0x3fff86683c80!
> [852719.001634] KVM can't copy data from 0x3fff86683c80!
> [852719.001637] KVM can't copy data from 0x3fff66a83c80!
> [852719.001645] KVM can't copy data from 0x3fff86683c80!
> [852719.001655] KVM can't copy data from 0x3fff86683c80!
> [852719.001658] KVM can't copy data from 0x3fff66a83c80!
> [852719.001665] KVM can't copy data from 0x3fff86683c80!
> [852719.001668] KVM can't copy data from 0x3fff66a83c80!
> [852719.001678] KVM can't copy data from 0x3fff66a83c80!
> [852719.001686] KVM can't copy data from 0x3fff86683c80!
>
>
> "KVM can't copy data from" error is being logged at very high rate
> by kernel. rate limiting this error should fix this issue.
>
>
> == Comment: #20 - VIPIN K. PARASHAR <email address hidden> - 2017-02-23 12:01:26 ==
> Fix for this is submitted to PowerPC mailing list
> and is available at below location:
>
> https://patchwork.ozlabs.org/patch/728831/
>
> It reads:
>
> "KVM: PPC: Book3S: Ratelimit copy data failure error messages"
>
> ** Affects: ubuntu
>      Importance: Undecided
>      Assignee: Taco Screen team (taco-screen-team)
>          Status: New
>
>
> ** Tags: architecture-ppc64le bugnameltc-151239 severity-medium targetmilestone-inin1704

--
Michael Hohnbaum
OIL Program Manager
Power (ppc64el) Development Project Manager
Canonical, Ltd.

Please include this for Ubuntu 16.10, 16.04.02 along with Ubuntu 17.04 release.