BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid

[Rahul]

Two days back my KVM base machine got hung up all of a sudden.
Not sure what exactly happened.

cat /proc/version_signature
Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12

-Rahul N.

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug.

To give us some more information on your system, could you please run 'apport-collect 977391' ?

[Rahul (rahul-nair)]

Is there any alternate way I can provide you the information?
I am actually not comfortable in executing this command on the system.

-Rahul N.

On Mon, Apr 9, 2012 at 11:59 PM, Serge Hallyn <email address hidden>wrote:

> Thanks for reporting this bug.
>
> To give us some more information on your system, could you please run
> 'apport-collect 977391' ?
>
> ** Package changed: kvm (Ubuntu) => qemu-kvm (Ubuntu)
>
> ** Changed in: qemu-kvm (Ubuntu)
> Status: New => Incomplete
>
> ** Also affects: linux (Ubuntu)
> Importance: Undecided
> Status: New
>
> ** Changed in: linux (Ubuntu)
> Status: New => Incomplete
>
> ** Changed in: qemu-kvm (Ubuntu)
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/977391
>
> Title:
> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*]
>
> Status in “linux” package in Ubuntu:
> Incomplete
> Status in “qemu-kvm” package in Ubuntu:
> Incomplete
>
> Bug description:
> Two days back my KVM base machine got hung up all of a sudden.
> Not sure what exactly happened.
>
> cat /proc/version_signature
> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>
>
> -Rahul N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/977391/+subscriptions
>

--
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india

[Serge Hallyn (serge-hallyn)]

Quoting Rahul (<email address hidden>):
> Is there any alternate way I can provide you the information?
> I am actually not comfortable in executing this command on the system.

The most important information given the nature of the bug would be the
configuration (in particular what is sitting under the backing store) of
your VM, and the relevant contents of /var/log/syslog and /var/log/dmesg*.

Are you able to reprocuce this at will?

[Rahul (rahul-nair)]

Serge,

I attaching the logs along with this mail.

-Rahul N.

On Tue, Apr 10, 2012 at 1:39 AM, Serge Hallyn <email address hidden>wrote:

> Quoting Rahul (<email address hidden>):
> > Is there any alternate way I can provide you the information?
> > I am actually not comfortable in executing this command on the system.
>
> The most important information given the nature of the bug would be the
> configuration (in particular what is sitting under the backing store) of
> your VM, and the relevant contents of /var/log/syslog and /var/log/dmesg*.
>
> Are you able to reprocuce this at will?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/977391
>
> Title:
> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*]
>
> Status in QEMU:
> New
> Status in “linux” package in Ubuntu:
> Confirmed
> Status in “qemu-kvm” package in Ubuntu:
> Confirmed
>
> Bug description:
> Two days back my KVM base machine got hung up all of a sudden.
> Not sure what exactly happened.
>
> cat /proc/version_signature
> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>
>
> -Rahul N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/977391/+subscriptions
>

[Serge Hallyn (serge-hallyn)]

Thanks - unfortunately they seem to be from after the event.

If it happens again, please do post the new logs.

And if you know how to trigger this at will, please let us know.

[penalvch (penalvch)]

rahul, thank you for reporting this and helping make Ubuntu better. Could you please capture the oops following https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Capturing_OOPs ?

[Rahul (rahul-nair)]

Serge,

These are the logs at the time when event occurred.
But unfortunately nothing helpful was there in the logs.
I am not sure about how can I provide you the proper logs even if this
happened again.

-Rahul N.

On Tue, Apr 10, 2012 at 3:23 AM, Serge Hallyn <email address hidden>wrote:

> Thanks - unfortunately they seem to be from after the event.
>
> If it happens again, please do post the new logs.
>
> And if you know how to trigger this at will, please let us know.
>
> ** Summary changed:
>
> - BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*]
> + BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/977391
>
> Title:
> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>
> Status in QEMU:
> Confirmed
> Status in “linux” package in Ubuntu:
> Confirmed
> Status in “qemu-kvm” package in Ubuntu:
> Confirmed
>
> Bug description:
> Two days back my KVM base machine got hung up all of a sudden.
> Not sure what exactly happened.
>
> cat /proc/version_signature
> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>
>
> -Rahul N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/977391/+subscriptions
>

--
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india

[Rahul (rahul-nair)]

Please check the attached screenshot that I took during the issue.
I cant find anything useful in the logs.

-Rahul N.

On Tue, Apr 10, 2012 at 3:31 AM, Christopher M. Penalver <
<email address hidden>> wrote:

> rahul, thank you for reporting this and helping make Ubuntu better.
> Could you please capture the oops following
> https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Capturing_OOPs
> ?
>
> ** Tags added: lucid
>
> ** Changed in: linux (Ubuntu)
> Status: Confirmed => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/977391
>
> Title:
> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>
> Status in QEMU:
> Confirmed
> Status in “linux” package in Ubuntu:
> Incomplete
> Status in “qemu-kvm” package in Ubuntu:
> Confirmed
>
> Bug description:
> Two days back my KVM base machine got hung up all of a sudden.
> Not sure what exactly happened.
>
> cat /proc/version_signature
> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>
>
> -Rahul N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/977391/+subscriptions
>

--
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india

[Rahul (rahul-nair)]

Serge,

May be this screenshot (attached) will help.

-Rahul N.

On Tue, Apr 10, 2012 at 4:37 AM, Rahul Nair <email address hidden> wrote:

> Serge,
>
> These are the logs at the time when event occurred.
> But unfortunately nothing helpful was there in the logs.
> I am not sure about how can I provide you the proper logs even if this
> happened again.
>
> -Rahul N.
>
> On Tue, Apr 10, 2012 at 3:23 AM, Serge Hallyn <email address hidden>wrote:
>
>> Thanks - unfortunately they seem to be from after the event.
>>
>> If it happens again, please do post the new logs.
>>
>> And if you know how to trigger this at will, please let us know.
>>
>> ** Summary changed:
>>
>> - BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*]
>> + BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/977391
>>
>> Title:
>> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>>
>> Status in QEMU:
>> Confirmed
>> Status in “linux” package in Ubuntu:
>> Confirmed
>> Status in “qemu-kvm” package in Ubuntu:
>> Confirmed
>>
>> Bug description:
>> Two days back my KVM base machine got hung up all of a sudden.
>> Not sure what exactly happened.
>>
>> cat /proc/version_signature
>> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>>
>>
>> -Rahul N.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/qemu/+bug/977391/+subscriptions
>>
>
>
>
> --
> -Rahul N.
> IT Department
> In2M Technologies Pvt Ltd. (Finicity)
> Website: www.finicity.com/india
>
>

--
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india

[Rahul (rahul-nair)]

Were you able to find any clues...?

Rahul N.
On Tuesday, April 10, 2012, Rahul Nair <email address hidden> wrote:
> Please check the attached screenshot that I took during the issue.
> I cant find anything useful in the logs.
>
> -Rahul N.
>
> On Tue, Apr 10, 2012 at 3:31 AM, Christopher M. Penalver <
<email address hidden>> wrote:
>>
>> rahul, thank you for reporting this and helping make Ubuntu better.
>> Could you please capture the oops following
>> https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Capturing_OOPs
>> ?
>>
>> ** Tags added: lucid
>>
>> ** Changed in: linux (Ubuntu)
>> Status: Confirmed => Incomplete
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/977391
>>
>> Title:
>> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>>
>> Status in QEMU:
>> Confirmed
>> Status in “linux” package in Ubuntu:
>> Incomplete
>> Status in “qemu-kvm” package in Ubuntu:
>> Confirmed
>>
>> Bug description:
>> Two days back my KVM base machine got hung up all of a sudden.
>> Not sure what exactly happened.
>>
>> cat /proc/version_signature
>> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>>
>>
>> -Rahul N.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/qemu/+bug/977391/+subscriptions
>
>
>
> --
> -Rahul N.
> IT Department
> In2M Technologies Pvt Ltd. (Finicity)
> Website: www.finicity.com/india
>

--
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india

[Serge Hallyn (serge-hallyn)]

Sorry, the screenshot does show that the deadlock is on a spinlock during flush_tlb_others(), perhaps at

        if (nr_cpu_ids > NUM_INVALIDATE_TLB_VECTORS)
                raw_spin_lock(&f->tlbstate_lock);

[Serge Hallyn (serge-hallyn)]

Uh, I meant "sorry, but the reason that happened isn't shown".

[Serge Hallyn (serge-hallyn)]

Is there anything in /var/log/kern.log.* ? A fuller oops would be valuable.

If this vm was spawned with libvirt, could you post the xml description? What OS was the guest running?

[Rahul (rahul-nair)]

Serge,

This is a KVM base machine which faced the issue not a guest machine.
In kern.log there was no Call Trace found during the time of issue.
Also nothing was found in /var/crash/* too.
Are you aware of any possible reason due to which its not logged in any of
the log files?

-Rahul N.

On Tue, Apr 10, 2012 at 8:35 PM, Serge Hallyn <email address hidden>wrote:

> Is there anything in /var/log/kern.log.* ? A fuller oops would be
> valuable.
>
> If this vm was spawned with libvirt, could you post the xml description?
> What OS was the guest running?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/977391
>
> Title:
> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>
> Status in QEMU:
> Confirmed
> Status in “linux” package in Ubuntu:
> Confirmed
> Status in “qemu-kvm” package in Ubuntu:
> Confirmed
>
> Bug description:
> Two days back my KVM base machine got hung up all of a sudden.
> Not sure what exactly happened.
>
> cat /proc/version_signature
> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>
>
> -Rahul N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/977391/+subscriptions
>

--
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india

[Rahul (rahul-nair)]

Also FYI:

OS: Ubuntu 10.04
kernel: 2.6.32-28

-Rahul N.

On Wed, Apr 11, 2012 at 4:49 PM, Rahul Nair <email address hidden> wrote:

> Serge,
>
> This is a KVM base machine which faced the issue not a guest machine.
> In kern.log there was no Call Trace found during the time of issue.
> Also nothing was found in /var/crash/* too.
> Are you aware of any possible reason due to which its not logged in any of
> the log files?
>
> -Rahul N.
>
> On Tue, Apr 10, 2012 at 8:35 PM, Serge Hallyn <email address hidden>wrote:
>
>> Is there anything in /var/log/kern.log.* ? A fuller oops would be
>> valuable.
>>
>> If this vm was spawned with libvirt, could you post the xml description?
>> What OS was the guest running?
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/977391
>>
>> Title:
>> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>>
>> Status in QEMU:
>> Confirmed
>> Status in “linux” package in Ubuntu:
>> Confirmed
>> Status in “qemu-kvm” package in Ubuntu:
>> Confirmed
>>
>> Bug description:
>> Two days back my KVM base machine got hung up all of a sudden.
>> Not sure what exactly happened.
>>
>> cat /proc/version_signature
>> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>>
>>
>> -Rahul N.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/qemu/+bug/977391/+subscriptions
>>
>
>
>
> --
> -Rahul N.
> IT Department
> In2M Technologies Pvt Ltd. (Finicity)
> Website: www.finicity.com/india
>
>

--
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india

[Serge Hallyn (serge-hallyn)]

Quoting Rahul (<email address hidden>):
> Serge,
>
> This is a KVM base machine which faced the issue not a guest machine.

Yes, but it's possible that using the same guest OS will make reproducing
the bug possible. Since there isn't enough collected debug info, I'll
need to be able to reproduce it.

> In kern.log there was no Call Trace found during the time of issue.

Thanks.

> Also nothing was found in /var/crash/* too.

Drat.

> Are you aware of any possible reason due to which its not logged in any of
> the log files?

No.

[arun gathiya (arun-asg2005)]

found that there is patch available for this bug but the patch is for Ubuntu 8.04.4 LTS(2.6.24-26). I am looking if the same patch is available for Ubuntu 10.04.2 LTS. Here is the link describing about the bug and its available patch. please advise.
https://bugs.launchpad.net/ubuntu/hardy/+source/linux/+bug/790557

[Rahul (rahul-nair)]

Plz co-ordinate with launchpad via bug we have created.
I need you to focus on this and get root cause by next week as per
postmortem.

-Rahul N.

On Saturday, April 14, 2012, arun gathiya <email address hidden> wrote:
> found that there is patch available for this bug but the patch is for
Ubuntu 8.04.4 LTS(2.6.24-26). I am looking if the same patch is available
for Ubuntu 10.04.2 LTS. Here is the link describing about the bug and its
available patch. please advise.
> https://bugs.launchpad.net/ubuntu/hardy/+source/linux/+bug/790557
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/977391
>
> Title:
> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>
> Status in QEMU:
> Confirmed
> Status in “linux” package in Ubuntu:
> Confirmed
> Status in “qemu-kvm” package in Ubuntu:
> Confirmed
>
> Bug description:
> Two days back my KVM base machine got hung up all of a sudden.
> Not sure what exactly happened.
>
> cat /proc/version_signature
> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>
>
> -Rahul N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/977391/+subscriptions
>

--
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india

[Rahul (rahul-nair)]

Plz co-ordinate with launchpad via bug we have created.
I need you to focus on this and get root cause by next week as per
postmortem.

-Rahul

On Saturday, April 14, 2012, arun gathiya <email address hidden> wrote:
> found that there is patch available for this bug but the patch is for
Ubuntu 8.04.4 LTS(2.6.24-26). I am looking if the same patch is available
for Ubuntu 10.04.2 LTS. Here is the link describing about the bug and its
available patch. please advise.
> https://bugs.launchpad.net/ubuntu/hardy/+source/linux/+bug/790557
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/977391
>
> Title:
> BUG: soft lockup - CPU#8 stuck for 61s! [kvm:*] in lucid
>
> Status in QEMU:
> Confirmed
> Status in “linux” package in Ubuntu:
> Confirmed
> Status in “qemu-kvm” package in Ubuntu:
> Confirmed
>
> Bug description:
> Two days back my KVM base machine got hung up all of a sudden.
> Not sure what exactly happened.
>
> cat /proc/version_signature
> Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12
>
>
> -Rahul N.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/977391/+subscriptions
>

--
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india

[Serge Hallyn (serge-hallyn)]

Thanks. The bug you linked was fixed upstream as of 2.6.25.

Can you post some information on your storage backend?

[arun gathiya (arun-asg2005)]

You mean there is patch available for 10.04.2? OR same below patch is applicable for Ubuntu 10.04.2 LTS ?
which specific information you want on storage backend? sorry I did't get you there.
:
found that there is patch available for this bug but the patch is for Ubuntu 8.04.4 LTS(2.6.24-26). I am looking if the same patch is available for Ubuntu 10.04.2 LTS. Here is the link describing about the bug and its available patch. please advise.
https://bugs.launchpad.net/ubuntu/hardy/+source/linux/+bug/790557

[penalvch (penalvch)]

Rahul, this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? If so, could you please test for this with the latest server release of Ubuntu? ISO images are available from http://releases.ubuntu.com/raring/ .

If it remains an issue, could you please run the following command in the development release from a Terminal (Applications->Accessories->Terminal), as it will automatically gather and attach updated debug information to this report:

apport-collect -p linux <replace-with-bug-number>

Also, could you please test the latest upstream kernel available following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Please do not test the daily folder, but the one all the way at the bottom. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested. For example:
kernel-fixed-upstream-v3.11-rc7

This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description. As well, please remove the tag:
needs-upstream-testing

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

As well, please remove the tag:
needs-upstream-testing

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding.

[Thomas Huth (th-huth)]

This sounds like a kernel bug, so it should not be tracked via the QEMU bug tracker.

[Robie Basak (racb)]

Setting Incomplete for qemu-kvm as well, pending the reporter's response on whether this is still an issue.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for qemu-kvm (Ubuntu) because there has been no activity for 60 days.]