| 2015-04-30 18:19:20 |
Taloth Saldono |
bug |
|
|
added bug |
| 2015-04-30 18:30:08 |
Brad Figg |
linux (Ubuntu): status |
New |
Confirmed |
|
| 2015-04-30 19:20:11 |
Taloth Saldono |
summary |
mono occassionally crashes since kernel 3.13.0-46 on multi-cpu vm |
mono occassionally crashes since kernel 3.13.0-48 on multi-cpu vm |
|
| 2015-05-01 08:12:41 |
penalvch |
tags |
amd64 apport-bug trusty |
amd64 apport-bug bisect-done regression-update reverse-bisect-done trusty |
|
| 2015-05-01 08:13:21 |
penalvch |
tags |
amd64 apport-bug bisect-done regression-update reverse-bisect-done trusty |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty |
|
| 2015-05-01 08:13:43 |
penalvch |
linux (Ubuntu): importance |
Undecided |
Medium |
|
| 2015-05-01 08:13:43 |
penalvch |
linux (Ubuntu): status |
Confirmed |
Triaged |
|
| 2015-05-04 21:15:22 |
Chris J Arges |
nominated for series |
|
Ubuntu Trusty |
|
| 2015-05-04 21:15:22 |
Chris J Arges |
bug task added |
|
linux (Ubuntu Trusty) |
|
| 2015-05-04 21:15:22 |
Chris J Arges |
nominated for series |
|
Ubuntu Vivid |
|
| 2015-05-04 21:15:22 |
Chris J Arges |
bug task added |
|
linux (Ubuntu Vivid) |
|
| 2015-05-04 21:15:22 |
Chris J Arges |
nominated for series |
|
Ubuntu Utopic |
|
| 2015-05-04 21:15:22 |
Chris J Arges |
bug task added |
|
linux (Ubuntu Utopic) |
|
| 2015-05-04 21:17:57 |
Chris J Arges |
linux (Ubuntu Utopic): importance |
Undecided |
Medium |
|
| 2015-05-04 21:17:58 |
Chris J Arges |
linux (Ubuntu Trusty): importance |
Undecided |
Medium |
|
| 2015-05-04 21:18:03 |
Chris J Arges |
linux (Ubuntu Utopic): status |
New |
In Progress |
|
| 2015-05-04 21:18:05 |
Chris J Arges |
linux (Ubuntu Vivid): status |
Triaged |
In Progress |
|
| 2015-05-04 21:18:08 |
Chris J Arges |
linux (Ubuntu Trusty): assignee |
|
Chris J Arges (arges) |
|
| 2015-05-04 21:18:10 |
Chris J Arges |
linux (Ubuntu Utopic): assignee |
|
Chris J Arges (arges) |
|
| 2015-05-04 21:18:13 |
Chris J Arges |
linux (Ubuntu Vivid): assignee |
|
Chris J Arges (arges) |
|
| 2015-05-04 21:18:16 |
Chris J Arges |
linux (Ubuntu Trusty): status |
New |
In Progress |
|
| 2015-05-05 15:29:10 |
Chris J Arges |
description |
Gradually since late March more and more users started to complain about frequent SIGSEGV crashes in our .net/mono application. Early April I started to investigate it actively.
After eliminating possible native libraries, and testing various mono versions I discovered the crashes would occur more frequently on a vbox vm with multiple cpus configured. And discovered that the mono bug-18026.cs testcase would fairly consistently crash. At that point it was reported to the mono bug tracker.
I finally got a break when we found a correlation with the kernel version. 3.13.0-46 didn't crash while 3.13.0-48,49 did.
More and more users upgrade to these newer kernel versions and start running into issues, which explains the gradual increase in reports.
Early this week I performed a full git bisect on the kernel between 3.13.0-46 and -48 and isolated the commit that seems to trigger the crashes.
Namely http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git/commit/?id=11f4e0339c8dc8d760483258efd9f15b4c6dcda2
At this point I don't know if the commit messed up something, or that mono simply handles it incorrectly. However, a few commits for linux 4.x seem to fix it:
https://github.com/torvalds/linux/commit/80f7fdb1c7f0f9266421f823964fd1962681f6ce
https://github.com/torvalds/linux/commit/0a4e6be9ca17c54817cf814b4b5aa60478c6df27
I applied these commits myself on top of commit 11f4e033, compiled and ran the testcase... didn't crash in the 200x test runs I did.
Although I don't know if those two patches have unknown side-effects.
I'm not an expert on the kernel, not even remotely. But I thought it would be nice to be able to point at a possible solution.
My current test vm is a virtualbox vm 64bit installed using the 14.04.2 server iso running on an older i7 quad core Windows 7 64bit host.
In the vm I've tested numerous mono and kernel combinations. Last test was with kernel 3.16.0-36 and 3.13.0-51 and mono 4.0.1, in which the problem still occurs.
By now I've debugged the app using gdb several dozen times on various user setups, compiled mono half a dozen times, and then the 8x3h compile kernel bisect :) Speaking of down the rabbit-hole...
So I'm pretty desperate for some expert to help me out here. :D
Reference to mono bug report: https://bugzilla.xamarin.com/show_bug.cgi?id=29212
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-51-generic 3.13.0-51.84
ProcVersionSignature: Ubuntu 3.13.0-51.84-generic 3.13.11-ckt18
Uname: Linux 3.13.0-51-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Apr 30 18:53 seq
crw-rw---- 1 root audio 116, 33 Apr 30 18:53 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory: 'iw'
CurrentDmesg: [ 9.379188] init: plymouth-upstart-bridge main process ended, respawning
Date: Thu Apr 30 19:45:43 2015
HibernationDevice: RESUME=UUID=b35ef328-166d-4476-a418-e7e80d22cb30
InstallationDate: Installed on 2015-04-22 (7 days ago)
InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
IwConfig:
eth0 no wireless extensions.
lo no wireless extensions.
Lsusb:
Bus 001 Device 002: ID 80ee:0021 VirtualBox USB Tablet
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: innotek GmbH VirtualBox
ProcEnviron:
TERM=screen
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-51-generic root=UUID=68da7e09-1a91-4107-859d-bf452f9ed992 ro
RelatedPackageVersions:
linux-restricted-modules-3.13.0-51-generic N/A
linux-backports-modules-3.13.0-51-generic N/A
linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/01/2006
dmi.bios.vendor: innotek GmbH
dmi.bios.version: VirtualBox
dmi.board.name: VirtualBox
dmi.board.vendor: Oracle Corporation
dmi.board.version: 1.2
dmi.chassis.type: 1
dmi.chassis.vendor: Oracle Corporation
dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr:
dmi.product.name: VirtualBox
dmi.product.version: 1.2
dmi.sys.vendor: innotek GmbH |
[Impact]
The addition of the commit:
http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git/commit/?id=11f4e0339c8dc8d760483258efd9f15b4c6dcda2
Causes SIGSEGVs when running certain workloads on multi-cpu VMs.
[Test Case]
Mono test case here that causes the SIGSEGV
https://bugzilla.xamarin.com/show_bug.cgi?id=29212
[Fix]
These two commits are required for fixing this issue:
https://github.com/torvalds/linux/commit/80f7fdb1c7f0f9266421f823964fd1962681f6ce
https://github.com/torvalds/linux/commit/0a4e6be9ca17c54817cf814b4b5aa60478c6df27
--
Gradually since late March more and more users started to complain about frequent SIGSEGV crashes in our .net/mono application. Early April I started to investigate it actively.
After eliminating possible native libraries, and testing various mono versions I discovered the crashes would occur more frequently on a vbox vm with multiple cpus configured. And discovered that the mono bug-18026.cs testcase would fairly consistently crash. At that point it was reported to the mono bug tracker.
I finally got a break when we found a correlation with the kernel version. 3.13.0-46 didn't crash while 3.13.0-48,49 did.
More and more users upgrade to these newer kernel versions and start running into issues, which explains the gradual increase in reports.
Early this week I performed a full git bisect on the kernel between 3.13.0-46 and -48 and isolated the commit that seems to trigger the crashes.
Namely http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git/commit/?id=11f4e0339c8dc8d760483258efd9f15b4c6dcda2
At this point I don't know if the commit messed up something, or that mono simply handles it incorrectly. However, a few commits for linux 4.x seem to fix it:
https://github.com/torvalds/linux/commit/80f7fdb1c7f0f9266421f823964fd1962681f6ce
https://github.com/torvalds/linux/commit/0a4e6be9ca17c54817cf814b4b5aa60478c6df27
I applied these commits myself on top of commit 11f4e033, compiled and ran the testcase... didn't crash in the 200x test runs I did.
Although I don't know if those two patches have unknown side-effects.
I'm not an expert on the kernel, not even remotely. But I thought it would be nice to be able to point at a possible solution.
My current test vm is a virtualbox vm 64bit installed using the 14.04.2 server iso running on an older i7 quad core Windows 7 64bit host.
In the vm I've tested numerous mono and kernel combinations. Last test was with kernel 3.16.0-36 and 3.13.0-51 and mono 4.0.1, in which the problem still occurs.
By now I've debugged the app using gdb several dozen times on various user setups, compiled mono half a dozen times, and then the 8x3h compile kernel bisect :) Speaking of down the rabbit-hole...
So I'm pretty desperate for some expert to help me out here. :D
Reference to mono bug report: https://bugzilla.xamarin.com/show_bug.cgi?id=29212
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-51-generic 3.13.0-51.84
ProcVersionSignature: Ubuntu 3.13.0-51.84-generic 3.13.11-ckt18
Uname: Linux 3.13.0-51-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Apr 30 18:53 seq
crw-rw---- 1 root audio 116, 33 Apr 30 18:53 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory: 'iw'
CurrentDmesg: [ 9.379188] init: plymouth-upstart-bridge main process ended, respawning
Date: Thu Apr 30 19:45:43 2015
HibernationDevice: RESUME=UUID=b35ef328-166d-4476-a418-e7e80d22cb30
InstallationDate: Installed on 2015-04-22 (7 days ago)
InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
IwConfig:
eth0 no wireless extensions.
lo no wireless extensions.
Lsusb:
Bus 001 Device 002: ID 80ee:0021 VirtualBox USB Tablet
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: innotek GmbH VirtualBox
ProcEnviron:
TERM=screen
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-51-generic root=UUID=68da7e09-1a91-4107-859d-bf452f9ed992 ro
RelatedPackageVersions:
linux-restricted-modules-3.13.0-51-generic N/A
linux-backports-modules-3.13.0-51-generic N/A
linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/01/2006
dmi.bios.vendor: innotek GmbH
dmi.bios.version: VirtualBox
dmi.board.name: VirtualBox
dmi.board.vendor: Oracle Corporation
dmi.board.version: 1.2
dmi.chassis.type: 1
dmi.chassis.vendor: Oracle Corporation
dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr:
dmi.product.name: VirtualBox
dmi.product.version: 1.2
dmi.sys.vendor: innotek GmbH |
|
| 2015-05-08 17:49:15 |
Brad Figg |
linux (Ubuntu Vivid): status |
In Progress |
Fix Committed |
|
| 2015-05-08 17:49:19 |
Brad Figg |
linux (Ubuntu Utopic): status |
In Progress |
Fix Committed |
|
| 2015-05-08 17:49:22 |
Brad Figg |
linux (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2015-05-18 18:29:30 |
Badr CHOUFFAI |
linux (Ubuntu): status |
In Progress |
Fix Released |
|
| 2015-05-18 23:11:39 |
penalvch |
linux (Ubuntu): status |
Fix Released |
In Progress |
|
| 2015-05-29 12:04:47 |
BrandonG777 |
bug |
|
|
added subscriber BrandonG777 |
| 2015-05-31 19:38:52 |
Launchpad Janitor |
linux (Ubuntu): status |
In Progress |
Fix Released |
|
| 2015-05-31 19:38:52 |
Launchpad Janitor |
cve linked |
|
2015-0275 |
|
| 2015-05-31 19:38:52 |
Launchpad Janitor |
cve linked |
|
2015-4036 |
|
| 2015-06-01 09:29:07 |
Luis Henriques |
tags |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty verification-needed-trusty |
|
| 2015-06-01 09:32:28 |
Luis Henriques |
tags |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty verification-needed-trusty |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty verification-needed-trusty verification-needed-utopic |
|
| 2015-06-01 09:34:45 |
Luis Henriques |
tags |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty verification-needed-trusty verification-needed-utopic |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty verification-needed-trusty verification-needed-utopic verification-needed-vivid |
|
| 2015-06-01 17:05:16 |
Taloth Saldono |
tags |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty verification-needed-trusty verification-needed-utopic verification-needed-vivid |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty verification-done-trusty verification-done-utopic verification-needed-vivid |
|
| 2015-06-04 16:21:18 |
Brad Figg |
tags |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty verification-done-trusty verification-done-utopic verification-needed-vivid |
amd64 apport-bug bisect-done cherry-pick regression-update reverse-bisect-done trusty verification-done-trusty verification-done-utopic verification-done-vivid |
|
| 2015-06-08 10:50:18 |
pav |
bug |
|
|
added subscriber pav |
| 2015-06-10 15:22:45 |
Launchpad Janitor |
linux (Ubuntu Vivid): status |
Fix Committed |
Fix Released |
|
| 2015-06-10 15:25:02 |
Launchpad Janitor |
linux (Ubuntu Utopic): status |
Fix Committed |
Fix Released |
|
| 2015-06-10 15:26:42 |
Launchpad Janitor |
linux (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
