[Alienware M15x] Keyboard becomes all but unusable after Precise runs for so long

[Jeff Lane ]

At random lenghts of runtime, after booting Precise on my Alienware M15x, the keyboard goes completely wonky.

The symptoms for this are varied. Usually, they involve what seems like a hung keypress, so, for example, if I press Alt-Tab to switch windows, and release the Tab, as long as I have Alt pressed the switcher rapidly scrolls through open windows unstopped.

Or, if I press and hold the space bar, the system continuously spits out spaces. OR, if I scroll a web page using the up or down arrows (or try navigating large files using less in a terminal window) the arrow key appears to become stuck until I press a different arrow key.

Another symptom is that keys stop responding on the first press. I may have to type each key twice to get the letter to appear in whatever I am typing in.

Logwise, I consistently see atkbd error messages flooding syslog like this:

Apr 17 08:17:02 klaatu CRON[20974]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 17 09:17:01 klaatu CRON[21071]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 17 10:17:01 klaatu CRON[22610]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 17 10:55:33 klaatu kernel: [84813.139562] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Apr 17 10:55:33 klaatu kernel: [84813.139571] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
Apr 17 10:55:58 klaatu kernel: [84838.402852] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Apr 17 10:55:58 klaatu kernel: [84838.402861] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
Apr 17 11:02:50 klaatu kernel: [85249.681077] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Apr 17 11:02:50 klaatu kernel: [85249.681086] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
Apr 17 11:15:27 klaatu kernel: [86005.742967] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Apr 17 11:15:27 klaatu kernel: [86005.742976] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
Apr 17 11:17:01 klaatu CRON[23088]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 17 11:39:18 klaatu kernel: [87435.419939] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio2/input/input19
Apr 17 11:39:18 klaatu kernel: [87435.432907] atkbd serio2: Unknown key pressed (raw set 2, code 0x18 on isa0060/serio2).
Apr 17 11:39:18 klaatu kernel: [87435.432915] atkbd serio2: Use 'setkeycodes 18 <keycode>' to make it known.
Apr 17 11:39:18 klaatu kernel: [87435.437285] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/serio2).
Apr 17 11:39:18 klaatu kernel: [87435.437293] atkbd serio2: Use 'setkeycodes 00 <keycode>' to make it known.
Apr 17 11:39:18 klaatu kernel: [87435.456454] atkbd serio2: Unknown key pressed (raw set 2, code 0x8 on isa0060/serio2).
Apr 17 11:39:18 klaatu kernel: [87435.456461] atkbd serio2: Use 'setkeycodes 08 <keycode>' to make it known.
Apr 17 11:39:18 klaatu kernel: [87435.458832] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/serio2).
Apr 17 11:39:18 klaatu kernel: [87435.458840] atkbd serio2: Use 'setkeycodes 00 <keycode>' to make it known.
Apr 17 11:39:18 klaatu kernel: [87435.481099] atkbd serio2: Unknown key pressed (raw set 2, code 0x8 on isa0060/serio2).
Apr 17 11:39:18 klaatu kernel: [87435.481108] atkbd serio2: Use 'setkeycodes 08 <keycode>' to make it known.
Apr 17 11:39:18 klaatu kernel: [87435.483830] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/serio2).
Apr 17 11:39:18 klaatu kernel: [87435.483836] atkbd serio2: Use 'setkeycodes 00 <keycode>' to make it known.
Apr 17 11:39:18 klaatu kernel: [87435.509882] atkbd serio2: Unknown key pressed (raw set 2, code 0x18 on isa0060/serio2).
Apr 17 11:39:18 klaatu kernel: [87435.509891] atkbd serio2: Use 'setkeycodes 18 <keycode>' to make it known.
Apr 17 11:39:18 klaatu kernel: [87435.511006] atkbd serio2: Unknown key pressed (raw set 2, code 0x17c on isa0060/serio2).
Apr 17 11:39:18 klaatu kernel: [87435.511013] atkbd serio2: Use 'setkeycodes 7c <keycode>' to make it known.
Apr 17 11:39:18 klaatu kernel: [87435.512055] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/serio2).
Apr 17 11:39:18 klaatu kernel: [87435.512059] atkbd serio2: Use 'setkeycodes 00 <keycode>' to make it known.

At this point, the only thing I can do is reboot the system to recover. Needless to say, this makes Precise on my system a very frustrating experience.

Unfortunately, I can use apport to add logs because of this bug: #984256 which makes apport all but unusable.
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu7
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: STAC92xx Analog [STAC92xx Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: bladernr 2170 F.... pulseaudio
 /dev/snd/controlC0: bladernr 2170 F.... pulseaudio
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xf0f20000 irq 47'
   Mixer name : 'IDT 92HD83C1X5'
   Components : 'HDA:111d7604,102802a2,00100104'
   Controls : 19
   Simple ctrls : 10
Card1.Amixer.info:
 Card hw:1 'NVidia'/'HDA NVidia at 0xcdefc000 irq 16'
   Mixer name : 'Nvidia GPU 0a HDMI/DP'
   Components : 'HDA:10de000a,10de0101,00100100'
   Controls : 24
   Simple ctrls : 4
CheckboxSubmission: b23ce36b87dcd99529399d76f1dc5b1d
CheckboxSystem: 5f30ac82cc48ed91bb5240b61cb4e295
DistroRelease: Ubuntu 12.04
HibernationDevice: RESUME=UUID=c2f4cbc3-a904-4dbf-acd4-0a28f482d683
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Alpha amd64 (20120307)
MachineType: Alienware M15x
NonfreeKernelModules: nvidia
Package: linux (not installed)
ProcEnviron:
 TERM=xterm
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-24-generic root=UUID=e7eedf5d-5aad-4230-b123-47162b1acea6 ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 3.2.0-24.37-generic 3.2.14
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-24-generic N/A
 linux-backports-modules-3.2.0-24-generic N/A
 linux-firmware 1.79
SourcePackage: linux
Tags: precise precise
Uname: Linux 3.2.0-24-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
dmi.bios.date: 08/24/2010
dmi.bios.vendor: Alienware
dmi.bios.version: A08
dmi.board.vendor: Alienware
dmi.board.version: A08
dmi.chassis.type: 8
dmi.chassis.vendor: Alienware
dmi.chassis.version: A08
dmi.modalias: dmi:bvnAlienware:bvrA08:bd08/24/2010:svnAlienware:pnM15x:pvrA08:rvnAlienware:rn:rvrA08:cvnAlienware:ct8:cvrA08:
dmi.product.name: M15x
dmi.product.version: A08
dmi.sys.vendor: Alienware

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 984265

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Do you know if this issue happened in a previous version of Ubuntu, or is this a new issue?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.4kernel[1] (Not a kernel in the daily directory). Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag(Only that one tag, please leave the other tags). This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.4-rc3-precise/

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

apport information

[Jeff Lane  (bladernr)]

I will try the mainline kernel, but over the last few days, this issue has gotten worse. Now I'm seeing instances where random characters start getting dumped to terminal (when the atkbd messages start appearing in syslog). eventually, this causes the system to simply shutdown or reboot... and not cleanly.

I'm currently running 3.2.0-24.37 and just had this happen as I was sitting here trying to work. Suddenly, my terminal was flooded wtih random characters and became unresponsive, and then the machine just crashed and powered off. :/

[Jeff Lane  (bladernr)]

Ok, I've had the mainline kernel running for almost 15 hours so far:

bladernr@klaatu:~$ uptime
 11:40:27 up 14:44, 0 users, load average: 0.98, 0.89, 0.77
bladernr@klaatu:~$ uname -a
Linux klaatu 3.4.0-999-generic #201205100405 SMP Thu May 10 08:06:15 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

And I have not experienced any of the previous keyboard wonkiness. This had started, usually within a couple hours of booting at the most and becomming progressively worse until I had to reboot to clear it up. So far, after nearly 15 hours, it's still stable.

The downside to this is that I need to run nVidia_current which doesn't build for the mainline kernel, so I get a nasty resolution on my system that makes it ... challenging ... to use. I'll keep running mainline through the weekend, however, to see if it remains stable.

[Jeff Lane  (bladernr)]

ok... after running the mainline kernel all weekend long I never once ran into an issue with either psmouse and the touchpad or the atkbd driver and the keyboard as I've seen frequently with the ubuntu kernels. The mainline kernel seemed to be rather stable as far as this bug goes.

[Jeff Lane  (bladernr)]

To recap... ran the mainline kernel from Friday afternoon to Monday morning, non-stop, and never had an issue. Rebooted into the current Precise kernel this morning, and made it about 7 hours before I had to reboot due to the keyboard going insane.

[Jeff Lane  (bladernr)]

Please bump this thread up in prioirty. This is causing me to have to reboot my machine at least once per day, and I've discovered can trigger very suprising and unpleasant things to happen.

So when atkbd goes nuts, it starts generating random keypresses. I've seen these keypresses appear in whatever window or terminal happens to have focus at the time, and they can include ANY key events, from alpha-numeric to F-Keys to, as I've discovered today, SysRq combos.

Today, my computer appeared to crash. I had opened a bug for that before: bug #984273 where the syslog was suddenly flooded with Emergency Thaw messages, triggering a buffer overflow on klogd, causing malformed messages written to the log, memory filling up until there was none left and eventually, system crash.

This happened again this morning and I finally know what's triggering it. It's THIS bug, with the atkbd problem I've been experiencing since I installed Precise on this machine.

Here are some excerpts:
The first instance happened at 0854 this morning: (The atkbd messages have been appearing longer, but the first time it hit SysRq was at 0854).

Jun 20 08:54:04 klaatu kernel: [85700.498471] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/serio2).
Jun 20 08:54:04 klaatu kernel: [85700.498480] atkbd serio2: Use 'setkeycodes 00 <keycode>' to make it known.
Jun 20 08:54:04 klaatu kernel: [85700.543991] atkbd serio2: Unknown key pressed (raw set 2, code 0x8 on isa0060/serio2).
Jun 20 08:54:04 klaatu kernel: [85700.544000] atkbd serio2: Use 'setkeycodes 08 <keycode>' to make it known.
Jun 20 08:54:04 klaatu kernel: [85700.545264] SysRq : HELP : loglevel(0-9) reBoot Crash terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount show-blocked-tasks(W) dump-ftrace-buffer(Z)
Jun 20 08:54:04 klaatu kernel: [85700.548084] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/serio2).
Jun 20 08:54:04 klaatu kernel: [85700.548093] atkbd serio2: Use 'setkeycodes 00 <keycode>' to make it known.
Jun 20 08:54:04 klaatu kernel: [85700.645263] atkbd serio2: Unknown key pressed (raw set 2, code 0x18 on isa0060/serio2).
Jun 20 08:54:04 klaatu kernel: [85700.645272] atkbd serio2: Use 'setkeycodes 18 <keycode>' to make it known.

Next, it hit the SysRq combo to change log level:

Jun 20 08:54:06 klaatu kernel: [85702.601963] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/serio2).
Jun 20 08:54:06 klaatu kernel: [85702.601971] atkbd serio2: Use 'setkeycodes 00 <keycode>' to make it known.
Jun 20 08:54:06 klaatu kernel: [85702.678295] atkbd serio2: Unknown key pressed (raw set 2, code 0x8 on isa0060/serio2).
Jun 20 08:54:06 klaatu kernel: [85702.678303] atkbd serio2: Use 'setkeycodes 08 <keycode>' to make it known.
Jun 20 08:54:06 klaatu kernel: [85702.679470] SysRq : Changing Loglevel
Jun 20 08:54:06 klaatu kernel: [85702.679479] Loglevel set to 0
Jun 20 08:54:06 klaatu kernel: [85702.681515] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/...

[Jeff Lane  (bladernr)]

Also, I have a 3GiB log file full of these messages if you want it :)

[Jeff Lane  (bladernr)]

So over the course of last night, these messages caused syslog to grow to 10GB in size filling up my root partition (root is only about 25GiB on this machine w/ /home taking up the other 225GiB). Needless to say, this caused a crash and I had to reboot, delete the log file and try again.

2 hours later, my machine spontaneously shut itself down because these random keypress events generated an Alt-SysRq event AGAIN.

This is getting to be more than just annoying and inconvenient. is anything being done on this? I'm on the verge of dumping precise and going back to Oneiric, which maybe the only solution to this problem, and a disappointing one at that. This is my primary workstation, and I can't get work done if I have to keep fighting random keystrokes appearing in my consoles while writing code, or causing my system to be unstable, or requiring me to reboot my machine several times a day.

What more do I need to do to help figure this thing out?

[Joseph Salisbury (jsalisbury)]

From previous comments, it sounds like this bug is fixed upstream. Can you confirm it is fixed in the Quantal kernel[0]? If it is, we can perform a reverse bisect to identify the fix.

For Quantal kernels, you need to install both the linux-image and linux-image-extras .deb packages.

[0] https://launchpad.net/ubuntu/+source/linux/3.5.0-4.4/+build/3643662

[Jeff Lane  (bladernr)]

Joeseph: I just got back from Holiday and picked this back up. I had already finished installing the 3.5.0-7 backports kernel from the X-swat PPA so I went with that instead:

bladernr@klaatu:~$ uname -r
3.5.0-7-generic
bladernr@klaatu:~$ uptime
 22:07:56 up 1:04, 1 user, load average: 0.25, 0.29, 0.31

So just an hour of uptime with no indication of the problematic keyboard messages or keypress injections. I'm gonna let this run overnight with some system stress and see if it stands up...

[Jeff Lane  (bladernr)]

bladernr@klaatu:~$ uptime
 08:47:07 up 11:43, 1 user, load average: 0.89, 0.32, 0.20

I ran a simulated stress test for 8 hours on it overnight, and have nearly 12 hours uptime with no problems at all.

I should add that the second problem (the touchpad going wonky requiring me to rmmod psmouse ALSO seems to have went away with the quantal kernel 3.5.0-7.

Yesterday with the current Precise kernel running, the touchpad would go wonky after about 5 - 15 minutes uptime, requiring me to remove the psmouse module completely and just just my USB mouse. the Set Keycode error messages would appear within an hour, max, and I had to reboot the machine 4 times yesterday while running the precise kernel due to these issues.

So I'm going to say that 3.5.0-7 Quantal LTS Backport fixes the problem too.

What kernel should I try next?

[Joseph Salisbury (jsalisbury)]

We can perform a reverse bisect to identify the commit that fixes this bug. Can you test the following kernels and report back:

v3.3 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-precise/
v3.4-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.4-rc1-precise/

[Jeff Lane  (bladernr)]

Tried both of those kernels. Neither one booted. I'll try to get some info on the actual issue after some meetings today.

[Jeff Lane  (bladernr)]

So, I tried booting 3.4-rc1 twice, and both times it failed to boot. I went in and removed the quiet, splash and vt_handoff options and set gfxpayload to text so I could figure out why it was hanging at boot.

And it booted.

So now I'm in 3.4-rc1:

bladernr@klaatu:~$ uname -r
3.4.0-030400rc1-generic

and I'll run this for 24 hours with normal workloads and later on I'll run stress testing and see if the issues crop up.

[Jeff Lane  (bladernr)]

Tried 3.3 after running 3.4-rc1 all day (yeah, impatient). 3.3 finally booted, and I had working TTYs, but never started due to nvidia_drv crashing.

So there was no point in continuing as this bug only occurs in X, not in TTYs.

I am back in 3.4-rc1 and will kick off a solid 8 hour stress run in a moment just to be sure.

As for 3.3, Im about to attach some logs.

I should add, that after all day of normal work usage, I did not see one single KeyCode error message, nor did my touchpad act up at all.

[Jeff Lane  (bladernr)]

This is syslog for the attempt to boot the 3.3 kernel.

[Jeff Lane  (bladernr)]

this is dmesg for the attempt to boot 3.3

[Jeff Lane  (bladernr)]

this is Xorg.0.log showing where X stopped with the nVidia driver

[Jeff Lane  (bladernr)]

OK... ran 3.4-rc1 for nearly 24 hours without a single incident, even with a total of 10 hours of systemwide stress overnight too.

No keyboard funk, and the trackpad is still sane.

[Joseph Salisbury (jsalisbury)]

Can you try to test v3.3.8:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3.8-quantal/

It would be helpful to confirm that the bug does exist in v3.3, so we can bisect properly.

[Jeff Lane  (bladernr)]

not only does 3.3.8 boot, but it also recompiled the nvidia-current driver so I have full resolution. I'll run this for work today and run a long stress run overnight as before and see how this kernel fares.

[Jeff Lane  (bladernr)]

Ok, so with 3.3.8, I got the keycode messages, but only in a couple instances:

syslog.1:Aug 2 16:01:00 klaatu kernel: [15038.967755] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
syslog.1:Aug 2 16:02:20 klaatu kernel: [15119.091221] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
syslog.1:Aug 2 16:02:38 klaatu kernel: [15136.781661] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
syslog.1:Aug 2 16:05:59 klaatu kernel: [15337.380699] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
syslog.1:Aug 2 16:08:33 klaatu kernel: [15491.449191] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
syslog.1:Aug 2 17:00:00 klaatu kernel: [18574.882233] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
syslog.1:Aug 2 17:02:09 klaatu kernel: [18703.813750] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
syslog.1:Aug 2 17:06:22 klaatu kernel: [18956.636771] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
syslog.1:Aug 2 17:43:41 klaatu kernel: [21193.819427] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

Those were all I got after booting up 3.3.8 and running that for a while. So not fixed, but much better. I did still experience the wonky touchpad thing, so I still had to rmmod psmouse, unlike with 3.4x or 3.5x kernels.

So what's next to test?

[Jeff Lane  (bladernr)]

I should add, with 3.3.8 I ran into a related keyboard symptom where the keyrepeat is either not honored or gets locked on.

For example, if I press and hold the left arrow key, the cursor may only move one space and stop, it may move left and NOT STOP even after I release the key, or it may not move at all.

[Joseph Salisbury (jsalisbury)]

So I think we can say that v3.3 has the bug and v3.4-rc1 does not have the bug.

I started a reverse kernel bisect between v3.3 and v3.4-rc1. This should help us identify the commit that fixes this bug.

The first test kernel is available and is built up to commit:
b2094ef840697bc8ca5d17a83b7e30fad5f1e9fa

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

OK, ran that one friday afternoon to saturday afternoon without a hitch, but... didn't do a lot of actual system use.
I ran some simulated stress, but I wonder if that has any real affect at all.

Ran 3.3.8-030308-generic today all day for work and it was ok until just a little while ago when it started crashing horribly again. This time I even left it long enough to see that the strange phantom keystrokes were even appearing in TTYs as well. So it's not limited to just X sessions or the like.

floods of the same error messages again in syslog, plus it once again hit enough random keys fast enough to fake an Alt-SysRq combo and generated a memory info dump. :( at least it didn't hit one of the destructive combos.

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
bbdb32cb5b73597386913d052165423b9d736145

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

Ok, so that latest kernel ran for approximately 1day 7 hours according to the 'uptime' command before symptoms were noticed. First, the touchpad went crazy, causing me to again rmmod the psmouse driver. Then the keyboard errors started appearing in dmesg. I'm thinking that I will need a minimum of 18 hours, or two business days, of usage to determine if a given kernel fixes the issue or no. :/

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
5f0e685f316a1de6d3af8b23eaf46651faca32ab

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

Still broken with the latest test kernel.

bladernr@klaatu:~/development$ uname -a
Linux klaatu 3.3.0-030300-generic #201208101657 SMP Fri Aug 10 16:01:13 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
02c502566ef505d0469fa27567f48766c1f5f7af

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

Still broken in the latest test kernel. Though the problem did not seem as bad (didn't occur as quickly as with other test kernels, but definitely did happen).

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
828006de1bddf83b6ecf03ec459c15f7c7c22db7

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

OK... that kernel has run over the weekend and has shown no problem:

bladernr@klaatu:~/Downloads$ uptime
 08:31:32 up 4 days, 16:48, 1 user, load average: 0.20, 0.26, 0.30
bladernr@klaatu:~/Downloads$ uname -a
Linux klaatu 3.3.0-030300rc4-generic #201208202102 SMP Mon Aug 20 20:06:58 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

I did a day and a half normal work and then a weekend worth of intermittant usage and have not seen a single atkbd message appear in dmesg logs... the trackpad DID finally go wonky last night, after roughly 4 days of use, but the keyboard has remained stable thus far.

[Jeff Lane  (bladernr)]

Joseph: Ignore that last comment. The system seemed stable for nearly 5 days, as noted, but at 1:30PM after I returned from lunch and started working again, the messages occurred and the system crashed. I was forced to power cycle it to restore a working state.

[Jeff Lane  (bladernr)]

I should add that on rebooting, the system had been up a total of 464 seconds according to dmesg before this started again :(

[Jeff Lane  (bladernr)]

This is on the most recent reboot:
[ 55.482734] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 55.943148] wlan0: no IPv6 routers present
[ 464.782111] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
[ 464.782138] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
[ 467.151377] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
[ 467.151383] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

On the previous test, The first instance didn't appear until way longer:

Aug 27 13:31:52 klaatu kernel: [423704.700498] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Aug 27 13:31:52 klaatu kernel: [423704.700527] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
Aug 27 13:31:54 klaatu kernel: [423707.030819] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio2/input/input20

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
cd8df932d894f3128c884e3ae1b2b484540513db

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

Yep, broke :)

bladernr@klaatu:~/development/bender/add-touchpad-driver-info/scripts$ uname -a
Linux klaatu 3.3.0-030300rc3-generic #201208281432 SMP Tue Aug 28 13:34:12 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[ 36.147311] eth0: no IPv6 routers present
[ 42.649691] wlan0: no IPv6 routers present
[40444.006865] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
[40444.006870] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
[40449.452380] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
[40449.452389] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
[40716.286755] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
[40716.286765] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
1ab142d499294b844ecc81e8004db4ce029b0b61

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

Bug still present in this latest kernel.

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
5556e147083fb4d473d5c1a82f73205b8b145cd9

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

Ok this one made it 37,000 seconds before it happened.

[25164.192477] cfg80211: (5735000 KHz - 5835000 KHz @ 40000 KHz), (300 mBi, 3000 mBm)
[25164.215953] wlan0: moving STA cc:b2:55:fe:77:62 to state 3
[37118.490879] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio2/input/input19
[37118.501419] atkbd serio2: Unknown key pressed (raw set 2, code 0x8 on isa0060/serio2).
bladernr@klaatu:~$ uname -a
Linux klaatu 3.3.0-030300rc1-generic #201209122010 SMP Wed Sep 12 19:14:34 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
c6b76d1f02e2ab1109d8549877a3a24c6a2b4587

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[kenorb (kenorb)]

The same here.
https://bugzilla.kernel.org/show_bug.cgi?id=47741

[Jeff Lane  (bladernr)]

Joseph: I'm sprinting in Montreal this week and on holiday next week. I'll try to run this one over next week but it'll be touch and go because I'll be out of town mostly. Don't expect an update until week after next realistically.

kenorb: The thaw messages are a symptom of the issue here, not the actual issue. But I'm glad I'm not the only person who's seen the thaw messages kill a machine :)

[Joseph Salisbury (jsalisbury)]

Thanks for the update, Jeff.

[Jeff Lane  (bladernr)]

Ok... I have the latest kernel up and running. One problem, however, is that the audio drivers don't work on this kernel...

bladernr@klaatu:~/coreutils-8.13/src$ uname -a
Linux klaatu 3.3.0-030300rc1-generic #201209172209 SMP Mon Sep 17 21:10:56 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

dmesg:
[ 22.651589] snd_hda_intel 0000:00:1b.0: irq 48 for MSI/MSI-X
[ 22.703012] hda_codec: cannot build controls for #0 (error -16)
[ 22.703287] hda_intel: Disabling MSI

The sound settings tool doesn't show any available input or output devices :( which makes team meetings difficult.

[Jeff Lane  (bladernr)]

Oct 1 12:30:12 klaatu kernel: [47350.310497] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Oct 1 12:30:12 klaatu kernel: [47350.310505] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

Made it a few hours this time.

I let it run and kept working on this kernel all day yesterday and let it sit over night... by this morning, the machine had frozen and I had to power cycle it to get things back in order.

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
7907ae3e50613ae1c6d1a10f34fcd63f4123b93d

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

That one too dies. Are we getting anywhere near close? this seems to be going on forever :( I know there are probably thousands of patches you're going through (and I can't even begin to imagine how many test kernels you are building each day for bisection testing). So just thought I'd ask. Eventually, I will have to upgrade this machine to Quantal, but I am trying to keep it at precise long enough to get this solved.

bladernr@klaatu:~/development/build-area$ uname -a
Linux klaatu 3.3.0-030300rc1-generic #201210021653 SMP Tue Oct 2 15:56:35 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[ 67.958559] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[17515.759827] CIFS VFS: default security mechanism requested. The default security mechanism will be upgraded from ntlm to ntlmv2 in kernel release 3.3
[25724.678092] iwlwifi 0000:07:00.0: Tx aggregation enabled on ra = cc:b2:55:fe:77:62 tid = 0
[29654.058516] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
[29654.058524] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

[Jeff Lane  (bladernr)]

Acually, we may be getting a lot closer than I thought. After I saw those, I left the machine running.

Before, once those started appearing, the machine would go for so long then become pretty unusable. I STILL ran into the problem where random phantom keypresses would appear on whatever I happened to have focused (terminal or web browser or Xchat) BUT the problem would clear itself up after a short while.

It's weird... it's as if these errors start, then calm down, then start up again, then calm down.

I'm running the above test kernel STILL:

bladernr@klaatu:~/development/sleep-test-resume-time/jobs$ uname -a
Linux klaatu 3.3.0-030300rc1-generic #201210021653 SMP Tue Oct 2 15:56:35 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

with a good bit of uptime at this point

bladernr@klaatu:~/development/sleep-test-resume-time/jobs$ uptime
 16:29:03 up 4 days, 8:40, 1 user, load average: 0.12, 0.14, 0.19

I HAVE, however, seen this fail after almost a week of uptime before, though, so I'll keep it up and running for a while longer. But so far the behaviour has been annoying, but not catastrophic as it was with other kernels.

[Joseph Salisbury (jsalisbury)]

We are getting closer. There are 3 more kernels to test tops:

Bisecting: 9 revisions left to test after this (roughly 3 steps)

The next test kernel is available and is built up to commit:
22f8d055350066b4a87de4adea8c5213cac54534

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

OK... so far so good. Still had the trackpad go nuts (that is usually the first indication. Trackpad goes wonky, I have to remove psmouse and then the atkbd messages start appearing along with weird keyboard activity.

This kernel has felt (subjectively) more stable so far.
bladernr@klaatu:~/development/merge-client-whitelists-2$ uptime
 17:43:27 up 1 day, 4:32, 1 user, load average: 0.24, 0.37, 0.34

after that time, so far, the atkbd messages have appeared, however, there have been very few and thus far the system hasn't spiraled off into the weeds. The first instance:
[ 4630.670657] sdhci-pci 0000:09:07.1: Will use DMA mode even though HW doesn't fully claim to support it.
[71711.611959] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
[71711.611968] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

and there have only been 5 messages so far. I'll continue running this one without rebooting to see what happens.

[Jeff Lane  (bladernr)]

Ok, So I've been running that last test kernel for a little while now
bladernr@klaatu:~$ uptime
 09:32:40 up 4 days, 20:22, 1 user, load average: 0.15, 0.39, 0.75

As mentioned in my last comment, the atkbd messages started appearing at about 71711 seconds and quickly settled down. Another one appeared at 85025 seconds and then went away until a very long time afterwards.

[85025.138764] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
[85025.138773] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.
[158468.069172] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
[158468.069181] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

So they still appear, but are even less common, IMO than in the last test kernel, and seem to be one or two-off instances only now, not the overwhelming flood that caused the system instability problems noted originally. I'll keep running this kernel straight until it dies or there's a new kernel to try.

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
cb3f2adc03ab055b19c677a6283523861fafebdd

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

Ok, that latest test kernel lasted only a short while before it exploded. And this time it really exploded. psmouse went nuts about 5 minutes in, forcing me to remove it. Then the atkbd messages appeared at 970 seconds in. Then the real flood began at about 4500 seconds in and that's when the computer exploded. When they started up again at 4500 seconds, they hit so hard that I had to power-cycle the system. random keypresses were happeneing all over hte place, causing tabs to open in my browser, any text box that got focus would fill with random characters, numlock came on and wouldn't go away, I couldn't even drop to a console. I had to power-cycle the machine :/

So the difference between 3.3.0-030300rc4.201210151833 and 3.3.0-030300rc4.201210092026 is polar opposites. The latest one was incredibly unstable, the previous one lasted for days and days and was nice and stable even though the atkbd messages DID still appear every now and then and psmouse went just nuts early on, as it always does.

Oct 19 10:16:42 klOct 19 09:16:32 klaatu kernel: [ 970.354748] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio2/input/input19
Oct 19 09:16:32 klaatu kernel: [ 970.366585] atkbd serio2: Unknown key pressed (raw set 2, code 0x8 on isa0060/serio2).
Oct 19 09:16:32 klaatu kernel: [ 970.366593] atkbd serio2: Use 'setkeycodes 08 <keycode>' to make it known.
Oct 19 09:16:32 klaatu kernel: [ 970.369077] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/serio2).
Oct 19 09:16:32 klaatu kernel: [ 970.369085] atkbd serio2: Use 'setkeycodes 00 <keycode>' to make it known.

bladernr@klaatu:~$ uname -a
Linux klaatu 3.3.0-030300rc4-generic #201210151833 SMP Mon Oct 15 17:35:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
ca3e9297e29c48eebb010d0225ea580fbcc64c88

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

The latest test kernel lasted a while before exploding.

the atkbd messages appeared only soradically for a while:
Oct 23 14:23:10 klaatu kernel: [ 4944.628478] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Oct 23 14:23:10 klaatu kernel: [ 4944.628489] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

Oct 23 14:46:17 klaatu kernel: [ 6330.318678] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Oct 23 14:46:17 klaatu kernel: [ 6330.318682] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

Oct 23 17:46:34 klaatu kernel: [17137.358919] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Oct 23 17:46:34 klaatu kernel: [17137.358928] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

and then just a little while ago, the flood begain:
Oct 24 15:23:55 klaatu kernel: [94900.613716] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio2/input/input19
Oct 24 15:23:55 klaatu kernel: [94900.625984] atkbd serio2: Unknown key pressed (raw set 2, code 0x8 on isa0060/serio2).
Oct 24 15:23:55 klaatu kernel: [94900.625992] atkbd serio2: Use 'setkeycodes 08 <keycode>' to make it known.
Oct 24 15:23:55 klaatu kernel: [94900.628092] atkbd serio2: Unknown key pressed (raw set 2, code 0x0 on isa0060/serio2).

this went on for a while until the system was hard locked and I had to power-cycle again

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
6681bc0deba495fad0d6fb349e40524abd1b1732

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

I'll load it up and start running, however, I am headed to UDS tomorrow morning, so it'll just be sitting idle for a long while. I'll be back on the 7th of November.

[Jeff Lane  (bladernr)]

Sorry for taking so long... the machine ran this kernel for about 11 days, at idle without issue, but was rebooted during a power outage :(

this time around, it's ran for about almost 2 days before issues started:

bladernr@klaatu:~/development/868678/checkbox/scripts$ uptime
 16:42:49 up 1 day, 17:22, 1 user, load average: 0.15, 0.18, 0.15
bladernr@klaatu:~/development/868678/checkbox/scripts$ uname -a
Linux klaatu 3.3.0-030300rc4-generic #201210252051 SMP Thu Oct 25 19:53:22 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

First instance was quite a ways in:
Nov 13 09:35:50 klaatu kernel: [123203.539592] atkbd serio0: Unknown key released (translated set 2, code 0xe0 on isa0060/serio0).
Nov 13 09:35:50 klaatu kernel: [123203.539603] atkbd serio0: Use 'setkeycodes e060 <keycode>' to make it known.

[Joseph Salisbury (jsalisbury)]

The next test kernel is available and is built up to commit:
424a6f6ef990b7e9f56f6627bfc6c46b493faeb4

This kernel is available from:
http://people.canonical.com/~jsalisbury/lp984265

Can you test that kernel and report back if it has the bug or not?

[Jeff Lane  (bladernr)]

Ok... so holiday delay...

This latest kernel ran for over 7 days with only minor hiccups until just now when it went off the rails and started dumping random characters to anything that had focus, including vterms.

here's the uptime:
00:01:37 up 7 days, 14:55, 1 user, load average: 0.77, 0.86, 0.85

logs show strings of the atkbd messages appearing all along, but they usually clear up. after nearly 8 days though, they finally overwhelmed the system.

[Joseph Salisbury (jsalisbury)]

Were you testing the kernel listed in comment #88 ?

[Jeff Lane  (bladernr)]

Yes... Linux klaatu 3.3.0-030300-generic #201211142008 SMP Wed Nov 14 21:02:35 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[Joseph Salisbury (jsalisbury)]

The reverse bisect reported commit 9fc377799bc9bfd8d5cb35d0d1ea2e2458cbdbb3 as the fix for this bug. However, that does not make sense. That commit is a merge of scsi-misc.

We may have reported a commit as bad when it was actually good during the bisect process. Maybe we just didn't test for long enough.

The easiest solution might be to use either the quantal or raring backport kernel if you plan on running Precise. Is that something you would consider? If so, it would be good to test the backport kernel for several days to ensure it really fixes the bug.

[Jeff Lane  (bladernr)]

It's possible. Some of these kernels will fail within a few hours. I recall having to reboot multiple times a day with some, while others appear to take almost a week of uptime before the symptoms become so bad that the system fails. Though the atkbd messages DO appear, they just don't start flooding the system until after several days.

I'll start with the Quantal backport and run that until it dies and see how long it takes... then perhaps try Raring.

[Jeff Lane  (bladernr)]

Of course, now that I said that, where are the backports? Is this the correct PPA: https://launchpad.net/~ubuntu-x-swat/+archive/q-lts-backport

I have run this before, and I now recall that one big issue I had was that the nvidia-current driver wouldn't build for it. Nouveau doesn't work on my system (not well, at least, it's barely usable) so I need to have nvidia proprietary drivers :(

I could just upgrade to Quantal, but have been holding off on that to see if this bug can be resolved.

What are your thoughts on that?

[Joseph Salisbury (jsalisbury)]

Can you test the Quantal kernel and let it run for a few days. If you can confirm that the bug does not exist there, it may be best to just upgrade to Quantal. If the bug is fixed in Quantal, then there will also be a fix available for others by installing the backports kernel.

[penalvch (penalvch)]

Jeff Lane, as per http://www.dell.com/support/drivers/us/en/19/Product/alienware-m15x an update is available for your BIOS (A09). If you update to this following https://help.ubuntu.com/community/BiosUpdate , does it change anything? If it doesn't, could you please both specify what happened, and provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

Thank you for your understanding.

[Jeff Lane  (bladernr)]

Became fine again on Saucy, now running Trusty without issue. The fact that this comes and goes with varying kernel families tells me this is tied into the kernel somewhere... but in any case, it's working now and I really don't care to fight with this any more (this is actually the second long-running bug I've had about this issue, and countless production, devel and test kernels attempted with varying degrees of success.