Corrupted low memory after resume from suspend after updating to saucy

[benpicco]

After upgrading to saucy and Linux 3.11 my Gigabyte 965P-DS3 board no longer properly resumes from suspend.
The desktop still works fine after resume, but the ethernet interface won't come up again.
dmesg after resume warns about corrupted low memory:

[40560.864036] ------------[ cut here ]------------
[40560.864044] WARNING: CPU: 0 PID: 24687 at /build/buildd/linux-3.11.0/arch/x86/kernel/check.c:140 check_for_bios_corruption+0x10f/0x120()
[40560.864046] Memory corruption detected in low memory
[40560.864048] Modules linked in: pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) cuse ipt_MASQUERADE(F) iptable_nat(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) xt_conntrack(F) nf_conntrack(F) ipt_REJECT(F) xt_CHECKSUM(F) iptable_mangle(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) joydev(F) xpad ff_memless hid_generic usbhid hid rfcomm bnep bluetooth binfmt_misc(F) kvm_intel(F) kvm(F) gpio_ich ppdev(F) ir_lirc_codec lirc_dev ir_sanyo_decoder ir_mce_kbd_decoder ir_sony_decoder ir_jvc_decoder ir_rc6_decoder ir_rc5_decoder ir_nec_decoder e4000 rtl2832 usb_storage(F) snd_usb_audio snd_usbmidi_lib uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core videodev dvb_usb_rtl28xxu rtl2830 dvb_usb_v2 dvb_core rc_core snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep(F) snd_pcm(F) snd_page_alloc(F) snd_seq_midi(F) snd_seq_midi_event(F) microcode(F) pcspkr serio_raw(F) snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) snd(F) lpc_ich soundcore(F) nvidia(POF) parport_pc(F) drm mac_hid it87 hwmon_vid coretemp lp(F) parport(F) pata_acpi sky2 pata_jmicron ahci(F) libahci(F)
[40560.864148] CPU: 0 PID: 24687 Comm: kworker/0:1 Tainted: PF W O 3.11.0-13-generic #20-Ubuntu
[40560.864151] Hardware name: Gigabyte Technology Co., Ltd. 965P-DS3/965P-DS3, BIOS F14d 12/18/2008
[40560.864155] Workqueue: events check_corruption
[40560.864158] 0000000000000009 ffff880064be9d30 ffffffff816e54ba ffff880064be9d78
[40560.864162] ffff880064be9d68 ffffffff81061dbd 0000000000000000 ffff880000010000
[40560.864166] ffffffff81ea73b0 0000000000000001 ffff880000000000 ffff880064be9dc8
[40560.864171] Call Trace:
[40560.864178] [<ffffffff816e54ba>] dump_stack+0x45/0x56
[40560.864183] [<ffffffff81061dbd>] warn_slowpath_common+0x7d/0xa0
[40560.864187] [<ffffffff81061e2c>] warn_slowpath_fmt+0x4c/0x50
[40560.864191] [<ffffffff8104e5bf>] check_for_bios_corruption+0x10f/0x120
[40560.864195] [<ffffffff8104e5de>] check_corruption+0xe/0x40
[40560.864200] [<ffffffff8107d05c>] process_one_work+0x17c/0x430
[40560.864204] [<ffffffff8107dcac>] worker_thread+0x11c/0x3c0
[40560.864208] [<ffffffff8107db90>] ? manage_workers.isra.24+0x2a0/0x2a0
[40560.864212] [<ffffffff810847b0>] kthread+0xc0/0xd0
[40560.864217] [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
[40560.864221] [<ffffffff816f51ec>] ret_from_fork+0x7c/0xb0
[40560.864225] [<ffffffff810846f0>] ? kthread_create_on_node+0x120/0x120
[40560.864228] ---[ end trace a5d0e3f4744a1e9d ]---

This did not happen with raring/Linux 3.8

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: linux-image-3.11.0-13-generic 3.11.0-13.20
ProcVersionSignature: Ubuntu 3.11.0-13.20-generic 3.11.6
Uname: Linux 3.11.0-13-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.12.5-0ubuntu2.1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: benpicco 3071 F.... pulseaudio
 /dev/snd/controlC0: benpicco 3071 F.... pulseaudio
Date: Mon Nov 18 14:12:08 2013
HibernationDevice: RESUME=UUID=81e8d3af-3d5b-4a61-a550-702a919449c5
InstallationDate: Installed on 2012-02-27 (629 days ago)
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111012)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.

 virbr0 no wireless extensions.
MachineType: Gigabyte Technology Co., Ltd. 965P-DS3
MarkForUpload: True
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.11.0-13-generic root=UUID=5b216424-8d68-4640-9d4c-494eb0e00e9d ro quiet splash
RelatedPackageVersions:
 linux-restricted-modules-3.11.0-13-generic N/A
 linux-backports-modules-3.11.0-13-generic N/A
 linux-firmware 1.116
RfKill:

SourcePackage: linux
UpgradeStatus: Upgraded to saucy on 2013-11-16 (2 days ago)
dmi.bios.date: 12/18/2008
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: F14d
dmi.board.name: 965P-DS3
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrF14d:bd12/18/2008:svnGigabyteTechnologyCo.,Ltd.:pn965P-DS3:pvr:rvnGigabyteTechnologyCo.,Ltd.:rn965P-DS3:rvr:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: 965P-DS3
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Can you give the latest 3.12 kernel[0] a test, to see if this is already fixed in mainline? If the bug still exits, we can perform a bisect to find the commit that introduced this.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.12-trusty/

[benpicco (benpicco)]

> Can you give the latest 3.12 kernel[0] a test, to see if this is
> already fixed in mainline?

The issue persists with 3.12

[ 0.000000] Linux version 3.12.0-031200-generic (apw@gomeisa) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #201311071835 SMP Thu Nov 7 23:36:07 UTC 2013
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.12.0-031200-generic root=UUID=5b216424-8d68-4640-9d4c-494eb0e00e9d ro quiet splash

[…]

[ 180.960026] Corrupted low memory at ffff88000000e4b8 (e4b8 phys) = 400000000000
[ 180.960029] ------------[ cut here ]------------
[ 180.960036] WARNING: CPU: 0 PID: 36 at /home/apw/COD/linux/arch/x86/kernel/check.c:140 check_for_bios_corruption+0x10e/0x120()
[ 180.960037] Memory corruption detected in low memory
[ 180.960089] Modules linked in: snd_hrtimer cuse ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables rfcomm bnep bluetooth hid_generic usbhid hid binfmt_misc joydev xpad ff_memless kvm_intel kvm usb_storage ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sanyo_decoder ir_sony_decoder ir_jvc_decoder ir_rc6_decoder ir_rc5_decoder ir_nec_decoder e4000 gpio_ich rtl2832 ppdev dvb_usb_rtl28xxu rtl2830 dvb_usb_v2 dvb_core rc_core uvcvideo videobuf2_vmalloc videobuf2_memops snd_hda_codec_realtek videobuf2_core snd_usb_audio videodev snd_hda_intel snd_usbmidi_lib snd_hda_codec snd_hwdep microcode snd_pcm snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq serio_raw pcspkr lpc_ich snd_seq_device snd_timer snd it87 hwmon_vid coretemp parport_pc soundcore lp mac_hid parport pata_acpi pata_jmicron ahci sky2 libahci
[ 180.960109] CPU: 0 PID: 36 Comm: kworker/0:1 Not tainted 3.12.0-031200-generic #201311071835
[ 180.960110] Hardware name: Gigabyte Technology Co., Ltd. 965P-DS3/965P-DS3, BIOS F14d 12/18/2008
[ 180.960113] Workqueue: events check_corruption
[ 180.960117] 000000000000008c ffff880196b9fce8 ffffffff81742357 ffffffff81c4adc8
[ 180.960120] ffff880196b9fd38 ffff880196b9fd28 ffffffff8106782c ffff880196b9fd28
[ 180.960123] 0000000000000000 ffff880000010000 ffffffff81eaa550 0000000000000001
[ 180.960124] Call Trace:
[ 180.960130] [<ffffffff81742357>] dump_stack+0x46/0x58
[ 180.960134] [<ffffffff8106782c>] warn_slowpath_common+0x8c/0xc0
[ 180.960137] [<ffffffff81067916>] warn_slowpath_fmt+0x46/0x50
[ 180.960140] [<ffffffff8105392e>] check_for_bios_corruption+0x10e/0x120
[ 180.960142] [<ffffffff8105394e>] check_corruption+0xe/0x40
[ 180.960145] [<ffffffff8108436f>] process_one_work+0x17f/0x4d0
[ 180.960148] [<ffffffff810855cb>] worker_thread+0x11b/0x3d0
[ 180.960151] [<ffffffff810854b0>] ? manage_workers.isra.20+0x1b0/0x1b0
[ 180.960154] [<ffffffff8108c790>] kthread+0xc0/0xd0
[ 180.960158] [<ffffffff8108c6d0>] ? flush_kthread_worker+0xb0/0xb0
[ 180.960162] [<ffffffff81757afc>] ret_from_fork+0x7c/0xb0
[ 180.960165] [<ffffffff8108c6d0>] ? flush_kthread_worker+0xb0/0xb0
[ 180.960167] ---[ end trace deb605f2a3100e54 ]---

(no proprietary modules linked in this time)

[benpicco (benpicco)]

Some further testing shows that it still works with 3.9 and 3.10.
With 3.11-rc1 the system won't suspend at all, but freeze when trying
to go to sleep.

[Joseph Salisbury (jsalisbury)]

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

Please follow the instructions on the wiki page[0]. The first step is to email the appropriate mailing list. If no response is received, then a bug may be opened on bugzilla.kernel.org.

Once this bug is reported upstream, please add the tag: 'kernel-bug-reported-upstream'.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[Joseph Salisbury (jsalisbury)]

In addition to reporting this upstream, we can perform a bisect to identify the commit that introduced this. Can you give 3.11-rc2 a test to see if it has the bug:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.11-rc2-saucy/

To perform a bisect, we need to identify the last good kernel and the first bad kernel version.

[benpicco (benpicco)]

The first kernel with the issue is 3.11-rc3, however rc1 and rc2 will
fail to suspend entirely.

[Joseph Salisbury (jsalisbury)]

I found a similar issue in Arch[0]. That bug was closed mentioning the systems BIOS corrupted the memory. Could you check that you are running the latest BIOS for your machine? If you are, we can perform a bisect since this issue wasn't happening in prior kernel versions.

[0] https://bugs.archlinux.org/task/35312

[benpicco (benpicco)]

F14 is the latest BIOS version available for this board.
I guess it will take two bisect runs, one to find the commit that fixed the failing suspend in rc1 and rc2, and then use that commit to bisect rc1/2, given that it didn't introduce this issue.

[Brian Wright (bdw)]

I'm also experienced this bug with kernel 3.11.0-14-generic:

2013-11-21 21:16:45 bdw-desktop kernel [33960.928028] Corrupted low memory at ffff88000000c6a8 (c6a8 phys) = 400000000000
2013-11-21 21:16:45 bdw-desktop kernel [33960.928034] ------------[ cut here ]------------
2013-11-21 21:16:45 bdw-desktop kernel [33960.928042] WARNING: CPU: 0 PID: 36 at /build/buildd/linux-3.11.0/arch/x86/kernel/check.c:140 check_for_bios_corruption+0x10f/0x120()
2013-11-21 21:16:45 bdw-desktop kernel [33960.928044] Memory corruption detected in low memory
2013-11-21 21:16:45 bdw-desktop kernel [33960.928045] Modules linked in: btusb nls_utf8 nls_iso8859_1(F) ext2(F) isofs(F) msdos(F) snd_hrtimer(F) pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) rpcsec_gss_krb5 nfsv4(F) nfsd(F) rfcomm auth_rpcgss(F) bnep nfs_acl(F) nfs(F) lockd(F) sunrpc(F) fscache(F) binfmt_misc(F) dm_crypt(F) fglrx(POF) coretemp kvm_intel(F) kvm(F) snd_hda_codec_hdmi gpio_ich snd_hda_codec_realtek ppdev(F) uvcvideo snd_hda_intel snd_usb_audio snd_hda_codec snd_usbmidi_lib snd_seq_midi(F) videobuf2_vmalloc snd_hwdep(F) snd_seq_midi_event(F) microcode(F) psmouse(F) snd_pcm(F) videobuf2_memops snd_rawmidi(F) snd_seq(F) videobuf2_core snd_page_alloc(F) snd_seq_device(F) videodev snd_timer(F) serio_raw(F) snd(F) soundcore(F) bluetooth lpc_ich lp(F) amd_iommu_v2 mac_hid parport_pc(F) parport(F) raid10(F) raid456(F) async_raid6_recov(F) async_memcpy(F) async_pq(F) async_xor(F) async_tx(F) xor(F) raid6_pq(F) raid1(F) raid0(F) multipath(F) linear(F) hid_generic usbhid hid vesafb(F) ses enclosure
2013-11-21 21:16:45 bdw-desktop kernel pata_acpi usb_storage(F) firewire_ohci firewire_core crc_itu_t(F) r8169 pata_jmicron mii(F) ahci(F) libahci(F) [last unloaded: btusb]
2013-11-21 21:16:45 bdw-desktop kernel [33960.928122] CPU: 0 PID: 36 Comm: kworker/0:1 Tainted: PF O 3.11.0-14-generic #21-Ubuntu
2013-11-21 21:16:45 bdw-desktop kernel [33960.928124] Hardware name: Gigabyte Technology Co., Ltd. EP45-UD3P/EP45-UD3P, BIOS F6 11/14/2008
2013-11-21 21:16:45 bdw-desktop kernel [33960.928127] Workqueue: events check_corruption
2013-11-21 21:16:45 bdw-desktop kernel [33960.928129] 0000000000000009 ffff8802243c7d30 ffffffff816e593a ffff8802243c7d78
2013-11-21 21:16:45 bdw-desktop kernel [33960.928133] ffff8802243c7d68 ffffffff81061dbd 0000000000000000 ffff880000010000
2013-11-21 21:16:45 bdw-desktop kernel [33960.928137] ffffffff81ea73b0 0000000000000001 ffff880000000000 ffff8802243c7dc8
2013-11-21 21:16:45 bdw-desktop kernel [33960.928140] Call Trace:
2013-11-21 21:16:45 bdw-desktop kernel [33960.928146] [<ffffffff816e593a>] dump_stack+0x45/0x56
2013-11-21 21:16:45 bdw-desktop kernel [33960.928150] [<ffffffff81061dbd>] warn_slowpath_common+0x7d/0xa0
2013-11-21 21:16:45 bdw-desktop kernel [33960.928154] [<ffffffff81061e2c>] warn_slowpath_fmt+0x4c/0x50
2013-11-21 21:16:45 bdw-desktop kernel [33960.928157] [<ffffffff8104e5bf>] check_for_bios_corruption+0x10f/0x120
2013-11-21 21:16:45 bdw-desktop kernel [33960.928160] [<ffffffff8104e5de>] check_corruption+0xe/0x40
2013-11-21 21:16:45 bdw-desktop kernel [33960.928164] [<ffffffff8107d05c>] process_one_work+0x17c/0x430
2013-11-21 21:...

[Joseph Salisbury (jsalisbury)]

I started a bisect between 3.11-rc2 and 3.11-rc3 The kernel bisect will require testing of about 7-10 test kernels. This will be the first of two bisects. We'll perform this bisect to identify the fix to allow your system to suspend. Once we have that, we can then see if the original bug was introduced in rc1 or rc2, then start the second bisect.

I built the first test kernel, up to the following commit:
c7dad2343f494359f6e45f62ff97055749b99670

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1252266

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[Joseph Salisbury (jsalisbury)]

@Brian, are you also unable to suspend with the v3.11-rc1 kernel?

[Brian Wright (bdw)]

@Joseph - unfortunately, I wasn't able to install the kernel, the headers wouldn't install and neither would the kernel image:

sudo dpkg -i linux-headers-3.11.0-031100rc2-generic_3.11.0-031100rc2.201311221305_amd64.deb
(Reading database ... 381064 files and directories currently installed.)
Preparing to replace linux-headers-3.11.0-031100rc2-generic 3.11.0-031100rc2.201311221305 (using linux-headers-3.11.0-031100rc2-generic_3.11.0-031100rc2.201311221305_amd64.deb) ...
Unpacking replacement linux-headers-3.11.0-031100rc2-generic ...
dpkg: dependency problems prevent configuration of linux-headers-3.11.0-031100rc2-generic:
 linux-headers-3.11.0-031100rc2-generic depends on linux-headers-3.11.0-031100rc2; however:
  Package linux-headers-3.11.0-031100rc2 is not installed.

dpkg: error processing linux-headers-3.11.0-031100rc2-generic (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 linux-headers-3.11.0-031100rc2-generic

sudo dpkg -i linux-image-3.11.0-031100rc2-generic_3.11.0-031100rc2.201311221305_amd64.deb

Selecting previously unselected package linux-image-3.11.0-031100rc2-generic.
(Reading database ... 376237 files and directories currently installed.)
Unpacking linux-image-3.11.0-031100rc2-generic (from linux-image-3.11.0-031100rc2-generic_3.11.0-031100rc2.201311221305_amd64.deb) ...
Done.
Setting up linux-image-3.11.0-031100rc2-generic (3.11.0-031100rc2.201311221305) ...
Running depmod.
update-initramfs: deferring update (hook will be called later)
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.11.0-031100rc2-generic /boot/vmlinuz-3.11.0-031100rc2-generic
run-parts: executing /etc/kernel/postinst.d/dkms 3.11.0-031100rc2-generic /boot/vmlinuz-3.11.0-031100rc2-generic
ERROR (dkms apport): kernel package linux-headers-3.11.0-031100rc2-generic is not supported
ERROR (dkms apport): kernel package linux-headers-3.11.0-031100rc2-generic is not supported
Error! Bad return status for module build on kernel: 3.11.0-031100rc2-generic (x86_64)
Consult /var/lib/dkms/fglrx-updates/13.101/build/make.log for more information.
Error! Bad return status for module build on kernel: 3.11.0-031100rc2-generic (x86_64)
Consult /var/lib/dkms/vboxhost/4.3.2/build/make.log for more information.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 3.11.0-031100rc2-generic /boot/vmlinuz-3.11.0-031100rc2-generic
update-initramfs: Generating /boot/initrd.img-3.11.0-031100rc2-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
run-parts: executing /etc/kernel/postinst.d/pm-utils 3.11.0-031100rc2-generic /boot/vmlinuz-3.11.0-031100rc2-generic
run-parts: executing /etc/kernel/postinst.d/update-notifier 3.11.0-031100rc2-generic /boot/vmlinuz-3.11.0-031100rc2-generic
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 3.11.0-031100rc2-generic /boot/vmlinuz-3.11.0-031100rc2-generic
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.11.0-031100rc2-generic
Found initrd image: /boot/initrd.img-3.11.0-031100rc2-generic
Found linux image: /boot/vmlinuz-3.11.0-14-generic
Found initrd image: /boot/initrd.img-3.11.0-14-generic
Found memtest86+ image:...

[Joseph Salisbury (jsalisbury)]

@benpicco, are you also unable to install the kernel?

[benpicco (benpicco)]

> I built the first test kernel, up to the following commit:
> c7dad2343f494359f6e45f62ff97055749b99670
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1252266
>
> Can you test that kernel and report back if it has the bug or not. I
> will build the next test kernel based on your test results.

This one suspends, but corrupts memory.
So I guess that would be 'good'.

[benpicco (benpicco)]

…or rather 'bad' as you are looking for the commit that fixed it

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
4f3cc4809a98a165a9708b72b47de71643797bbd

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1252266

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[benpicco (benpicco)]

> I built the next test kernel, up to the following commit:
> 4f3cc4809a98a165a9708b72b47de71643797bbd
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1252266
>
> Can you test that kernel and report back if it has the bug or not. I
> will build the next test kernel based on your test results.

Suspends, corrupts memory

[Rafael Belmonte (eaglescreen)]

Same corruption bug here. But my laptop suspends well.

    [ 63.096597] Corrupted low memory at c000659c (659c phys) = 00001a00
    [ 63.096628] ------------[ cut here ]------------
    [ 63.096640] WARNING: CPU: 3 PID: 61 at /build/buildd/linux-3.11.0/arch/x86/kernel/check.c:140 check_for_bios_corruption+0xbe/0xd0()
    [ 63.096643] Memory corruption detected in low memory
    [ 63.096646] Modules linked in: parport_pc(F) ppdev(F) bnep rfcomm bluetooth binfmt_misc(F) joydev(F) coretemp arc4(F) kvm(F) hp_wmi sparse_keymap ath9k ath9k_common ath9k_hw snd_hda_codec_hdmi ath snd_hda_codec_idt uvcvideo mac80211 snd_hda_intel videobuf2_vmalloc dm_multipath(F) scsi_dh(F) videobuf2_memops snd_hda_codec videobuf2_core i7core_edac microcode(F) videodev psmouse(F) edac_core serio_raw(F) snd_hwdep(F) cfg80211 snd_pcm(F) snd_page_alloc(F) snd_seq_midi(F) snd_seq_midi_event(F) snd_rawmidi(F) mei_me lpc_ich snd_seq(F) mei snd_seq_device(F) snd_timer(F) fglrx(POF) snd(F) soundcore(F) hp_accel lis3lv02d input_polldev mac_hid lp(F) parport(F) dm_mirror(F) dm_region_hash(F) dm_log(F) vesafb(F) hid_generic hid_logitech_dj usbhid hid r8169 wmi ahci(F) libahci(F) mii(F) video(F)
    [ 63.096733] CPU: 3 PID: 61 Comm: kworker/3:1 Tainted: PF O 3.11.0-12-generic #19-Ubuntu
    [ 63.096734] Hardware name: Hewlett-Packard HP Pavilion dv6 Notebook PC/1448, BIOS F.29 11/07/2011
    [ 63.096736] Workqueue: events check_corruption
    [ 63.096737] 00000000 00000000 f0d13e8c c162566b f0d13ecc f0d13ebc c105273e c17f3948
    [ 63.096742] f0d13ee8 0000003d c17f3974 0000008c c10458ae c10458ae 00000000 c0010000
    [ 63.096746] c1a58470 f0d13ed4 c1052793 00000009 f0d13ecc c17f3948 f0d13ee8 f0d13efc
    [ 63.096750] Call Trace:
    [ 63.096755] [<c162566b>] dump_stack+0x41/0x52
    [ 63.096758] [<c105273e>] warn_slowpath_common+0x7e/0xa0
    [ 63.096760] [<c10458ae>] ? check_for_bios_corruption+0xbe/0xd0
    [ 63.096762] [<c10458ae>] ? check_for_bios_corruption+0xbe/0xd0
    [ 63.096765] [<c1052793>] warn_slowpath_fmt+0x33/0x40
    [ 63.096767] [<c10458ae>] check_for_bios_corruption+0xbe/0xd0
    [ 63.096769] [<c10458d0>] check_corruption+0x10/0x40
    [ 63.096773] [<c1069f88>] process_one_work+0x118/0x380
    [ 63.096775] [<c105f1da>] ? mod_timer+0xea/0x1c0
    [ 63.096778] [<c106ad61>] worker_thread+0x101/0x340
    [ 63.096780] [<c106ac60>] ? manage_workers.isra.26+0x250/0x250
    [ 63.096783] [<c1070164>] kthread+0x94/0xa0
    [ 63.096785] [<c1070000>] ? __kthread_parkme+0x60/0x70
    [ 63.096788] [<c1632c37>] ret_from_kernel_thread+0x1b/0x28
    [ 63.096790] [<c10700d0>] ? kthread_create_on_node+0xc0/0xc0
    [ 63.096792] ---[ end trace 2a24700c96ce56b5 ]---

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
5f0e5afa0de4522abb3ea7d1369039b94e740ec5

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1252266

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[benpicco (benpicco)]

> I built the next test kernel, up to the following commit:
> 5f0e5afa0de4522abb3ea7d1369039b94e740ec5
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1252266

Suspends, corrupts memory

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
1e0f7a21b2fffc70f27cc4a454c60321501045b1

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1252266

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[benpicco (benpicco)]

> I built the next test kernel, up to the following commit:
> 1e0f7a21b2fffc70f27cc4a454c60321501045b1
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1252266

suspends, corrupts memory

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
b7649158a0d241f8d53d13ff7441858539e16656

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1252266

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[benpicco (benpicco)]

> I built the next test kernel, up to the following commit:
> b7649158a0d241f8d53d13ff7441858539e16656
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1252266

Suspends, corrupts memory

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
bb6acb289fbaac0e99eb552abdefc80a2186ef3f

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1252266

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[benpicco (benpicco)]

> I built the next test kernel, up to the following commit:
> bb6acb289fbaac0e99eb552abdefc80a2186ef3f
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1252266

Suspends, corrupts memory

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
c6cc142dac52e62e1e8a2aff5de1300202b96c66

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1252266

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[benpicco (benpicco)]

> I built the next test kernel, up to the following commit:
> c6cc142dac52e62e1e8a2aff5de1300202b96c66
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1252266

Suspends, corrupts memory

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
c1a15d08f497150a91ba4e61bab54b8f5c8b49b9

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1252266

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

Thanks in advance

[benpicco (benpicco)]

> I built the next test kernel, up to the following commit:
> c1a15d08f497150a91ba4e61bab54b8f5c8b49b9
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1252266

Suspends, corrupts memory

[Joseph Salisbury (jsalisbury)]

The reverse bisect indicates the following commit fixes the suspend issue:

commit c1a15d08f497150a91ba4e61bab54b8f5c8b49b9
Author: Roger Pau Monne <email address hidden>
Date: Wed Apr 17 20:18:55 2013 +0200

    xen-blkback: print stats about persistent grants

That really doesn't make sense, but we can check it out. I can build a 3.11-rc1 and 3.11-rc2 kernel with this commit to now bisect for the original bug.

Before starting the second bisect, can you give the latest mainline kernel a test, to see if the issue was already resolved? It is available from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc3-trusty/

v3.13-rc4 is the current mainline kernel, but it has issues with building, so a test kernel is not available for it. Depending on when you read this comment, the 3.13-rc5 kernel may be available, since it is released on Fridays:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc5-trusty/

[benpicco (benpicco)]

> The reverse bisect indicates the following commit fixes the suspend
> issue:
>
> commit c1a15d08f497150a91ba4e61bab54b8f5c8b49b9
> Author: Roger Pau Monne <email address hidden>
> Date: Wed Apr 17 20:18:55 2013 +0200
>
> xen-blkback: print stats about persistent grants
>
>
> That really doesn't make sense, but we can check it out. I can build
> a 3.11-rc1 and 3.11-rc2 kernel with this commit to now bisect for the
> original bug.

That can't be, I have no xen modules in use and this commit only
changes xen-blkback/blkback.c

Maybe it's easier to see what the first commit between 3.10 and
3.11-rc1 is that introduces an issue.
I could probably bisect this too, but make-kpkg takes an awful lot of
time because it always seems to rebuild everything - is there a way
around that?

> Before starting the second bisect, can you give the latest mainline
> kernel a test, to see if the issue was already resolved? It is
> available from:
> http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc3-trusty/

This one won't wake up anymore, probably a kernel panic but the display
won't get initialized again, no network and nothing written to the log.

> v3.13-rc4 is the current mainline kernel, but it has issues with
> building, so a test kernel is not available for it. Depending on when
> you read this comment, the 3.13-rc5 kernel may be available, since it
> is released on Fridays:
>
> http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc5-trusty/

Nothing there yet, may try again later.

[Joseph Salisbury (jsalisbury)]

Actually the v3.13-rc7 kernel is now out. Can you see if this resolves the original bug? If not, we can investigate further:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc7-trusty/

[Carlo Pires (carlopires)]

This kernel (v3.13.0-031300rc7-generic) worked for me (Ubuntu 13.10 + mb Gigabyte P35-DQ6). The system now suspends and resumes perfectly. Memory corruption is gone. Thanks!!!

[Carlo Pires (carlopires)]

Oops, my bad. After some time (about 5 minutes) from resume. I got this message in dmesg:

[ 420.828062] Corrupted low memory at ffff88000000e3f8 (e3f8 phys) = 400000000000

But I'm not sure if it is related to same problem.

[Carlo Pires (carlopires)]

I just confirmed that this kernel (v3.13.0-031300rc7-generic) does not solve the problem. Same error as before.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Theo (theobez95)]

Utopic here.

[13998.659944] Corrupted low memory at ffff88000000afa8 (afa8 phys) = ff575351ff221f1e
[13998.659945] Corrupted low memory at ffff88000000afb0 (afb0 phys) = ffc0bbb8ffbfbab7
[13998.659946] Corrupted low memory at ffff88000000afb8 (afb8 phys) = ff221f1eff948f8c
[13998.659947] Corrupted low memory at ffff88000000afc0 (afc0 phys) = ff7e7a77ff736e6c
[13998.659948] Corrupted low memory at ffff88000000afc8 (afc8 phys) = ffb3aeabffbeb8b5
[13998.659949] Corrupted low memory at ffff88000000afd0 (afd0 phys) = ff312e2cff474342
[13998.659950] Corrupted low memory at ffff88000000afd8 (afd8 phys) = ff2d2b2aff252221
[13998.659951] Corrupted low memory at ffff88000000afe0 (afe0 phys) = ffb0a9a7ff575352

etc

Lots of this (>3k).

Linux widly 3.16.0-24-generic #32-Ubuntu SMP Tue Oct 28 13:07:32 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Gábor Szécsi (deje07)]

Based on forum thread at Arch (https://bbs.archlinux.org/viewtopic.php?id=189483) this seems to be a bug in the BIOS. And it seems that the message is just for diagnostics, and has no direct cause on suspend/resume problems.

However I still experience the same issue, that after resuming from sleep mode the ethernet does not work. Most of time killall NetworkManager works, but not always. And I also have a Gigabyte board...

Using 12.04.5 I had no issues with suspend/resume.

[Andri Möll (moll)]

For the sake of completeness, got this on my first try of suspend to RAM, too, with a Gigabyte GA-X48-DS4 board, after resume with Wake-on-LAN. On 14.04.3 LTS with Linux v3.13.0-77-generic.