[Gigabyte P75-D3] I/O Problems

[zaxx]

I built a new system and installed Ubuntu 12.10. The system has the following specifications:

Samsung 840 Pro 128 GB
Gigabyte P75-D3
Intel i7 3770
32 GB Ram
Western Digital 1 TB Black (mounted as /home)
Western Digital 3 TB Green (mounted as /media/data_drive)

The initial Ubuntu 12.10 install took almost 6 hours as transfer rates were very slow. After reboot everything appeared very fast. However when I did an "apt-get update" the download took 20 seconds and the processing of lists took 19 minutes.

I had 12.10 installed on another computer which was working great (older Althon system) so I installed the drive and tested the new machine. It suffered the same performance problems.

I installed Windows 7 and performed speeds tests, it reported great results.

I installed 12.04 onto the 1 TB drive and it did not suffer the same performance problems. Everything ran great.

I then upgraded 12.10 to 13.04 (January 27, 2013) and it too suffered from the performance issue.

It appears that a change between 12.04 to 12.10 impacts AHCI based systems specifically running the Intel B75 chipset. I am not certain but I think that either the motherboard is misrepresenting something (that 12.04 ignores but it impacts 12.10) or the kernel is not detecting the system correctly.

I created a couple of pastebins with data collected:

12.10 Install: http://pastebin.ubuntu.com/1561660/
12.04 Live Disk (was checked against installed version and data is mostly the same): http://paste.ubuntu.com/1565140/

I have a thread on the ubuntu forums that may provide additional details: http://ubuntuforums.org/showthread.php?t=2104709

I also started an e-mail thread to the ubuntu-users mailing list: http://ubuntu.5.n6.nabble.com/Seeking-Help-td5010422.html

I have done a lot of troubleshooting and I feel that the problem must either be the kernel or the bios misrepresenting information. In either case I am stuck without any idea of where to go next with the debuging and diagnoses path.

If any additional information needs to be collected, please let me know and I will work to assist.
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu17.1
Architecture: i386
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: bakers 1729 F.... pulseaudio
 /dev/snd/controlC0: bakers 1729 F.... pulseaudio
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Card0.Amixer.info:
 Card hw:0 'PCH'/'HDA Intel PCH at 0xf7110000 irq 45'
   Mixer name : 'Realtek ALC887-VD'
   Components : 'HDA:10ec0887,1458a002,00100302'
   Controls : 39
   Simple ctrls : 20
Card1.Amixer.info:
 Card hw:1 'NVidia'/'HDA NVidia at 0xf7080000 irq 17'
   Mixer name : 'Nvidia GPU 42 HDMI/DP'
   Components : 'HDA:10de0042,14583553,00100100'
   Controls : 18
   Simple ctrls : 3
DistroRelease: Ubuntu 12.04
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
MachineType: Gigabyte Technology Co., Ltd. To be filled by O.E.M.
MarkForUpload: True
NonfreeKernelModules: nvidia
Package: linux (not installed)
ProcEnviron:
 LANGUAGE=en_CA:en
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-36-generic-pae root=UUID=4f30252c-fa56-4d53-be78-117cddab55f4 ro quiet splash
ProcVersionSignature: Ubuntu 3.2.0-36.57-generic-pae 3.2.35
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-36-generic-pae N/A
 linux-backports-modules-3.2.0-36-generic-pae N/A
 linux-firmware 1.79.1
RfKill:

StagingDrivers: mei
Tags: precise running-unity staging
Uname: Linux 3.2.0-36-generic-pae i686
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
dmi.bios.date: 08/21/2012
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F9
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: P75-D3
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF9:bd08/21/2012:svnGigabyteTechnologyCo.,Ltd.:pnTobefilledbyO.E.M.:pvrTobefilledbyO.E.M.:rvnGigabyteTechnologyCo.,Ltd.:rnP75-D3:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To be filled by O.E.M.
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1107150

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

apport information

[zaxx (geoff-thebakershome)]

All that collected information is from the working 12.04 installation. I need to reboot to resubmit on the faulted system.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.8 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-rc5-raring/

[zaxx (geoff-thebakershome)]

I have installed 13.04 which runs the 3.8 Kernel and the problem persists. I tried testing as per your suggestion, but could not find a 3.8 kernel that would safely install on 12.10.

Also I cannot get apport-collect to run on 12.10. I think it is due to this bug: https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1023964

[Mark Vels (mark-vels)]

I was hunted by this bug to but observed some behaviour that might help...

I required a i386 12.10 desktop install for some rebuilds. Therefore I installed it on a VirtualBox install on my Intel Core i7-2600 CPU @ 3.40GHz with 32GB RAM. At first it ran fine, but since I was in a hurry doing full rebuilds, I assigned more RAM and cores to the VM instance. Since then I am suffering the same performance degredation.

At first I thought it just had to do with VirtualBox, so I installed the 12.10 i386 desktop natively on a build machine (Intel Core i5-2300 CPU @ 2.80GHz) with 16 GB RAM). After installation it ran fine for a while but then it also began to behave weird after I ran apt-get upgrade this morning.

Since it ran initially fine on the VirtualBox instance, I recalled I changed the config. After restoring the settings to to what I think it used to be, all was fine.

I have measured performance on my VirtualBox instance, varying ONLY the amount of assigned RAM. All other settings have been kept constant. I used time(1) to measure performance of 'sudo apt-get update'. I took 3 measurements, the times are below.

I think it's safe to assume that the amount of RAM is of importance for this bug. I have not been able to test with a 64-bit kernel but it might not be reproducible with that.

Please let me know if you require additional information about one of my systems.

HTH, Mark

Base Memory: 16384 MB
 Reading package lists... 42% <- aborted at this point
 real 9m42.187s
 user 0m7.952s
 sys 0m7.548s

Base Memory: 15360 MB
 real 0m44.337s
 user 0m2.556s
 sys 0m1.096s

Base Memory: 12288 MB
 real 0m10.399s
 user 0m1.192s
 sys 0m0.456s

[Joseph Salisbury (jsalisbury)]

@zaxx,

Is it possible for you to reduce the amout of memory you have to see if you are seeing the same thing as Mark in comment #25?

@Mark, so it appears the threshold is ~15GB of memory when things go bad?

Also, the final v3.8 kernel[0] is now available. Would it be possible to test this kernel to see if this bug also exists upstream?

Thanks in advance!

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-raring/

[mudshark79 (mudshark79)]

@jsalisbury

can confirm the bug with almost identical setup (Z75 Chipset, i7, 32GB, 120GB SATA3 SSD). Reducing the amount of memory from 32 to 16GB helps immediately, didn't try (and won't at this point because 16GB is sufficient at the moment) mainline kernel as being short of time

[zaxx (geoff-thebakershome)]

Sorry for the delay.

I removed most of my ram, leaving 8 GB in the system and booted Ubuntu 12.10. I ran "time sudo apt-get update" and it completed in 17.106 seconds.

@Mark Vels it appears that I owe you one. You nailed it. The Ram size seems to impact the performance of the system.

I am updating to Ubuntu 13.04 to see if the problem is corrected.

[Mark Vels (mark-vels)]

@Zaxx, that's great news. Glad that I could be of help.

@Jalisbury: I need the VM currently too badly for a project to do this testing (and not enough disk space to clone the VM). I expect to deliver really soon, I will check an upstream kernel then to see if it is already fixed and try to gather little more info from strace or friends to try to figure out what it is doing.

[zaxx (geoff-thebakershome)]

I installed the latest 13.04 development and got the following:

real time: 21m52.609s

The problem persists.

output from uname -a:
3.8.0-10-generic

I am running a 32-bit kernel, which means PAE is active. Could the problem be in PAE?

[Joseph Salisbury (jsalisbury)]

Hmm, this may be specific to PAE. Is it possible for someone affected by this to also test the 64 bit kernel?

[Mark Vels (mark-vels)]

I installed 12.10 amd64 on the same pysical machine (Intel Core i5-2300 CPU @ 2.80GHz) with 16 GB RAM) and it runs fine as expected.....

[Mark Vels (mark-vels)]

As promised, I did some testing with the latest kernel:

linux-3.5.0-26-generic: 12 MByte
real 0m10.678s
user 0m1.376s
sys 0m0.348s

linux-3.5.0-26-generic 16 Mbyte
real 11m50.436s
user 0m9.133s
sys 0m7.500s

linux-3.8.4-030804-generic 16 MByte
real 12m25.996s
user 0m11.732s
sys 0m8.388s

I guess we can conclude that it hasn't been fixed yet in 3.8.4 either...

[Joseph Salisbury (jsalisbury)]

@Mark Vels, Would it be possible for you to test the v3.9-rc4 kernel? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.9-rc4-raring/

If the bug still exists in that latest mainline kernel, I can perform a kernel bisect to identify the commit that first introduced this regression.

[Mark Vels (mark-vels)]

mmm no improvement :-(

me@vbox-ubuntu-1210-i386:~$ uname -a
Linux vbox-ubuntu-1210-i386 3.9.0-030900rc4-generic #201303232035 SMP Sun Mar 24 00:44:55 UTC 2013 i686 i686 i686 GNU/Linux
me@vbox-ubuntu-1210-i386:~$ time sudo apt-get update

16GByte
    real 17m36.608s
    user 0m15.096s
    sys 0m11.608s

12GByte
    real 0m10.705s
    user 0m1.748s
    sys 0m0.428s

I have ran strace as 'sudo strace apt-get update' and it shows that the slow-down is really due to abdominal bad kernel I/O performance. Alsmost all the time is actually spend while blocking on the read() system call (which is not much of a surprise I guess if we're already looking at kernel versions.. ) Just wanted to do the exercise anyway..

What's the next step? Going backwards indeed to find out when it was introduced?
Is the VirtualBox VM settings "Extended features: Enable PAE/ NX" relevant?

I'm more of an embedded dev so not too familiar with x86 hw

[Joseph Salisbury (jsalisbury)]

I'd like to perform a bisect to figure out what commit caused this regression. It would be very helpful to know the earliest kernel where the issue started happening as well as the latest kernel that did not have this issue.

Can you test the following kernels and report back? We are looking for the first kernel version that exhibits this bug:

v3.3 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-precise/
v3.4 final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.4-quantal/
v3.5-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.5-rc4-quantal/

You don't have to test every kernel, just up until the kernel that first has this bug.

One thing to note, you will need to install both the linux-image and linux-image-extra .deb packages.

Thanks in advance!

[Mark Vels (mark-vels)]

Hi,

The first one tried (3.3) is already bad news..

Linux vbox-ubuntu-1210-i386 3.3.0-030300-generic-pae #201203182135 SMP Mon Mar 19 01:50:11 UTC 2012 i686 i686 i686 GNU/Linux
12 GByte
 real 0m9.863s
 user 0m1.552s
 sys 0m0.452s

16 gByte
 real 19m27.304s
 user 0m8.849s
 sys 0m20.501s

Please note that there wasn't a linux-image-extra packge available for 3.3..

SInce this is the oldest one, I didn't bother trying the others....

[Joseph Salisbury (jsalisbury)]

Can you also test the following 3.2 kernel:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.41-precise/

[Mark Vels (mark-vels)]

Yah, finally one that works...

Linux vbox-ubuntu-1210-i386 3.2.41-030241-generic-pae #201303201717 SMP Wed Mar 20 21:38:04 UTC 2013 i686 i686 i686 GNU/Linux

time sudo apt-get update
12 GByte
 real 0m6.692s
 user 0m1.524s
 sys 0m0.444s

16 GByte
 real 0m8.030s
 user 0m1.144s
 sys 0m0.380s

me@vbox-ubuntu-1210-i386:~$ cat /proc/meminfo
MemTotal: 16431944 kB
MemFree: 15976260 kB
Buffers: 62232 kB
Cached: 258716 kB
SwapCached: 0 kB
Active: 107812 kB
Inactive: 266648 kB
Active(anon): 59880 kB
Inactive(anon): 356 kB
Active(file): 47932 kB
Inactive(file): 266292 kB
Unevictable: 31196 kB
Mlocked: 31196 kB
HighTotal: 15863752 kB
HighFree: 15510228 kB
LowTotal: 568192 kB
LowFree: 466032 kB
SwapTotal: 8386556 kB
SwapFree: 8386556 kB
Dirty: 140 kB
Writeback: 0 kB
AnonPages: 84768 kB
Mapped: 40268 kB
Shmem: 940 kB
Slab: 24124 kB
SReclaimable: 10556 kB
SUnreclaim: 13568 kB
KernelStack: 2240 kB
PageTables: 2788 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 16602528 kB
Committed_AS: 844892 kB
VmallocTotal: 122880 kB
VmallocUsed: 4440 kB
VmallocChunk: 118140 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 12280 kB
DirectMap2M: 901120 kB

[Joseph Salisbury (jsalisbury)]

So it looks like the regression was introduced in v3.3.

Can you test the following kernels and report back? We are looking for the earliest kernel version that has this bug:

v3.3-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-rc4-precise

If v3.3-rc4 exhibits the bug then test v3.3-rc6:
v3.3-rc6: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-rc6-precise

If v3.3-rc4 does not exhibit the bug then test v3.3-rc2:
v3.3-rc2: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-rc2-precise

[Mark Vels (mark-vels)]

I think you want me to test the other way... If rc4 has it, we should take earlier versions, right?

Anyway, based on my test results, it seems that the regression was introduced between v3.2 and v3.3-rc1.

A simple grep on 'PAE' results in only one commit ae5cd8 but I'm not familiar enough with x86 to assess if this changes is actually relevant.

Can we do more bisects using snapshots?

Grtz, Mark
8<-------------------------------------------------------------------------

Linux vbox-ubuntu-1210-i386 3.3.0-030300rc4-generic-pae #201202181935 SMP Sun Feb 19 00:53:06 UTC 2012 i686 i686 i686 GNU/Linux
16GByte:
    real 14m57.164s
    user 0m10.613s
    sys 0m7.040s

Linux vbox-ubuntu-1210-i386 3.3.0-030300rc6-generic-pae #201203032235 SMP Sun Mar 4 03:52:38 UTC 2012 i686 i686 i686 GNU/Linux
16GByte
    real 15m54.399s
    user 0m12.837s
    sys 0m9.829s

Linux vbox-ubuntu-1210-i386 3.3.0-030300rc2-generic-pae #201201311735 SMP Tue Jan 31 22:56:13 UTC 2012 i686 i686 i686 GNU/Linux
16GByte
    real 13m44.685s
    user 0m12.553s
    sys 0m7.432s

Linux vbox-ubuntu-1210-i386 3.3.0-030300rc1-generic-pae #201201191835 SMP Thu Jan 19 23:51:25 UTC 2012 i686 i686 i686 GNU/Linux
16GByte
    real 14m56.419s
    user 0m13.861s
    sys 0m11.057s

[Joseph Salisbury (jsalisbury)]

Thanks for all the testing. So it looks like the regression was introduced in a commit between v3.2 final and v3.3-rc1. I'll start a kernel bisect between those two versions and post a test kernel shortly. The bisect process will require testing 7 - 10 kernels, but it should identify the commit that introduced this regression.

[Joseph Salisbury (jsalisbury)]

I started a kernel bisect between v3.2 final and v3.3-rc1.

I built the first test kernel, up to the following commit:
2ac9d7aaccbd598b5bd19ac40761b723bb675442

The test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

One thing to note, you will need to install both the linux-image and linux-image-extra .deb packages.

Thanks in advance

[Mark Vels (mark-vels)]

Hi Joseph,

Trying to install your kernel but notice there is only a linux-headers*-all.deb and a linux-headers*-amd64.deb but no i386.
Is that okay?

[Mark Vels (mark-vels)]

Mmm the linux-image and linux-image-extra are missing for i386 too..

Can you have a look, please?

Cheers

[Joseph Salisbury (jsalisbury)]

Thanks for pointing that out. I'll investigate why the 32 bit kernels didn't build.

[Mark Vels (mark-vels)]

Thanks, can't test without :-(

[Joseph Salisbury (jsalisbury)]

I re-built the first test kernel, up to the following commit:
2ac9d7aaccbd598b5bd19ac40761b723bb675442

The i386 test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[Mark Vels (mark-vels)]

This one works fine:

Linux vbox-ubuntu-1210-i386 3.8.0-18-generic #28 SMP Thu Apr 11 20:44:45 UTC 2013 i686 i686 i686 GNU/Linux

16 GByte
    real 0m7.909s
    user 0m1.628s
    sys 0m0.484s

[Mark Vels (mark-vels)]

Hi Joseph,

Can you give me another build to test?

If not, can you please give me some pointers on how to do a build myself? We've come too far to not finish this one.

Thanks in advance,

Mark

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
a429638cac1e5c656818a45aaff78df7b743004e

The test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[Mark Vels (mark-vels)]

This one seems fine:

me@vbox-ubuntu-1210-i386:~$ uname -a
Linux vbox-ubuntu-1210-i386 3.2.0-030200-generic #201304191246 SMP Tue Apr 23 20:11:17 UTC 2013 i686 i686 i686 GNU/Linux

sudo time apt-get update
    1.37user
    0.34system
    0:05.11elapsed

33%CPU (0avgtext+0avgdata 25244maxresident)k
123480inputs+104624outputs (26major+18307minor)pagefaults 0swaps

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
16008d641670571ff4cd750b416c7caf2d89f467

The test kernel can be downloaded from:
http://people.canonical.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[Mark Vels (mark-vels)]

This one seems fine too...

Linux vbox-ubuntu-1210-i386 3.2.0-030200-generic #201304291319 SMP Mon Apr 29 17:46:37 UTC 2013 i686 i686 i686 GNU/Linux

sudo time apt-get update
    1.78user
    0.53system
    0:08.61elapsed

26%CPU (0avgtext+0avgdata 25240maxresident)k
117504inputs+115800outputs (23major+22768minor)pagefaults 0swaps

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
2bd43341217b6e8b75e382243328f458ac67fcbe

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.
http://kernel.ubuntu.com/~jsalisbury/lp1107150/

[Mark Vels (mark-vels)]

This one's fine...

uname -a
Linux vbox-ubuntu-1210-i386 3.2.0-030200-generic #201305021257 SMP Thu May 2 17:06:29 UTC 2013 i686 i686 i686 GNU/Linux

sudo time apt-get update
    1.28 user
    0.43 system
    0:04.36 elapsed
39%CPU (0avgtext+0avgdata 25260maxresident)k
126776inputs+95224outputs (23major+17131minor)pagefaults 0swaps

[Joseph Salisbury (jsalisbury)]

Sorry for the delay, I was traveling last week.

I built the next test kernel, up to the following commit:
6a488979f574cb4287880db2dbc8b13cee30c5be

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.
http://kernel.ubuntu.com/~jsalisbury/lp1107150/

[Mark Vels (mark-vels)]

On 10-05-13 22:11, Joseph Salisbury wrote:
> Sorry for the delay, I was traveling last week.
>
> I built the next test kernel, up to the following commit:
> 6a488979f574cb4287880db2dbc8b13cee30c5be
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1107150
>
> Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.
> http://kernel.ubuntu.com/~jsalisbury/lp1107150/
>
Hi Joseph,

Sorry for my late reply. I have been off the grid as well for a couple of days.

I'm having some trouble installing the linux-image due to a clash on some module. Will need to downgrade first; remove
the old 3.2 kernel and reinstall but don't have the time right now. Hope to be able to pick this up end of the week.

Best regards,

Mark

[Mark Vels (mark-vels)]

uname -a
Linux vbox-ubuntu-1210-i386 3.2.0-030200-generic #201305101230 SMP Fri May 10 17:42:27 UTC 2013 i686 i686 i686 GNU/Linux

time sudo apt-get update
    real 0m6.518s
    user 0m1.124s
    sys 0m0.232s

BR, Mark

[Joseph Salisbury (jsalisbury)]

Sorry for the delay. I ran into some build issues with the latest commit the bisect suggested.

Before continuing the bisect, can you test the latest mainline kernel which is v3.10-rc6:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.10-rc6-saucy/

[Joseph Salisbury (jsalisbury)]

At this point it would be best to test v3.11-rc2 instead of v3.10-rc6:
 http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.11-rc2-saucy/

[avdd (avdd)]

I was having the problem originally described: very slow 'reading package lists' in 'apt-get update'

Using 32 bit kernel 3.5.0-39-generic on latest thinkpad X230 with 16 GB RAM. Samsung 840 pro.

Either of two things made the problem go away:

 - reduce RAM to 8 GB
 - downgrade to kernel 3.2.0-52-generic-pae

Didn't try the available 3.8 package as the vbox drivers had previously failed to compile so I removed it.

[Someone (s4910321931-deactivatedaccount)]

The problem is still an issue on i686 PAE kernel in raring.
Got a Lenovo W520 Laptop with 24GiB of RAM. "apt-get update" takes several minutes.
However I found a workaround as alternative to removing the memory physically.
Add "mem=0x200000000" kernel boot parameter in /etc/default/grub in run update-grub.
This will limit memory usage to first 8GiB.
After that even the boot became significantly faster.
I found that without the workaround, the value in /proc/sys/vm/dirty_bytes is always zero.
I believe the write caching becomes completely disabled in PAE kernel running in machine with a lot of RAM.

[Michisteiner (michisteiner)]

FYI: the same problem seems also still present in Saucy / kernel 3.11.0-12. Without lowering considerably the RAM at boot-time (e.g., 12GB instead of built-in 16GB) I/O write performance is horrible (and even with 12GB it does not seem right and deteriorates over time ...)

[Joseph Salisbury (jsalisbury)]

@Michisteiner,

Can you confirm the issue also happens in the final 3.12 version:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.12-saucy/

[Michisteiner (michisteiner)]

@jsalisbury,

The reference you give does not seem to have PAE kernels; at least there are no packages with -pae in the name. Did packages got unified 32-bit/PAE/64-bit with 3.12?

[Benjamin Leppard (7ul3h2nks)]

@jsalisbury,

I can verify both that the i386 kernels at the referenced PPA are PAE enabled and that Linux kernel 3.12.0 still exhibits the bug.

[Joseph Salisbury (jsalisbury)]

Thanks for the feedback, everyone. I restarted the kernel bisect.

I built the next test kernel, up to the following commit:
7c9c684160bc2c6668abbd2701b440e18bb9ef35

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[Benjamin Leppard (7ul3h2nks)]

That kernel doesn't have the bug.

No significant difference between the 8 GB vs 16 GB case. On my machine with kernels that experience the bug, reading package lists can go into the double digits on kernels that experience the bug.

# uname -a
Linux mediaserver 3.2.0-030200-generic #201311141219 SMP Thu Nov 14 17:55:52 UTC 2013 i686 i686 i686 GNU/Linux

8 GB:
# time apt-get update
Reading package lists... Done

real 0m53.971s
user 0m15.769s
sys 0m0.604s

16 GB:
Reading package lists... Done

real 0m53.954s
user 0m16.281s
sys 0m0.736s

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
010e646ba2fdfc558048a97da746381c35836280

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[Stijn Volckaert (stijn-volckaert)]

Great to finally see this bug being worked on... I found a bunch of similar bug reports that were finally closed (and supposedly solved) because the original bug reporter ended up installing a 64-bit kernel instead.

Here's what I found out so far:
* This bug only affects writing to block devices. Read speeds are absolutely fine. It would be nice if the other people could confirm this!
* This bug isn't related to the B75 chipset. I'm also experiencing this bug on a somewhat older chipset. I believe I'm using an Intel Q57 Express chipset but I doubt whether this is really relevant.
* It will only trigger with the x86 kernel, not the amd64 kernel.
* This bug _probably_ isn't limited to Ubuntu, Ubuntu is just the only distro that triggers it. I tested the following distributions:

  + Fedora 19 (32 bit) live cd (3.9.x kernel) -> works fine
  + OpenSuSe 13.1 (32 bit) live cd (3.11 kernel) -> works fine
  + Ubuntu 13.10 (32 bit) live cd (3.11 kernel) -> doesn't work
  + Ubuntu 13.10 (64 bit) live cd (3.11 kernel) -> works fine
  + Linux Mint 16 (32 bit) live cd (3.11 kernel) -> doesn't work
  + Debian 7.2 (32 bit) live cd (3.2 kernel) -> works fine

I then went back to Ubuntu 12.10 32 bit (3.5.5 kernel) and confirmed that:

  + Running with only 8 Gigs of RAM gives me write speeds of 180-200Mb/sec
  + Running with 16 Gigs of RAM gives me write speeds of 160-180Mb/sec
  + Running with 32 Gigs of RAM gives me write speeds of 1.5-5.5Mb/sec

I downloaded a stock 3.5.5 kernel tarball and compiled it with the ubuntu kernel config. The problem persists.

There is a similar bug where pc's suddenly get terrible write speeds to block devices after waking up from suspend. This bug is caused by the fact that on some pc's MTRR registers change after waking up. After comparing Ubuntu, Fedora and Suse kernel configs, I noticed that Ubuntu is the only distro using the MTRR sanitizer. I disabled it but the problem persists.

This definitely seems like an mm problem to me. I'd like to test the latest test kernel you've built but it's only available for amd64 (which never had the problem anyway).

[Benjamin Leppard (7ul3h2nks)]

Yes, as Stjin says, the provided URL doesn't contain an x86 kernel.

[Joseph Salisbury (jsalisbury)]

I skipped that commit since the x86 kernel would not build.

I built the next test kernel, up to the following commit:
747465ef7a082033e086dedc8189febfda43b015

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[Stijn Volckaert (stijn-volckaert)]

just tested. The x86 kernel still has the bug.

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
1886e5d2c694e7fb59434c717e704e7fd8475d2e

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[Stijn Volckaert (stijn-volckaert)]

still broken

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
412662d204eca981458156fd64d9d5f3b533d7b6

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[Stijn Volckaert (stijn-volckaert)]

not fixed

[Joseph Salisbury (jsalisbury)]

I built the next test kernel, up to the following commit:
252c3d84ed398b090ac2dace46fc6faa6cfaea99

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1107150

Can you test that kernel and report back if it has the bug or not. I will build the next test kernel based on your test results.

[Stijn Volckaert (stijn-volckaert)]

not fixed

[Joseph Salisbury (jsalisbury)]

The bisect identified the following commit:

commit 252c3d84ed398b090ac2dace46fc6faa6cfaea99
Author: RongQing.Li <email address hidden>
Date: Thu Jan 12 22:33:46 2012 +0000

    ipv6: release idev when ip6_neigh_lookup failed in icmp6_dst_alloc

I'm not sure if this is the real root cause. However, I built a mainline test kernel with this commit reverted. Can you give this kernel a test to see if it still exhibits the bug:

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1107150

[Stijn Volckaert (stijn-volckaert)]

This kernel has some other problems. it doesn't seem to contain the graphics driver for my card and I can't reboot. However, over ssh I get:

stijn@rogue:~$ dd if=/dev/zero of=/home/stijn/test bs=1M count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 6.3337 s, 170 MB/s

This is most definitely fixed. Care to upload the source .deb file as well?

Thanks

[Stijn Volckaert (stijn-volckaert)]

I reverted that same commit in my own kernel source and the problem is back. This wasn't the root cause.

[penalvch (penalvch)]

Stijn Volckaert, so your hardware may be tracked, could you please file a new report by executing the following in a terminal while booted into a Ubuntu repository kernel (not a mainline one) via:
ubuntu-bug linux

For more on this, please read the official Ubuntu documentation:
Ubuntu Bug Control and Ubuntu Bug Squad: https://wiki.ubuntu.com/Bugs/BestPractices#X.2BAC8-Reporting.Focus_on_One_Issue
Ubuntu Kernel Team: https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Filing_Kernel_Bug_reports
Ubuntu Community: https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette

When opening up the new report, please feel free to subscribe me to it.

Thank you for your understanding.

[penalvch (penalvch)]

zaxx, as per http://www.gigabyte.us/products/product-page.aspx?pid=4152#bios an update is available for your BIOS (F11). If you update to this following https://help.ubuntu.com/community/BiosUpdate , does it change anything? If it doesn't, could you please both specify what happened, and just provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

Please note your current BIOS is already in the Bug Description, so posting this on the old BIOS would not be helpful.

For more on BIOS updates and linux, please see https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette .

Thank you for your understanding.

[Michisteiner (michisteiner)]

Chris, this problem occurs for lots of different hardware and BIOSes (e.g., i have it on a ThinkPad W530 with uptodate BIOS); the bios-outdated tag is rather misleading and is very unlikely to be related to the problem?

[penalvch (penalvch)]

Michisteiner, we don't do "catch alls" here, as the root cause tends not to be the same. Despite this, if you are having a problem in Ubuntu, you are welcome to file a new report with Ubuntu by executing the following in a terminal while booted into a Ubuntu repository kernel (not a mainline one) via:
ubuntu-bug linux

For more on this, please read the official Ubuntu documentation:
Ubuntu Bug Control and Ubuntu Bug Squad: https://wiki.ubuntu.com/Bugs/BestPractices#X.2BAC8-Reporting.Focus_on_One_Issue
Ubuntu Kernel Team: https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Filing_Kernel_Bug_reports
Ubuntu Community: https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette

When opening up the new report, please feel free to subscribe me to it.

Thank you for your understanding.

Helpful bug reporting tips:
https://wiki.ubuntu.com/ReportingBugs

[Michisteiner (michisteiner)]

Chris. Ok, so i opened a new Bug #1290337. Note though that i follow this bug since several months and much of what was discussed here (baring the outdated BIOS hypothesis) seemed to match what i saw on my machine. So i am pretty convinced ``my'' bug is a duplicate of what zaxx was originally reporting ...

[Ralf Hildebrandt (ralf-hildebrandt)]

can confirm the bug with almost identical setup:

32GB RAM

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Q87 Express LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04)
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 04)

Reducing the amount of memory from 32 to 16GB helps immediately, didn't try (and won't at this point because 16GB is sufficient at the moment) mainline kernel as being short of time.

64 Bit kernel does NOT expose the problem!

[penalvch (penalvch)]

Ralf Hildebrandt, thank you for your comment. So your hardware and problem may be tracked, could you please file a new report with Ubuntu by executing the following in a terminal while booted into a Ubuntu repository kernel (not a mainline one) via:
ubuntu-bug linux

For more on this, please read the official Ubuntu documentation:
Ubuntu Bug Control and Ubuntu Bug Squad: https://wiki.ubuntu.com/Bugs/BestPractices#X.2BAC8-Reporting.Focus_on_One_Issue
Ubuntu Kernel Team: https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Filing_Kernel_Bug_reports
Ubuntu Community: https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette

When opening up the new report, please feel free to subscribe me to it.

Thank you for your understanding.

Helpful bug reporting tips:
https://wiki.ubuntu.com/ReportingBugs

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]