Activity log for bug #1328088

Date Who What changed Old value New value Message
2014-06-09 12:41:08 Rafael David Tinoco bug added bug
2014-06-09 12:41:29 Rafael David Tinoco attachment removed AlsaInfo.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128142/+files/AlsaInfo.txt
2014-06-09 12:41:41 Rafael David Tinoco attachment removed BootDmesg.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128143/+files/BootDmesg.txt
2014-06-09 12:41:51 Rafael David Tinoco attachment removed CurrentDmesg.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128144/+files/CurrentDmesg.txt
2014-06-09 12:42:00 Rafael David Tinoco attachment removed IwConfig.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128146/+files/IwConfig.txt
2014-06-09 12:42:11 Rafael David Tinoco attachment removed Lspci.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128147/+files/Lspci.txt
2014-06-09 12:42:59 Rafael David Tinoco summary Kernel network namespace performance regression during kernel development Kernel network namespace performance regression during rcu development on kernels above 3.8
2014-06-09 12:43:08 Rafael David Tinoco attachment removed Dependencies.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128145/+files/Dependencies.txt
2014-06-09 12:43:14 Rafael David Tinoco attachment removed ProcModules.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128152/+files/ProcModules.txt
2014-06-09 12:43:20 Rafael David Tinoco attachment removed ProcInterrupts.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128151/+files/ProcInterrupts.txt
2014-06-09 12:43:42 Rafael David Tinoco description I'll fill this in a few minutes. Just needed the bug # for now. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-29-generic 3.13.0-29.53 ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2 Uname: Linux 3.13.0-29-generic x86_64 NonfreeKernelModules: fglrx ApportVersion: 2.14.1-0ubuntu3.2 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/controlC2: inaddy 28156 F.... pulseaudio /dev/snd/controlC1: inaddy 28156 F.... pulseaudio /dev/snd/controlC0: inaddy 28156 F.... pulseaudio CRDA: Error: [Errno 2] No such file or directory: 'iw' CurrentDesktop: Unity Date: Mon Jun 9 09:30:11 2014 MachineType: To be filled by O.E.M. To be filled by O.E.M. ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=/dev/mapper/dezmil-root ro apparmor=0 RelatedPackageVersions: linux-restricted-modules-3.13.0-29-generic N/A linux-backports-modules-3.13.0-29-generic N/A linux-firmware 1.127.2 RfKill: 0: hci0: Bluetooth Soft blocked: no Hard blocked: no SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) WifiSyslog: dmi.bios.date: 01/11/2013 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1503 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: SABERTOOTH 990FX R2.0 dmi.board.vendor: ASUSTeK COMPUTER INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 3 dmi.chassis.vendor: To Be Filled By O.E.M. dmi.chassis.version: To Be Filled By O.E.M. dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1503:bd01/11/2013:svnTobefilledbyO.E.M.:pnTobefilledbyO.E.M.:pvrTobefilledbyO.E.M.:rvnASUSTeKCOMPUTERINC.:rnSABERTOOTH990FXR2.0:rvrRev1.xx:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.: dmi.product.name: To be filled by O.E.M. dmi.product.version: To be filled by O.E.M. dmi.sys.vendor: To be filled by O.E.M. I'll fill this in a few minutes. Just needed the public bug # for now.
2014-06-09 12:43:46 Rafael David Tinoco linux (Ubuntu): assignee Rafael David Tinoco (inaddy)
2014-06-09 12:43:56 Rafael David Tinoco linux (Ubuntu): status New In Progress
2014-06-09 12:44:11 Rafael David Tinoco attachment removed Lsusb.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128148/+files/Lsusb.txt
2014-06-09 12:44:21 Rafael David Tinoco attachment removed ProcCpuinfo.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128149/+files/ProcCpuinfo.txt
2014-06-09 12:44:29 Rafael David Tinoco attachment removed ProcEnviron.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128150/+files/ProcEnviron.txt
2014-06-09 12:44:38 Rafael David Tinoco attachment removed PulseList.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128153/+files/PulseList.txt
2014-06-09 12:44:47 Rafael David Tinoco attachment removed UdevDb.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128154/+files/UdevDb.txt
2014-06-09 12:44:57 Rafael David Tinoco attachment removed UdevLog.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1328088/+attachment/4128155/+files/UdevLog.txt
2014-06-09 12:45:56 Rafael David Tinoco affects linux (Ubuntu) linux
2014-06-09 13:11:17 Rafael David Tinoco description I'll fill this in a few minutes. Just needed the public bug # for now. Please, follow this in: http://people.canonical.com/~inaddy/lp1328088/. Same description on daily-basis updated text. -- It was brought to my attention that "fake router creation" scalability was affected during kernel development. The following script was used for all the tests and charts generation: http://people.canonical.com/~inaddy/lp1328088/make_fake_routers.sh http://people.canonical.com/~inaddy/lp1328088/parse.py I measured how many "fake routers" (above script) could be added per second from 0 to 4000 created routers mark. Using this script and a git bisect on kernel tree I was led to one specific commit causing regression: #911af505 "rcu: Provide compile-time control for no-CBs CPUs". It appeared that rcu, rcu callbacks and no-cb cpus were causing the issue so every commit that changed any of this files: "kernel/rcutree.c kernel/rcutree.h kernel/rcutree_plugin.h include/trace/events/rcu.h include/linux/rcupdate.h" was tested. The idea was to check performance regression during rcu development. In the worst case I would have data for performance regression during kernel development (since we have rcu commits from 3.8 to 3.14). All text below this refer to 2 groups of charts, generated during the study: 1) Kernel git tags from 3.8 to 3.14. http://people.canonical.com/~inaddy/lp1328088/charts/250-tag.html 2) Kernel git commits for rcu development (111 commits). http://people.canonical.com/~inaddy/lp1328088/charts/250.html Since there was difference in results depending on how many cpus or how the no-cb cpus were configured, 3 kernel config options were used on every measure: - CONFIG_RCU_NOCB_CPU (disabled: nocbno) - CONFIG_RCU_NOCB_CPU_ALL (enabled: nocball) - CONFIG_RCU_NOCB_CPU_NONE (enabled: nocbnone) After charts generation and study it was clear that NOCB_CPU_ALL (4 cpus) affected the "fake routers" creation process performance and this regression continues up to upstream version. It was also clear that, after this commit, there is no scalability executing this test with more than 1 cpu. Comparing standing out points (see charts): #81e5949 - good #911af50 - bad #6faf728 - not good enough I was able to see that from the script above the following lines were affected: 1) ip netns add -> huge performance regression 2) ip netns exec -> some performance regression # # Assumption # rcu callbacks being offloaded to other cpus caused regression in unshare(CLONE_NEWNET) code. # Specific kernel entry being investigated: unshare(CLONE_NEWNET)
2014-06-09 13:18:58 Rafael David Tinoco bug added subscriber Mark Russell
2014-06-09 13:19:07 Rafael David Tinoco bug added subscriber Chris J Arges
2014-06-09 13:19:17 Rafael David Tinoco bug added subscriber Dave Chiluk
2014-06-09 13:19:31 Rafael David Tinoco bug added subscriber Jay Vosburgh
2014-06-10 21:45:26 Chris J Arges bug task added linux (Ubuntu)
2014-06-11 04:56:57 Rafael David Tinoco description Please, follow this in: http://people.canonical.com/~inaddy/lp1328088/. Same description on daily-basis updated text. -- It was brought to my attention that "fake router creation" scalability was affected during kernel development. The following script was used for all the tests and charts generation: http://people.canonical.com/~inaddy/lp1328088/make_fake_routers.sh http://people.canonical.com/~inaddy/lp1328088/parse.py I measured how many "fake routers" (above script) could be added per second from 0 to 4000 created routers mark. Using this script and a git bisect on kernel tree I was led to one specific commit causing regression: #911af505 "rcu: Provide compile-time control for no-CBs CPUs". It appeared that rcu, rcu callbacks and no-cb cpus were causing the issue so every commit that changed any of this files: "kernel/rcutree.c kernel/rcutree.h kernel/rcutree_plugin.h include/trace/events/rcu.h include/linux/rcupdate.h" was tested. The idea was to check performance regression during rcu development. In the worst case I would have data for performance regression during kernel development (since we have rcu commits from 3.8 to 3.14). All text below this refer to 2 groups of charts, generated during the study: 1) Kernel git tags from 3.8 to 3.14. http://people.canonical.com/~inaddy/lp1328088/charts/250-tag.html 2) Kernel git commits for rcu development (111 commits). http://people.canonical.com/~inaddy/lp1328088/charts/250.html Since there was difference in results depending on how many cpus or how the no-cb cpus were configured, 3 kernel config options were used on every measure: - CONFIG_RCU_NOCB_CPU (disabled: nocbno) - CONFIG_RCU_NOCB_CPU_ALL (enabled: nocball) - CONFIG_RCU_NOCB_CPU_NONE (enabled: nocbnone) After charts generation and study it was clear that NOCB_CPU_ALL (4 cpus) affected the "fake routers" creation process performance and this regression continues up to upstream version. It was also clear that, after this commit, there is no scalability executing this test with more than 1 cpu. Comparing standing out points (see charts): #81e5949 - good #911af50 - bad #6faf728 - not good enough I was able to see that from the script above the following lines were affected: 1) ip netns add -> huge performance regression 2) ip netns exec -> some performance regression # # Assumption # rcu callbacks being offloaded to other cpus caused regression in unshare(CLONE_NEWNET) code. # Specific kernel entry being investigated: unshare(CLONE_NEWNET) Please, follow this in: http://people.canonical.com/~inaddy/lp1328088/. Same description on daily-basis updated text. -- It was brought to my attention that network namespace creation scalability was affected during kernel development. The following script was used for all the tests and charts generation: http://people.canonical.com/~inaddy/lp1328088/make_fake_routers.sh http://people.canonical.com/~inaddy/lp1328088/parse.py I measured how many "fake routers" (above script) could be added per second from 0 to 4000 created routers mark. Using this script and a git bisect on kernel tree I was led to one specific commit causing regression: #911af50 "rcu: Provide compile-time control for no-CBs CPUs". Even Though this change was experimental at that point, it introduced a performance scalability regression (explained below) that still last and seems to be the default option for distributions nowadays. RCU related code looked like to be responsible for the problem. With that, every commit from tag v3.8..master that changed any of this files: "kernel/rcutree.c kernel/rcutree.h kernel/rcutree_plugin.h include/trace/events/rcu.h include/linux/rcupdate.h" was tested. The idea was to check performance regression during rcu development. In the worst case, the regression not being related to rcu, I would still have data to interpret the performance/scalability regression. All text below this refer to 2 groups of charts, generated during the study: 1) Kernel git tags from 3.8 to 3.14. http://people.canonical.com/~inaddy/lp1328088/charts/250-tag.html 2) Kernel git commits for rcu development (111 commits). http://people.canonical.com/~inaddy/lp1328088/charts/250.html Since there was difference in results depending on how many cpus or how the no-cb cpus were configured, 3 kernel config options were used on every measure: - CONFIG_RCU_NOCB_CPU (disabled): nocbno - CONFIG_RCU_NOCB_CPU_ALL (enabled): nocball - CONFIG_RCU_NOCB_CPU_NONE (enabled): nocbnone Obs: For 1 cpu cases: nocbno, nocbnone, nocball behaves the same since w/ only 1 cpu there is no no-cb cpu After charts being generated it was clear that NOCB_CPU_ALL (4 cpus) affected the "fake routers" creation process performance and this regression continues up to upstream version. It was also clear that, after commit #911af50, having more than 1 cpu does not improve performance/scalability for netns, makes it worse. #911af50 ... +#ifdef CONFIG_RCU_NOCB_CPU_ALL + pr_info("\tExperimental no-CBs for all CPUs\n"); + cpumask_setall(rcu_nocb_mask); +#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */ ... Comparing standing out points (see charts): #81e5949 - good #911af50 - bad I was able to see that, from the script above, the following lines causes major impact on netns scalability/performance: 1) ip netns add -> huge performance regression: 1 cpu: no regression 4 cpu: regression for NOCB_CPU_ALL obs: regression from 250 netns/sec to 50 netns/sec on 500 netns already created mark 2) ip netns exec -> some performance regression 1 cpu: no regression 4 cpu: regression for NOCB_CPU_ALL obs: regression from 40 netns (+1 exec per netns creation) to 20 netns/sec on 500 netns created mark # Assumption (to be confirmed) rcu callbacks being offloaded to other cpus caused regression in copy_net_ns<-created_new_namespaces or unshare(clone_newnet).
2014-06-11 11:26:37 penalvch tags kernel-bug precise saucy trusty bisect-done kernel-bug precise saucy trusty
2014-08-22 11:29:45 Rafael David Tinoco description Please, follow this in: http://people.canonical.com/~inaddy/lp1328088/. Same description on daily-basis updated text. -- It was brought to my attention that network namespace creation scalability was affected during kernel development. The following script was used for all the tests and charts generation: http://people.canonical.com/~inaddy/lp1328088/make_fake_routers.sh http://people.canonical.com/~inaddy/lp1328088/parse.py I measured how many "fake routers" (above script) could be added per second from 0 to 4000 created routers mark. Using this script and a git bisect on kernel tree I was led to one specific commit causing regression: #911af50 "rcu: Provide compile-time control for no-CBs CPUs". Even Though this change was experimental at that point, it introduced a performance scalability regression (explained below) that still last and seems to be the default option for distributions nowadays. RCU related code looked like to be responsible for the problem. With that, every commit from tag v3.8..master that changed any of this files: "kernel/rcutree.c kernel/rcutree.h kernel/rcutree_plugin.h include/trace/events/rcu.h include/linux/rcupdate.h" was tested. The idea was to check performance regression during rcu development. In the worst case, the regression not being related to rcu, I would still have data to interpret the performance/scalability regression. All text below this refer to 2 groups of charts, generated during the study: 1) Kernel git tags from 3.8 to 3.14. http://people.canonical.com/~inaddy/lp1328088/charts/250-tag.html 2) Kernel git commits for rcu development (111 commits). http://people.canonical.com/~inaddy/lp1328088/charts/250.html Since there was difference in results depending on how many cpus or how the no-cb cpus were configured, 3 kernel config options were used on every measure: - CONFIG_RCU_NOCB_CPU (disabled): nocbno - CONFIG_RCU_NOCB_CPU_ALL (enabled): nocball - CONFIG_RCU_NOCB_CPU_NONE (enabled): nocbnone Obs: For 1 cpu cases: nocbno, nocbnone, nocball behaves the same since w/ only 1 cpu there is no no-cb cpu After charts being generated it was clear that NOCB_CPU_ALL (4 cpus) affected the "fake routers" creation process performance and this regression continues up to upstream version. It was also clear that, after commit #911af50, having more than 1 cpu does not improve performance/scalability for netns, makes it worse. #911af50 ... +#ifdef CONFIG_RCU_NOCB_CPU_ALL + pr_info("\tExperimental no-CBs for all CPUs\n"); + cpumask_setall(rcu_nocb_mask); +#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */ ... Comparing standing out points (see charts): #81e5949 - good #911af50 - bad I was able to see that, from the script above, the following lines causes major impact on netns scalability/performance: 1) ip netns add -> huge performance regression: 1 cpu: no regression 4 cpu: regression for NOCB_CPU_ALL obs: regression from 250 netns/sec to 50 netns/sec on 500 netns already created mark 2) ip netns exec -> some performance regression 1 cpu: no regression 4 cpu: regression for NOCB_CPU_ALL obs: regression from 40 netns (+1 exec per netns creation) to 20 netns/sec on 500 netns created mark # Assumption (to be confirmed) rcu callbacks being offloaded to other cpus caused regression in copy_net_ns<-created_new_namespaces or unshare(clone_newnet). SRU Justification: Impact: network namespace creation has performance regression since v3.5. Fix: my analysis, lklm discussion, upstream patch Testcase: http://people.canonical.com/~inaddy/lp1328088/make_fake_routers.sh http://people.canonical.com/~inaddy/lp1328088/parse.py http://people.canonical.com/~inaddy/lp1328088/charts/250.html http://people.canonical.com/~inaddy/lp1328088/charts/250-tag.html Running make_fake_routers.sh 4000 and using parse.py you can check if "fake routers" are being created in a good rate /sec (and you can compare with all generated charts). ---------------------------- Original Description: Please, follow this in: http://people.canonical.com/~inaddy/lp1328088/. Same description on daily-basis updated text. -- It was brought to my attention that network namespace creation scalability was affected during kernel development. The following script was used for all the tests and charts generation: http://people.canonical.com/~inaddy/lp1328088/make_fake_routers.sh http://people.canonical.com/~inaddy/lp1328088/parse.py I measured how many "fake routers" (above script) could be added per second from 0 to 4000 created routers mark. Using this script and a git bisect on kernel tree I was led to one specific commit causing regression: #911af50 "rcu: Provide compile-time control for no-CBs CPUs". Even Though this change was experimental at that point, it introduced a performance scalability regression (explained below) that still last and seems to be the default option for distributions nowadays. RCU related code looked like to be responsible for the problem. With that, every commit from tag v3.8..master that changed any of this files: "kernel/rcutree.c kernel/rcutree.h kernel/rcutree_plugin.h include/trace/events/rcu.h include/linux/rcupdate.h" was tested. The idea was to check performance regression during rcu development. In the worst case, the regression not being related to rcu, I would still have data to interpret the performance/scalability regression. All text below this refer to 2 groups of charts, generated during the study: 1) Kernel git tags from 3.8 to 3.14. http://people.canonical.com/~inaddy/lp1328088/charts/250-tag.html 2) Kernel git commits for rcu development (111 commits). http://people.canonical.com/~inaddy/lp1328088/charts/250.html Since there was difference in results depending on how many cpus or how the no-cb cpus were configured, 3 kernel config options were used on every measure: - CONFIG_RCU_NOCB_CPU (disabled): nocbno - CONFIG_RCU_NOCB_CPU_ALL (enabled): nocball - CONFIG_RCU_NOCB_CPU_NONE (enabled): nocbnone Obs: For 1 cpu cases: nocbno, nocbnone, nocball behaves the same since w/ only 1 cpu there is no no-cb cpu After charts being generated it was clear that NOCB_CPU_ALL (4 cpus) affected the "fake routers" creation process performance and this regression continues up to upstream version. It was also clear that, after commit #911af50, having more than 1 cpu does not improve performance/scalability for netns, makes it worse. #911af50 ... +#ifdef CONFIG_RCU_NOCB_CPU_ALL + pr_info("\tExperimental no-CBs for all CPUs\n"); + cpumask_setall(rcu_nocb_mask); +#endif /* #ifdef CONFIG_RCU_NOCB_CPU_ALL */ ... Comparing standing out points (see charts): #81e5949 - good #911af50 - bad I was able to see that, from the script above, the following lines causes major impact on netns scalability/performance: 1) ip netns add -> huge performance regression:     1 cpu: no regression     4 cpu: regression for NOCB_CPU_ALL     obs: regression from 250 netns/sec to 50 netns/sec          on 500 netns already created mark 2) ip netns exec -> some performance regression     1 cpu: no regression     4 cpu: regression for NOCB_CPU_ALL     obs: regression from 40 netns (+1 exec per netns          creation) to 20 netns/sec on 500 netns created          mark # Assumption (to be confirmed) rcu callbacks being offloaded to other cpus caused regression in copy_net_ns<-created_new_namespaces or unshare(clone_newnet).
2014-08-22 14:20:40 Tim Gardner linux (Ubuntu): status New Fix Committed
2014-08-22 14:20:55 Tim Gardner nominated for series Ubuntu Trusty
2014-08-22 14:20:55 Tim Gardner bug task added linux (Ubuntu Trusty)
2014-08-22 14:20:55 Tim Gardner nominated for series Ubuntu Utopic
2014-08-22 14:20:55 Tim Gardner bug task added linux (Ubuntu Utopic)
2014-08-22 14:21:30 Tim Gardner linux (Ubuntu Trusty): status New Fix Committed
2014-08-22 14:21:30 Tim Gardner linux (Ubuntu Trusty): assignee Rafael David Tinoco (inaddy)
2014-08-28 18:37:11 Launchpad Janitor linux (Ubuntu Utopic): status Fix Committed Fix Released
2014-09-04 18:32:53 Launchpad Janitor branch linked lp:ubuntu/trusty-proposed/linux-keystone
2014-09-05 15:09:26 Launchpad Janitor branch linked lp:ubuntu/precise-proposed/linux-lts-trusty
2014-09-08 20:29:31 Brad Figg tags bisect-done kernel-bug precise saucy trusty bisect-done kernel-bug precise saucy trusty verification-needed-trusty
2014-09-09 12:37:48 Rafael David Tinoco tags bisect-done kernel-bug precise saucy trusty verification-needed-trusty bisect-done kernel-bug precise saucy trusty verification-done
2014-09-09 12:38:01 Rafael David Tinoco tags bisect-done kernel-bug precise saucy trusty verification-done bisect-done kernel-bug precise saucy trusty verification-done-trusty
2014-09-22 22:45:50 Launchpad Janitor linux (Ubuntu Trusty): status Fix Committed Fix Released
2014-09-22 22:45:50 Launchpad Janitor cve linked 2014-3601
2014-09-22 22:45:50 Launchpad Janitor cve linked 2014-5077
2014-09-22 22:45:50 Launchpad Janitor cve linked 2014-5472
2014-09-22 23:43:49 Launchpad Janitor branch linked lp:ubuntu/precise-security/linux-lts-trusty
2014-10-10 23:25:02 Jorge Niedbalski tags bisect-done kernel-bug precise saucy trusty verification-done-trusty bisect-done cts kernel-bug precise saucy trusty verification-done-trusty
2015-05-02 03:27:53 Rafael David Tinoco linux: assignee Rafael David Tinoco (inaddy)
2015-05-02 03:27:56 Rafael David Tinoco linux (Ubuntu Trusty): assignee Rafael David Tinoco (inaddy)
2015-05-02 03:32:44 Rafael David Tinoco bug task deleted linux
2015-05-13 16:10:57 Rafael David Tinoco linux (Ubuntu): assignee Rafael David Tinoco (inaddy)
2015-10-15 19:59:43 Jorge Niedbalski linux (Ubuntu Trusty): assignee Rafael David Tinoco (inaddy)
2015-10-15 19:59:54 Jorge Niedbalski linux (Ubuntu Utopic): assignee Rafael David Tinoco (inaddy)
2017-06-14 01:39:16 Don Bowman bug added subscriber Don Bowman