Profiling and performance research

[Paul Everitt]

Spend some time loading up some data (perhaps using the existing loader [1]). Do a bit of profiling to see where we are spending our time, particularly on:

- /communities

- /offices/nyc

- /communities/somecommunity

Go ahead into post-research to refactor and speedup, if you feel it is warranted.

[1] bin/loadcontent -r 38 -f src/osi/osi/scripts/sampledata/ src/osi/osi/scripts/sampledata/small-generated.xml.gz

...where 38 is the number of randomized communities you want created. The loadcontent script sucks.

[Tres Seaver (tseaver)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Everitt wrote:
> Public bug reported:
>
> Spend some time loading up some data (perhaps using the existing loader
> [1]). Do a bit of profiling to see where we are spending our time,
> particularly on:
>
> - /communities
>
> - /offices/nyc
>
> - /communities/somecommunity
>
> Go ahead into post-research to refactor and speedup, if you feel it is
> warranted.
>
> [1] bin/loadcontent -r 38 -f src/osi/osi/scripts/sampledata/
> src/osi/osi/scripts/sampledata/small-generated.xml.gz
>
> ...where 38 is the number of randomized communities you want created.
> The loadcontent script sucks.
>
> ** Affects: karl3
> Importance: Medium
> Assignee: Tres Seaver (tseaver)
> Status: New
>
> ** Changed in: karl3
> Importance: Undecided => Medium
>
> ** Changed in: karl3
> Milestone: None => m12
> Assignee: (unassigned) => Tres Seaver (tseaver)

 status inprogress

- --
===================================================================
Tres Seaver [phone] <email address hidden>
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJ9gNW+gerLs4ltQ4RAgTRAJ9DVNYs9+77enTIrpkROixnmL6bfACff0Ug
unUQ5Z/V1brs1cldP2ZPkK4=
=RUTS
-----END PGP SIGNATURE-----

[Tres Seaver (tseaver)]

Hardware
--------

I'm testing on a fresh checkout / buildout on a box with dual-core,
1 Ghz AMD64 CPU::

 $ cat /proc/cpuinfo
 ...
 vendor_id : AuthenticAMD
 cpu family : 15
 model : 107
 model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4400+
 stepping : 2
 cpu MHz : 1000.000
 cache size : 512 KB
 ...

Running 'ab' locally with concurrency 4 across 100 requests, e.g.::

 $ ab -n 1000 -c 4 -A staff1:staff1 http://localhost:6543/communities/

The times I'm reporting are from the two "Time per request" fields in
ab's output: the "raw" number and the mean per request.

Case 1: start_over site with /communities/somecommunity added.
---------------------------------------------------------------

 URL Raw Mean
 =========================== ===== ====
 /communities 57 14

 /offices/nyc 110 28

 /communities/somecommunity 67 16

[Tres Seaver (tseaver)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Case 1: start_over site with /communities/somecommunity added.
- ---------------------------------------------------------------

 URL Raw Mean
 =========================== ===== ====
 /communities 57 14

 /offices/nyc 110 28

 /communities/somecommunity 67 16

Case 2: after running 'bin/loadcontent -r 38 ...'
- --------------------------------------------------

76 new communities added, bringing total objects catalogued to
7406.

 URL Raw Mean
 =========================== ===== ====
 /communities 115 29

 /offices/nyc 98 26

 /communities/somecommunity 65 16

Case 3: after running 'bin/loadcontent -r 300 ...'
- ----------------------------------------------------

600 new communities added, bringing total content objects
catalogued to 62,514.

 URL Raw Mean
 =========================== ===== ====
 /communities 133 33

 /offices/nyc 122 30

 /communities/somecommunity 93 23

Tres.
- --
===================================================================
Tres Seaver [phone] <email address hidden>
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJ918U+gerLs4ltQ4RAkyYAKCls5O0wnkRW2eI2vysZAcN9ZQGNQCdEyPE
Z4AG3UeiN4RrV8aDAt2cAvc=
=ZOzi
-----END PGP SIGNATURE-----

[Paul Everitt (paul-agendaless)]

Rolling over to M13.

[Tres Seaver (tseaver)]

I made chenges to repoze.bfg and repoze.lemonade (since released)
addresing the highest contributors to the time reported by the profiler.