Source code browser page is excruciatingly slow

[Richard Wall]

Takes up to 12s just to load the initial HTML.

I'm sure it used to be quicker than this.

It's important to me because I regularly click on source code links from the API documentation.

{{{
$ time curl -v -s http://twistedmatrix.com/trac/browser > /dev/null
* About to connect() to twistedmatrix.com port 80 (#0)
* Trying 66.35.39.65... connected
* Connected to twistedmatrix.com (66.35.39.65) port 80 (#0)
> GET /trac/browser HTTP/1.1
> User-Agent: curl/7.21.3 (i686-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: twistedmatrix.com
> Accept: */*
>
< HTTP/1.1 200 Ok
< Content-Length: 50309
< Set-Cookie: trac_form_token=295e6e945d1133d28221dd85; Path=/trac
< Set-Cookie: trac_session=68b751d3a65883d137554c80; expires=Wed, 29-Feb-2012 11:38:15 GMT; Path=/trac
< Server: TwistedWeb/10.0.0
< Cache-Control: must-revalidate
< Date: Thu, 01 Dec 2011 11:38:09 GMT
< Content-Type: text/html;charset=utf-8
<
{ [data not shown]
* Connection #0 to host twistedmatrix.com left intact
* Closing connection #0

real 0m6.408s
user 0m0.004s
sys 0m0.008s
richard@aziz:~$ time curl -s http://twistedmatrix.com/trac/browser > /dev/null

real 0m12.487s
user 0m0.008s
sys 0m0.004s
richard@aziz:~$ time curl -s http://twistedmatrix.com/trac/browser > /dev/null

real 0m11.628s
user 0m0.008s
sys 0m0.012s
richard@aziz:~$ time curl -s http://twistedmatrix.com/trac/browser > /dev/null

real 0m9.326s
user 0m0.004s
sys 0m0.008s

}}}

[Glyph Lefkowitz (glyph)]

I don't think this is a new issue with the site. Performance is highly variable, as our hardware is currently pretty overloaded.

$ time curl -s http://twistedmatrix.com/trac/browser > /dev/null

real 0m1.253s
user 0m0.009s
sys 0m0.009s
$ time curl -s http://twistedmatrix.com/trac/browser > /dev/null

real 0m1.350s
user 0m0.009s
sys 0m0.008s
$ time curl -s http://twistedmatrix.com/trac/browser > /dev/null

real 0m1.263s
user 0m0.009s
sys 0m0.008s
$ time curl -s http://twistedmatrix.com/trac/browser > /dev/null

real 0m1.330s
user 0m0.009s
sys 0m0.008s

If you can demonstrate in some more meaningful way that performance has suffered, it might be worth looking into further.

[Richard Wall (richardw)]

Here's a list of known issues:
 * http://trac.edgewall.org/wiki/TracSubversion#KnownIssues

Some other ideas:

Turn off Trac SvnMerge* plugins: it radically improved trac changeset (and source rendering) for us at work:
 * http://trac.edgewall.org/ticket/8459#comment:26

Trac database locking issues might be improved by using Sqlite write ahead log:
 * http://trac.edgewall.org/wiki/PySqlite#TheSQLitelibrary

Trac seems to be behind quite an old (slower?) version of Twisted 10.0.0 whereas twistedmatrix.com/ is served by Twisted 11.1.0
 * http://uptime.netcraft.com/up/graph?site=twistedmatrix.com

[Jean-Paul Calderone (exarkun)]

> Trac database locking issues might be improved by using Sqlite write ahead log:

We're deployed on postgresql.

[Jean-Paul Calderone (exarkun)]

> Turn off Trac SvnMerge* plugins: it radically improved trac changeset (and source rendering) for us at work:

Thanks for investigating this. Unfortunately, the average time of 20 requests to a particular source browser page went from 3.8s to 4.8s after I turned off these options.

[Jean-Paul Calderone (exarkun)]

Here's a graph of one hundred requests for a source browser page. Page load time in seconds on Y.

http://twistedmatrix.com/~exarkun/page-load-times.png

[Jean-Paul Calderone (exarkun)]

All pages served by trac seem to have gotten very slow. This seems to be a more general issue than just the source browser. I filed another ticket with a wider scope. Since I don't think it's likely that we'll do anything to the source browser specifically that speeds up just that, I've marked this a duplicate as the wider scope bug. Perhaps we'll revisit that decision as we learn more, though.