The source browser invites spiders destroy site performance with massive amounts of spurious crawling
Bug #898858 reported by
Jean-Paul Calderone
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Twisted Website |
Fix Committed
|
Undecided
|
Unassigned |
Bug Description
For example, http://
I don't think we need to have all the old revisions indexed. Keeping trunk@HEAD up to date in the various indexes is nice and seems sufficient.
To post a comment you must log in.
I see that robots.txt now disallows a bunch of browser and changeset related URLs.
Is this bug open because there is something you think should be changed with regards to spiders destroying site performance?