add robots.txt to varnish exclude

Bug #595156 reported by Greg Coit on 2010-06-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
DEPRECATED Pantheon
Fix Released
Undecided
Greg Coit

Bug Description

Per http://adminter.net/User-Agent.aspx/sogou%20spider, some search engine spiders don't support compression of the robots.txt file. I suggest we add robots.txt file to the install.php, update.php and cron.php excludes in our vlc.

Thomas Bonte (toemaz) wrote :

Instead use the instruction where the Accept-Encoding is removed:
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$" || req.url ~ "robots\.txt") {

This was, robots.txt is still cached by varnish but in plain text

Greg Coit (gregcoit) on 2010-06-21
Changed in pantheon:
status: New → Fix Released
Greg Coit (gregcoit) wrote :

Added Thomas Bonte's suggestion to default.vcl

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers