DEPRECATED Pantheon

add robots.txt to varnish exclude

Bug #595156 reported by Greg Coit
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
DEPRECATED Pantheon
Fix Released
Undecided
Greg Coit

Bug Description

Per http://adminter.net/User-Agent.aspx/sogou%20spider, some search engine spiders don't support compression of the robots.txt file. I suggest we add robots.txt file to the install.php, update.php and cron.php excludes in our vlc.

Revision history for this message
Thomas Bonte (toemaz) wrote :

Instead use the instruction where the Accept-Encoding is removed:
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$" || req.url ~ "robots\.txt") {

This was, robots.txt is still cached by varnish but in plain text

Greg Coit (gregcoit)
Changed in pantheon:
status: New → Fix Released
Revision history for this message
Greg Coit (gregcoit) wrote :

Added Thomas Bonte's suggestion to default.vcl

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.