Web server cache settings are hurting us

[Andrew Hutchings]

Our cache headers are set so bad that someone visiting the main site or docs a month later will still see the content from a month ago rather than current content.

* expires header is set way too high (year 2037)
* cache-control way too high (315360000 seconds)

There are lots of ways to work around this but they would be hard for us to implement. We need to fix this by using:

* cache-control: 3600; must-revalidate
* expires header set for now+1 hour
* use etags

Also as a side note we should be using deflate (or gzip at a push, browsers support deflate better) for html, css and js (ideally I'd like to use a minimizer too, but we are a long way from that stage).

I can do all this easily in httpd and lighttpd, but have no idea in nginx.

[Andrew Hutchings (linuxjedi)]

ah, sorry, webkit was lying to me, we are using gzip.

[Michael Shadle (mshadle)]

I've changed expires to be 30mins, instead of "max"

I've never had complaints for basic assets such as css/js/etc. for maximum cache expiry - html can be an issue though I suppose since docs are created with that extension.

I can also just make it so .html is treated normally and the rest are still expires max. Typically if you want to force refresh assets like javascript or css, msot people use "cache busting" mechanisms - cheating the system by query string.

Etags shouldn't be required necessarily. nginx doesn't support them natively and "last modified" should work properly.

nginx doesn't natively support deflate either. Is it that important?

If these are that limiting, I suggest we just wind up moving to Apache, but there should be no reason nginx can't fit the bill. I've hosted rpm repos on nginx, dynamic sites and static sites. Typically all with heavy expires headers (at least for assets) - the .html extension seems like the only possible difference or issue here.

[Andrew Hutchings (linuxjedi)]

Excellent, thanks for this. We can have several versions of the docs uploaded per day which can be 'fun' whilst reading cached versions of them ;)

css/js isn't a big problem, I've made most of the needed changes to them, any other changes can just have a filename change for version bump.

Etag and deflate is not really important. It appears to support gzip and I think most modern browsers support that anyway. IIRC only IE6 was a problem with gzip and nobody cares about that version any more (I suspect we don't render in it anyway).

Thanks again for jumping on so fast.