WADL cache durations fit poorly with daily iteration of API

Bug #714621 reported by Leonard Richardson on 2011-02-07
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Unassigned
launchpadlib
Undecided
Unassigned

Bug Description

The WADL and JSON representations of the service root are cached for a week. This made sense when we deployed Launchpad to production once a month. I don't think it makes sense anymore. Look in bug 316694 for people wanting to take advantage of a new feature immediately and having to manually invalidate their WADL caches to access it.

Back when we had an 'edge' server, the cache time on edge was more like one day. I believe the cache time on staging may currently be one day. This would be a good value for production.

Another problem is that the ETag for the WADL incorporates the current revision number of Launchpad, even if the web service hasn't changed since the previous version. This was not a problem earlier, since you'd get a month's worth of changes at a time, going from version N to version N+150. There was probably a web service change in there somewhere, and if not, you got the WADL unnecessarily but it only happened once a month. Now we go from version N to version N+1, and it's relatively unlikely that that version includes a change to a web service. If we reduce the cache time to one day, users may end up unneccessarily getting the WADL once a day.

This isn't a huge problem since the compressed WADL is about 150k, but it's bad enough to be an annoyance. We should reconsider how we calculate the web service ETags. Unfortunately, at the moment I don't have any ideas.

On ETAG, can we not just drop the revno and use the resulting content hash?

As for duration, I suggest upping the cache time to one year and
adding must-revalidate. If we also move wadl out of the appserver
(which we should, its static), then these things all combined will
result in:
 - immediate detection of new WADL
 - 1 conditional request on every launchpadlib new session - but that
will be approximately free
 - up to a years freshness when we are not changing the WADL.

Martin Pool (mbp) wrote :

On 8 February 2011 07:40, Robert Collins <email address hidden> wrote:
> On ETAG, can we not just drop the revno and use the resulting content
> hash?
>
> As for duration, I suggest upping the cache time to one year and
> adding must-revalidate.

+1

> If we also move wadl out of the appserver
> (which we should, its static), then these things all combined will
> result in:
>  - immediate detection of new WADL
>  - 1 conditional request on every launchpadlib new session - but that
> will be approximately free
>  - up to a years freshness when we are not changing the WADL.

Also, moving it out of the app server would let us more easy set a
last-modified date and not worry about the etag.

Doing this would probably also fix bug 607961, which is that wadl
generation is very slow/expensive. Conversely, if clients fetch the
wadl every time without that bug being fixed, it will slow them down
(30s startup delay) and perhaps load the server.

tags: added: nodowntime
summary: - Reevaluate our WADL caching strategy under continuous deployment
+ WADL cache durations fit poorly with daily iteration of API
Robert Collins (lifeless) wrote :

So this appears to be a launchpadlib issue to users; I've added an invalid task to have it turn up in the dup detector.

Changed in launchpadlib:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers