High traffic can result in HTTP 503 errors

[Barry Price]

In a deploy with a (default) 1GB maximum cache size, the cache filled up.

For some reason, nginx wasn't able to garbage-collect in order to make more room.

This left the service responding with HTTP 503 errors instead of the requested content.

[Barry Price (barryprice)]

Working theory is that this was caching sufficiently large objects, and had sufficient simultaneous slow clients connected, that every object in the cache was already being used at the point where it filled up, with no room for any more, but no candidates for deletion.

Lots of "ignore long locked inactive cache entry" in the nginx logs, but nothing else of interest - and no sign of nginx process restarts or segfaults.

[Barry Price (barryprice)]

I've reproduced this on staging via parallel requests for a large object combined with a small cache limit.

Having read through the docs pretty thoroughly, there appears to be no way, AFAICS, to tell nginx "If the on-disk cache is full, just pass-through the request without attempting to cache it" (coupled with an alert to inform the operator that the cache has problems).

We ought to be able to avoid this scenario with sensibly-large cache size limits, but I wonder whether another layer might make sense here, if we can't configure a single nginx to behave in an appropriate way.

[Barry Price (barryprice)]

The cache size issue may be a red herring here.

The charm configures each haproxy backend (including the local nginx service) with "maxconn 16".

Removing this snippet solves the 503 issue.

The charm needs to be amended so that this is configurable per-backend, with a sensible default if it's omitted..