get_container_info (or our use of it) needs to be smarter in the face of failures

[Tim Burke]

... because currently, Swift can do some pretty stupid things: http://paste.openstack.org/show/794669/

Container quota middleware tries to get container info, but the container's overloaded. Primaries are error limited, so it goes 6 deep into handoffs. Doesn't find anything; 503. That's OK, CQ says, "this will hopefully 404 later".

So we keep going down the pipeline. Versioned writes sees the request, tries to get container info. Container's still overloaded, we still go out to handoffs, we still get a 503. That's OK, VW says, I guess old-style versioning isn't enabled.

So we keep going down the pipeline. Object versioning sees the request, tries to get container info. Container's still overloaded, we still go out to handoffs, we still get a 503. That's OK, OV says, I guess new-style versioning isn't enabled.

So we keep going down the pipeline. Proxy server sees the request, tries to get container info, blah blah blah. *Finally* we can respond with a 503 to the client! After 24 useless requests :-(

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/swift/+/875819

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/swift/+/875819
Committed: https://opendev.org/openstack/swift/commit/b68cc893f74bf61a03439549d1ca5412cfc8d0ff
Submitter: "Zuul (22348)"
Branch: master

commit b68cc893f74bf61a03439549d1ca5412cfc8d0ff
Author: Tim Burke <email address hidden>
Date: Tue Feb 28 23:03:23 2023 -0800

    proxy: Reduce round-trips to memcache and backend on info misses

    Following a memcache restart in a SAIO, I've seen the following happen
    during an object HEAD:

    - etag_quoter wants to get account/container info to decide whether to
      quote-wrap or not
    - account info is a cache miss, so we make a no-auth'ed HEAD to the next
      filter in the pipeline
    - eventually this gets down to ratelimit, which *also* wants to get
      account info
    - still a cache miss, so we make a *separate* HEAD that eventually talks
      to the backend and populates cache
    - ratelimit realizes it can't ratelimit the request and lets the
      original HEAD through to the backend

    There's a related bug about how something similar can happen when the
    backend gets overloaded, but *everything is working* -- we just ought to
    be talking straight to the proxy app.

    Note that there's likely something similar going on with container info,
    but the hardcoded 10% sampling rate makes it harder to see if you're
    monitoring raw metric streams.

    I thought I fixed this in the related change, but no :-/

    Change-Id: I49447c62abf9375541f396f984c91e128b8a05d5
    Related-Change: If9249a42b30e2a2e7c4b0b91f947f24bf891b86f
    Related-Bug: #1883214