Access logging status code rewrites hide on-the-wire HTTP status code

[clayg]

Swift uses the status code 499 when a client disconnects

One of the most common ways we see this happen is clients who like open a new socket every request. When they get an error (maybe a 404) they'll close the socket (without reading the body) and make a new request.

Swift seems to be handling garbage collection in this case and closing sockets, but the access log line just says the client disconnected.

A 499 when a client fails to read the static body of an error message is dramatically different in terms of operational impact than a 499 when a client was reading the body of EC object and suddenly disconnects (or reads to slow, triggering a ChunkWriteTimeout to the client, default 60s - which ALSO logs as a 499)

To better understanding what our clients and clusters are doing - while still preserving all the benefits of the existing access logging behavior - we should include a new optional log template field that can demonstrate the value sent to the client.

This will make it easier to "fix" issues like https://bugs.launchpad.net/swift/+bug/1895691 (i.e. we can look for 499's and 503's with an on-the-wire 2XX)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/755639
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=fb08d477eb7bf5e678b9cd99b44a435842a7dfbf
Submitter: Zuul
Branch: master

commit fb08d477eb7bf5e678b9cd99b44a435842a7dfbf
Author: Clay Gerrard <email address hidden>
Date: Thu Oct 1 14:28:04 2020 -0500

    New proxy logging field for wire status

    Capture the on the wire status code for logging because we change the
    logged status code sometimes.

    Closes-Bug: #1896518
    Change-Id: I27feabe923a6520e983637a9c68a19ec7174a0df

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/swift 2.27.0 release.