SLO: range GET returns incorrect Http message when inner SLOs are used

Bug #1296941 reported by Gilles Gaillard
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Critical
Chuck Thier

Bug Description

This happens in the following case:
- swift-1.12.0
- range GET to SLO made of inner SLOs
- http connection is reused (proxy server sends a Keep-Alive)

The following test shows the issue. The target SLO ​in the test is: container: "test-1", name: "1_GB_chunk_1MB_ID". It's made of
two inner SLOs "xx-slo-000001" and "xx-slo-000002".
The first one has 1000 segments (000001 to 001000), the second has 24 (001001 to 001024) . Detailed names are in attached logs.

The test is single threaded, works with connection reuse and performs:
•range GET for bytes [0-1048575]
  X-Trans-Id: tx2684d6e70e094d84a27af-0053301200
  This worked fine.
• range GET for bytes [1048576-2097151]
  X-Trans-Id: txd6c62bf3fb7f41d7b8c7d-0053301200
  Http response + Data is received
•range GET for bytes [2097152-3145727]
  no http response is received
  instead received data for segment 001001, i.e. the first segment in slo-000002

When I look at the swift logs for the second call, I see that for the second transaction, the second storage server (data02), performs a GET to segment 000002 (which is correct), BUT immediately followed by a GET to slo-000002, then to segment 001001 which may explain why the client receives it afterward.

I had a discussion with David G. - who wrote:
So this is a bug - and a pretty weird one too as far as I can tell. I wrote a script to reproduce it:
  https://gist.github.com/dpgoetz/9747591

That runs on a SAIO but you have to lower your minimum segment size with the following in your proxy server conf:
  [filter:slo]
  use = egg:swift#slo
  min_segment_size = 1
He also said that milestone-1.10.0-rc1 should not show this issue.

Revision history for this message
Gilles Gaillard (gillouxgaillard) wrote :
description: updated
description: updated
description: updated
description: updated
Changed in swift:
importance: Undecided → Critical
milestone: none → next-icehouse
Changed in swift:
assignee: nobody → Chuck Thier (cthier)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/82895

Changed in swift:
status: New → In Progress
Changed in swift:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in swift:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in swift:
milestone: 1.13.1-rc1 → 1.13.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.