request_id middleware uses wrong request ID value

Bug #1289696 reported by Chris Buccella
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Chris Buccella
neutron
Fix Released
High
Akihiro Motoki
oslo-incubator
Fix Released
High
Chris Buccella

Bug Description

The request_id middleware is designed to generate a request ID during process_request() and attach this value to the as an HTTP header during process_response(). Unfortunately, it stores the request ID value in a variable within the RequestIdMiddleware class. This violates the "shared nothing" rule, and can cause problems when several requests are run concurrently. For example, if requests A and B come in back-to-back, and A completes first, A will have B's request ID value in the HTTP response.

This problem was discovered when running nova's api.compute.servers.test_instance_actions test in parallel while working on https://review.openstack.org/#/c/66903/

Changed in oslo:
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/79358

Changed in oslo:
assignee: nobody → Chris Buccella (chris-buccella)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/80482

Changed in cinder:
assignee: nobody → Chris Buccella (chris-buccella)
status: New → In Progress
Akihiro Motoki (amotoki)
Changed in neutron:
status: New → Confirmed
assignee: nobody → Akihiro Motoki (amotoki)
importance: Undecided → High
Akihiro Motoki (amotoki)
Changed in neutron:
milestone: none → icehouse-rc1
Changed in oslo:
assignee: Chris Buccella (chris-buccella) → Ben Nemec (bnemec)
Ben Nemec (bnemec)
Changed in oslo:
assignee: Ben Nemec (bnemec) → Chris Buccella (chris-buccella)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/79358
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=d7bd9dc37ac3d6bc171cd2e290c772633ad20a32
Submitter: Jenkins
Branch: master

commit d7bd9dc37ac3d6bc171cd2e290c772633ad20a32
Author: Chris Buccella <email address hidden>
Date: Mon Mar 10 14:54:33 2014 +0000

    Don't store the request ID value in middleware as class variable

    The original version of the request_id middleware generates a request ID during
    process_request() and stores the value in a variable, so it can be attached as
    header in process_request(). This is pretty dangerous, since subsequent
    requests will overwrite this value, and the wrong request ID may be attached to
    some responses. This is particularly apparent when requests arrive in parallel.

    This change replaces process_request() and process_response() with an override
    of __call__(). This allows the generated req_id to be used in the response
    header without needing to save it as a class-level variable.

    Change-Id: I498859140ba113412c9d95d09f0c4c6b8650749c
    Closes-bug: #1289696

Changed in oslo:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/83080

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/83195

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/83080
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3ec30fdb4b175f2dc65261f672372bdb44dce3f9
Submitter: Jenkins
Branch: master

commit 3ec30fdb4b175f2dc65261f672372bdb44dce3f9
Author: Akihiro Motoki <email address hidden>
Date: Wed Mar 26 23:28:49 2014 +0900

    Import request_id middleware bug fix from oslo

    There is a bug in request-id middleware that a subsequent API request will
    overwrite a request-id of a previous request when multiple API calls are
    processed in parallel in request_id middleware.
    This commit imports the fix in oslo.

    cherry-picked from oslo-incubator d7bd9dc37ac3d6bc171cd2e290c772633ad20a32
    Closes-Bug: #1289696
    ----
    Don't store the request ID value in middleware as class variable

    The original version of the request_id middleware generates a request ID during
    process_request() and stores the value in a variable, so it can be attached as
    header in process_request(). This is pretty dangerous, since subsequent
    requests will overwrite this value, and the wrong request ID may be attached to
    some responses. This is particularly apparent when requests arrive in parallel.

    This change replaces process_request() and process_response() with an override
    of __call__(). This allows the generated req_id to be used in the response
    header without needing to save it as a class-level variable.
    ----

    Change-Id: Ic20047f604394f1ce4900c3b6a00595223804cba

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/83195
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b5f7cf989c6f233fb2b0dd759cc33afb94761ec2
Submitter: Jenkins
Branch: master

commit b5f7cf989c6f233fb2b0dd759cc33afb94761ec2
Author: Chris Buccella <email address hidden>
Date: Wed Mar 26 19:48:16 2014 +0000

    Import request_id middleware bug fix from oslo

    There is a bug in request-id middleware that a subsequent API request
    will overwrite a request-id of a previous request when multiple API
    calls are processed in parallel in request_id middleware. The fix is
    a drop-in replacement; cinder code does not need to be modified to
    use the updated request_id middleware.

    oslo change to fix the middleware:
    d7bd9dc37ac3d6bc171cd2e290c772633ad20a32

    Closes-Bug: #1289696
    Change-Id: Ib211318c47681f841bcde04e490ccc2070bdf8de

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in oslo:
milestone: none → icehouse-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in oslo:
milestone: icehouse-rc1 → 2014.1
Thierry Carrez (ttx)
Changed in neutron:
milestone: icehouse-rc1 → 2014.1
Thierry Carrez (ttx)
Changed in cinder:
milestone: none → juno-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.