datasource-row-list API raise 400 exception when thread switching occur in update_from_datasource()

Bug #1473924 reported by Rui Chen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
congress
Fix Released
Undecided
Rui Chen

Bug Description

In nova and cinder datasource driver, self.stat is set as {} at the begin of update_from_datasource(), then driver fetch data, translate it and update self.stat, but if the running thread switch before self.stat is updated completely, self.state is empty dict, datasource-row-list API will raise 400 exception because the table can't be found in self.state.

reproduce steps:

1. add pdb break point at the begin of 'get_items' funcation in row_model.py.
2. run command 'congress datasource row list cinder volumes'
3. debug the code step by step slowly, if the datasource polling occur before we execute the code line "if tablename not in service_obj.state:" , error is raised.

error output:

(openstack) congress datasource row list cinder volumes
...
DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://10.250.10.29:1789/v1/data-sources/2e62a13d-ebcf-4362-b121-86c22543704d/tables/volumes/rows -H "User-Agent: python-congressclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}fc31193cd2dc0518a01e8349c48a2424a0e2a36e"
DEBUG: requests.packages.urllib3.connectionpool "GET /v1/data-sources/2e62a13d-ebcf-4362-b121-86c22543704d/tables/volumes/rows HTTP/1.1" 400 116
DEBUG: keystoneclient.session RESP: [400] date: Mon, 13 Jul 2015 09:38:37 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 116 x-openstack-request-id: req-02db330e-c79d-4d7f-8b3b-3df0d3d502b9
RESP BODY: {"error": {"message": "Unknown tablename 'volumes' for datasource 'cinder'", "error_data": 404, "error_code": 404}}

DEBUG: keystoneclient.session Request returned failure status: 400
ERROR: openstack Unknown tablename 'volumes' for datasource 'cinder' (HTTP 400) (Request-ID: req-02db330e-c79d-4d7f-8b3b-3df0d3d502b9)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cliff/app.py", line 314, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/cliff/display.py", line 92, in run
    column_names, data = self.take_action(parsed_args)
  File "/opt/stack/python-congressclient/congressclient/osc/v1/datasource.py", line 189, in take_action
    parsed_args.table)['results']
  File "/opt/stack/python-congressclient/congressclient/v1/client.py", line 137, in list_datasource_rows
    (datasource_name, table_name))
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 170, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 206, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 95, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/common/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 336, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 397, in request
    raise exceptions.from_response(resp, method, url)
BadRequest: Unknown tablename 'volumes' for datasource 'cinder' (HTTP 400) (Request-ID: req-02db330e-c79d-4d7f-8b3b-3df0d3d502b9)
DEBUG: openstackclient.shell clean_up ListDatasourceRows: Unknown tablename 'volumes' for datasource 'cinder' (HTTP 400) (Request-ID: req-02db330e-c79d-4d7f-8b3b-3df0d3d502b9)
Unknown tablename 'volumes' for datasource 'cinder' (HTTP 400) (Request-ID: req-02db330e-c79d-4d7f-8b3b-3df0d3d502b9)

Rui Chen (kiwik-chenrui)
Changed in congress:
assignee: nobody → Rui Chen (kiwik-chenrui)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to congress (master)

Fix proposed to branch: master
Review: https://review.openstack.org/206776

Changed in congress:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/206777

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/206778

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to congress (master)

Reviewed: https://review.openstack.org/206776
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=dd6ed4d1aa4eb241e9d082bdddb9dbb4f3b6a15e
Submitter: Jenkins
Branch: master

commit dd6ed4d1aa4eb241e9d082bdddb9dbb4f3b6a15e
Author: Rui Chen <email address hidden>
Date: Mon Jul 27 20:03:26 2015 +0800

    Add decorator for translating function

    Use decorator to check raw data before retranslating.
    If raw data is same with cached self.raw_state,
    don't translate data, return empty list directly.

    Change-Id: Ibc25a032adfb0323d7eb36f22b8b3b5dbc2a7f12
    Partial-Bug: #1473924

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/206777
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=b50388d38b58cf315e41ed24886e11ef9a0bfcae
Submitter: Jenkins
Branch: master

commit b50388d38b58cf315e41ed24886e11ef9a0bfcae
Author: Rui Chen <email address hidden>
Date: Tue Jul 28 15:58:09 2015 +0800

    Add update_state in DataSourceDriver

    Implements common logic about updating state in
    DataSourceDriver class, and call it in decorator
    @check_raw_data_changed_and_update_state in order
    to help the subclass datasource drivers to update
    self.state automatically when they use the decorator.

    Change-Id: Ib296ad0f32a2cb924e27d036eb0e644e00bfdc16
    Partial-Bug: #1473924

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/206778
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=42602f10ffdb11c6e9935032edb6031b38bb8246
Submitter: Jenkins
Branch: master

commit 42602f10ffdb11c6e9935032edb6031b38bb8246
Author: Rui Chen <email address hidden>
Date: Tue Jul 28 17:04:44 2015 +0800

    Do refactor for neutronv2 driver

    Aims to keep consistence of datasource driver state,
    and avoid the conflict between datasource row list and
    datasource updating state. Do some refactor for
    neutronv2 driver, use a temporary state to collect row data,
    then override the self.state table one by one.

    This is a demo for getting some feedback from other reviewers.
    If the way is approved, same refactor will been implemented on
    all other datasource drivers by following the pattern.

    Change-Id: I74102f35ca16b82419de145a5263eb5d135e82a3
    Partial-Bug: #1473924

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to congress (master)

Fix proposed to branch: master
Review: https://review.openstack.org/218180

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to congress (master)

Reviewed: https://review.openstack.org/218180
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=6c4665fd0e38fcf4de3d339562eab62cd2ef4865
Submitter: Jenkins
Branch: master

commit 6c4665fd0e38fcf4de3d339562eab62cd2ef4865
Author: Rui Chen <email address hidden>
Date: Wed Aug 26 15:52:06 2015 +0800

    Do refactor for datasource drivers

    Aims to keep consistence of datasource driver state,
    and avoid the conflict between datasource row list and
    datasource updating state. Do some refactor for datasource
    drivers, use a temporary state to collect row data,
    then override the self.state table one by one.

    Change-Id: I1344fe8932305491db5d0be6bae70c6861edb1d5
    Partial-Bug: #1473924

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to congress (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/224399

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to congress (stable/liberty)

Reviewed: https://review.openstack.org/224399
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=859584fbc0c742a51d80bc67372b7e36de523c15
Submitter: Jenkins
Branch: stable/liberty

commit 859584fbc0c742a51d80bc67372b7e36de523c15
Author: Rui Chen <email address hidden>
Date: Wed Aug 26 15:52:06 2015 +0800

    Do refactor for datasource drivers

    Aims to keep consistence of datasource driver state,
    and avoid the conflict between datasource row list and
    datasource updating state. Do some refactor for datasource
    drivers, use a temporary state to collect row data,
    then override the self.state table one by one.

    Change-Id: I1344fe8932305491db5d0be6bae70c6861edb1d5
    Partial-Bug: #1473924
    (cherry picked from commit 6c4665fd0e38fcf4de3d339562eab62cd2ef4865)

tags: added: in-stable-liberty
Tim Hinrichs (thinrichs)
Changed in congress:
status: In Progress → Fix Released
Tim Hinrichs (thinrichs)
Changed in congress:
milestone: none → mitaka-1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.