NSX plugin: punctual state synchronization might cause timeouts

Bug #1279497 reported by Salvatore Orlando
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Low
Salvatore Orlando
Havana
Fix Released
Low
Armando Migliaccio

Bug Description

When performing punctual state synchronization (*) the NSX plugin might trigger the infamous eventlet-mysql deadlock since a NSX API call is performed from within a DB transaction.

This should be avoided, especially since the transaction is not needed in this case.

(*) This can be enabled either via a configuration variable or by explicitly selecting the 'status' field in Neutron APIs.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/73033

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/73695

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/73695
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=85ddbde058d8bda0b938eb7a45ef73519a831b3b
Submitter: Jenkins
Branch: master

commit 85ddbde058d8bda0b938eb7a45ef73519a831b3b
Author: armando-migliaccio <email address hidden>
Date: Fri Feb 14 10:59:17 2014 -0800

    Fix request timeout errors during calls to NSX controller

    Sometimes two correlated exception traces are observed in
    the server log for the Neutron Server backed by NSX:
    RequestTimeout (The nsx request has timed out) and
    OperationalError (Lock wait timeout exceeded). This is
    generally described by Guru Salvatore Orlando as the,
    and I quote, the "infamous eventlet-mysql deadlock".

    This patch tries to address the issue by adding a
    cooperative yield in the nsx client code (it’s a good idea
    to call sleep(0) occasionally in any case) and also by
    avoiding the unnecessary spawning of another Greenthread
    within a call that is already executed in Greenthred
    itself.

    Closes-bug: #1267101
    Related-bug: #1279497

    Change-Id: I8e298468fb730f11a66fbd4211121ee7d3e2a548

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/havana)

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/75484

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/havana)

Reviewed: https://review.openstack.org/75484
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c1942623a5ac0be5df83d9511b56bb6009b64e69
Submitter: Jenkins
Branch: stable/havana

commit c1942623a5ac0be5df83d9511b56bb6009b64e69
Author: armando-migliaccio <email address hidden>
Date: Fri Feb 14 10:59:17 2014 -0800

    Fix request timeout errors during calls to NSX controller

    Sometimes two correlated exception traces are observed in
    the server log for the Neutron Server backed by NSX:
    RequestTimeout (The nsx request has timed out) and
    OperationalError (Lock wait timeout exceeded). This is
    generally described by Guru Salvatore Orlando as the,
    and I quote, the "infamous eventlet-mysql deadlock".

    This patch tries to address the issue by adding a
    cooperative yield in the nsx client code (it’s a good idea
    to call sleep(0) occasionally in any case) and also by
    avoiding the unnecessary spawning of another Greenthread
    within a call that is already executed in Greenthred
    itself.

    Closes-bug: #1267101
    Related-bug: #1279497

    Change-Id: I8e298468fb730f11a66fbd4211121ee7d3e2a548
    (cherry picked from commit 85ddbde058d8bda0b938eb7a45ef73519a831b3b)

tags: added: in-stable-havana
Thierry Carrez (ttx)
Changed in neutron:
milestone: icehouse-3 → icehouse-rc1
Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: icehouse-rc1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.