share manager fails to remove access rules on replicated share

Bug #1566815 reported by Valeriy Ponomaryov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Critical
Rodrigo Barbieri

Bug Description

ZFSonLinux driver CI, that runs replication tests, reports error on following "manila_tempest_tests.tests.api.test_replication.ReplicationTest.test_add_access_rule_create_replica_delete_rule" test:

paste: http://paste.openstack.org/show/493142/

logs: http://logs.openstack.org/22/302122/1/check/gate-manila-tempest-dsvm-postgres-zfsonlinux/9782eb5/logs/screen-m-shr.txt.gz?level=TRACE#_2016-04-06_10_08_36_537

raw:

 [req-d3194166-299e-4886-b978-0e01bc2e56f5 06409bf3a4fb4afcb297d802801f8fc1 69ed53f7aaff46549422b442180a93a1 - - -] Exception during message handling: Resource could not be found.
 Traceback (most recent call last):
   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
     incoming.message))
   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
     return self._do_dispatch(endpoint, method, ctxt, args)
   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
     result = func(ctxt, **new_args)
   File "/opt/stack/new/manila/manila/share/manager.py", line 146, in wrapped
     return f(self, *args, **kwargs)
   File "/opt/stack/new/manila/manila/utils.py", line 616, in wrapper
     return func(self, *args, **kwargs)
   File "/opt/stack/new/manila/manila/share/manager.py", line 138, in wrapped
     return locked_operation(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
     return f(*args, **kwargs)
   File "/opt/stack/new/manila/manila/share/manager.py", line 137, in locked_operation
     return operation(*_args, **_kwargs)
   File "/opt/stack/new/manila/manila/share/manager.py", line 1254, in delete_share_replica
     exc_context.reraise = False
   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
     self.force_reraise()
   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
     six.reraise(self.type_, self.value, self.tb)
   File "/opt/stack/new/manila/manila/share/manager.py", line 1240, in delete_share_replica
     share_server=share_server
   File "/opt/stack/new/manila/manila/share/access.py", line 105, in update_access_rules
     self._remove_access_rules(context, delete_rules, share_instance['id'])
   File "/opt/stack/new/manila/manila/share/access.py", line 174, in _remove_access_rules
     context, rule['id'], share_instance_id)
   File "/opt/stack/new/manila/manila/db/api.py", line 435, in share_instance_access_get
    return IMPL.share_instance_access_get(context, access_id, instance_id)
   File "/opt/stack/new/manila/manila/db/sqlalchemy/api.py", line 164, in wrapper
     return f(*args, **kwargs)
   File "/opt/stack/new/manila/manila/db/sqlalchemy/api.py", line 1782, in share_instance_access_get
     raise exception.NotFound()
 NotFound: Resource could not be found.

description: updated
tags: added: ci replication tempest
Changed in manila:
assignee: nobody → Goutham Pacha Ravi (gouthamr)
Changed in manila:
assignee: Goutham Pacha Ravi (gouthamr) → Rodrigo Barbieri (rodrigo-barbieri2010)
Changed in manila:
status: New → In Progress
Changed in manila:
importance: Undecided → Low
importance: Low → Medium
Revision history for this message
Valeriy Ponomaryov (vponomaryov) wrote :

This appears too often now in "dummy" driver CI job.

Changed in manila:
importance: Medium → High
milestone: none → newton-3
Revision history for this message
Valeriy Ponomaryov (vponomaryov) wrote :
Revision history for this message
Valeriy Ponomaryov (vponomaryov) wrote :
Revision history for this message
Valeriy Ponomaryov (vponomaryov) wrote :
Changed in manila:
importance: High → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/350588

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/350647

Changed in manila:
assignee: Rodrigo Barbieri (rodrigo-barbieri2010) → Valeriy Ponomaryov (vponomaryov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on manila (master)

Change abandoned by Rodrigo Barbieri (<email address hidden>) on branch: master
Review: https://review.openstack.org/350588

Changed in manila:
assignee: Valeriy Ponomaryov (vponomaryov) → Rodrigo Barbieri (rodrigo-barbieri2010)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/350647
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=d878826cf3524e9e2f1d972072c7844934509616
Submitter: Jenkins
Branch: master

commit d878826cf3524e9e2f1d972072c7844934509616
Author: Valeriy Ponomaryov <email address hidden>
Date: Tue Aug 2 20:27:46 2016 +0300

    Fix concurrent usage of update_access method for share instances

    In mitaka, update_access merged with a known concurrency issue.
    This concurrency has been randomly failing in our CIs. This change
    adds a lock mechanism to prevent that, while rules are being removed
    by a thread, a parallel thread adds back or handle the same rules
    that are being removed.

    Also, a late mitaka update_access patch [1] broke share migration access
    rules consistency, thus leaving stale access rule data on share server
    that hosted the share prior to its migration. This patch addresses this
    by preventing the refresh mechanism from adding back rules
    that are removed.

    [1] I0f863cbae4d8af0660114161deda7bf7aa60d71d

    Change-Id: Ief3b15eefc0fc325a2a5418fc7ac2724c315cc21
    Co-Authored-By: Rodrigo Barbieri <email address hidden>
    Co-Authored-By: Goutham Pacha Ravi <email address hidden>
    Closes-Bug: #1566815
    Closes-Bug: #1609414

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/354694

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 3.0.0.0b3

This issue was fixed in the openstack/manila 3.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/mitaka)

Reviewed: https://review.openstack.org/354694
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=49c2f3b8c035bd5c5372c5874363701906fd4bb1
Submitter: Jenkins
Branch: stable/mitaka

commit 49c2f3b8c035bd5c5372c5874363701906fd4bb1
Author: Valeriy Ponomaryov <email address hidden>
Date: Tue Aug 2 20:27:46 2016 +0300

    Fix concurrent usage of update_access method for share instances

    In mitaka, update_access merged with a known concurrency issue.
    This concurrency has been randomly failing in our CIs. This change
    adds a lock mechanism to prevent that, while rules are being removed
    by a thread, a parallel thread adds back or handle the same rules
    that are being removed.

    Also, a late mitaka update_access patch [1] broke share migration access
    rules consistency, thus leaving stale access rule data on share server
    that hosted the share prior to its migration. This patch addresses this
    by preventing the refresh mechanism from adding back rules
    that are removed.

    [1] I0f863cbae4d8af0660114161deda7bf7aa60d71d

    Change-Id: Ief3b15eefc0fc325a2a5418fc7ac2724c315cc21
    Co-Authored-By: Rodrigo Barbieri <email address hidden>
    Co-Authored-By: Goutham Pacha Ravi <email address hidden>
    Closes-Bug: #1566815
    Closes-Bug: #1609414
    (cherry picked from commit d878826cf3524e9e2f1d972072c7844934509616)

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.