OpenStack Shared File Systems Service (Manila)

Access rules concurrency in share migration

Bug #1623052 reported by Rodrigo Barbieri on 2016-09-13

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Fix Released	Medium	Rodrigo Barbieri

Bug Description

Following error pops up in CI sometimes during Share Migration test:

2016-09-05 02:57:10.648 12352 ERROR manila.share.manager [req-91d4a28b-68d2-4460-8e48-2a3e6cf620d8 04733f1fb0474c58a03db04c7364ae97 813046f2bc9d4ec9bfd6b70bec27cdd7 - - -] Driver migration completion failed for share 66267fc4-713a-4bb3-93f0-38a5db1ec800.
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager Traceback (most recent call last):
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/manager.py", line 1071, in migration_complete
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager dest_share_instance)
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/manager.py", line 1016, in _migration_complete_driver
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager helper.apply_new_access_rules(dest_share_instance)
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/migration.py", line 202, in apply_new_access_rules
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager self.migration_wait_access_rules_timeout)
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager File "/opt/stack/new/manila/manila/utils.py", line 687, in wait_for_access_update
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager raise exception.ShareMigrationFailed(reason=msg)
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager ShareMigrationFailed: Share migration failed: Timeout trying to update access rules on share instance 01772b4a-2f00-412b-b241-24ec71564927. Timeout was 180 seconds.
2016-09-05 02:57:10.648 12352 ERROR manila.share.manager
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server [req-91d4a28b-68d2-4460-8e48-2a3e6cf620d8 04733f1fb0474c58a03db04c7364ae97 813046f2bc9d4ec9bfd6b70bec27cdd7 - - -] Exception during message handling
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 617, in wrapper
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 1086, in migration_complete
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server raise exception.ShareMigrationFailed(reason=msg)
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server ShareMigrationFailed: Share migration failed: Driver migration completion failed for share 66267fc4-713a-4bb3-93f0-38a5db1ec800.
2016-09-05 02:57:10.732 12352 ERROR oslo_messaging.rpc.server

OpenStack Infra (hudson-openstack) on 2016-09-13

Changed in manila:
assignee:	nobody → Rodrigo Barbieri (rodrigo-barbieri2010)
status:	New → In Progress

Rodrigo Barbieri (rodrigo-barbieri2010) on 2016-09-14

Changed in manila:
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-14: Fix merged to manila (master)

Reviewed: https://review.openstack.org/365136
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=10487a14a3673d4a28a4ea85a62619e69d5c669b
Submitter: Jenkins
Branch: master

commit 10487a14a3673d4a28a4ea85a62619e69d5c669b
Author: Rodrigo Barbieri <email address hidden>
Date: Mon Sep 12 16:22:34 2016 -0300

Fix allow/deny error message and race in migration

    Share migration code [1] merged in newton intending to allow admins
    to mount the share between phase1 and phase2, but API code for
    allow_access and deny_access was set incorrectly, blocking it.

    After discussing this feature's purpose further, we decided we do
    not want this feature at this moment, so we are just fixing the
    allow_access and deny_access error messages.

    Also, addressed a small case of concurrency that was happening
    once in a while in CI. Update_access was being invoked while
    other rules were being applied, thus setting the access_rule_state
    to "UPDATING_MULTIPLE", ignoring the migration access rule change
    RPC request completely, failing migration. By refreshing the
    model we are able to assign the proper access_rule_state at the
    time the function is invoked, setting the access_rule_state
    correctly.

[1] If4bfaf7e9d963b83c13a6fea241c2eda14f7f409

APIImpact

Closes-bug: #1623051
Closes-bug: #1623052

Change-Id: I76a7d8c3bdd597b951e700350f8f3f82bfb21e03