Brocade Zone Manager does not wait for zoning transactions to complete

[Alex Meade]

It appears the synchronization in cinder/zonemanager/drivers/brocade/brcd_fc_zone_driver.py around 'add_connection' and 'delete_connection' is not sufficient. I believe the transaction on the FC switch is still in progress for a brief time after leaving these functions which can cause issue if another operation occurs within that window.

This was discovered by running tempest volume tests with concurrency of 4 and occurred consistently. It has not occurred while running with concurrency of 1.

LOGS:

2015-03-26 11:59:21.562 27678 ERROR cinder.zonemanager.drivers.brocade.brcd_fc_zone_client_cli [-] Error while running zoning CLI: (command=['zonecreate', '"openstack21000024ff406d51200100a09848f79a",', '"21:00:00:24:ff:40:6d:51;20:01:00:a0:98:48:f7:9a"'] error=trans_begin: there is an outstanding transaction, and you are not the owner of that transaction.).

[Walt Boring (walter-boring)]

Angela, can you take a look at this one please?

[Nagendra Rao Jaladanki (nagendra-jaladanki)]

Walt,

I am, Nagendra Rao, Angela's manager. She is on PTO until end of this
week. So i will have another engineer look into it and update the status.

Thanks,
Nagendra Rao

On Tue, Mar 31, 2015 at 8:24 AM, Walt Boring <email address hidden> wrote:

> Angela, can you take a look at this one please?
>
> ** Changed in: cinder
> Assignee: (unassigned) => Angela Smith (aallen-m)
>
> ** Changed in: cinder
> Importance: Undecided => Medium
>
> ** Changed in: cinder
> Status: New => Triaged
>
> --
> You received this bug notification because you are subscribed to Cinder.
> Matching subscriptions: Nagendra Jaladanki
> https://bugs.launchpad.net/bugs/1438722
>
> Title:
> Brocade Zone Manager does not wait for zoning transactions to complete
>
> Status in Cinder:
> Triaged
>
> Bug description:
> It appears the synchronization in
> cinder/zonemanager/drivers/brocade/brcd_fc_zone_driver.py around
> 'add_connection' and 'delete_connection' is not sufficient. I believe
> the transaction on the FC switch is still in progress for a brief time
> after leaving these functions which can cause issue if another
> operation occurs within that window.
>
> This was discovered by running tempest volume tests with concurrency
> of 4 and occurred consistently. It has not occurred while running with
> concurrency of 1.
>
> LOGS:
>
> 2015-03-26 11:59:21.562 27678 ERROR
> cinder.zonemanager.drivers.brocade.brcd_fc_zone_client_cli [-] Error
> while running zoning CLI: (command=['zonecreate',
> '"openstack21000024ff406d51200100a09848f79a",',
> '"21:00:00:24:ff:40:6d:51;20:01:00:a0:98:48:f7:9a"']
> error=trans_begin: there is an outstanding transaction, and you are
> not the owner of that transaction.).
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cinder/+bug/1438722/+subscriptions
>

[Pradeep Sathasivam (spradeepv)]

Walt,

Could you please share the log files?

Regards,
Pradeep. S

[Walt Boring (walter-boring)]

Alex Meade filed the bug.

Alex can you provide more logs for this one please?

[Angela Smith (aallen-m)]

Alex, We are unable to reproduce this issue in house. We've run tempest volume tests using run_tempest.sh and find no issue with outstanding transaction. If you are able to reproduce, can you provide us with the exact command you are issuing? We only found outstanding transaction issue when there was an existing transaction open in the CLI, which would be the expected behavior.

[Angela Smith (aallen-m)]

Change to invalid. Logs are not provided, unable to reproduce in our environment. If you obtain logs for this, please resubmit and provide the exact CLI used to run tempest tests so that we may attempt to reproduce again in our lab.