OpenStack Shared File Systems Service (Manila)

generic driver does not delete service port with deletion of share server

Bug #1453191 reported by Valeriy Ponomaryov on 2015-05-08

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Fix Released	Medium	Valeriy Ponomaryov	OpenStack Shared File Systems Service (Manila) 1.0.0 "liberty"

Bug Description

Generic driver creates ports in Neutron for each share server but does not delete it with deletion of share servers. It leads to orphaning of ports and fixed ip leak.

Here is error from manila-share service:

http://paste.openstack.org/show/217201/

2015-05-08 18:51:52.46 ERROR manila.network.neutron.api [req-9677492e-ea8d-49f5-ad2f-194178c44b81 ca779e1e8d3f46308b5d2341c7ae3e27 3e255d4c73c34d2bb1a2797cd3e8a4e1] Neutron error creating port on network 31baf6a4-aa9b-442f-9034-8948feacad40
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api Traceback (most recent call last):
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api File "/opt/stack/manila/manila/network/neutron/api.py", line 164, in create_port
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api port = self.client.create_port(port_req_body).get('port', {})
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api ret = self.function(instance, *args, **kwargs)
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 507, in create_port
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api return self.post(self.ports_path, body=body)
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 295, in post
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api headers=headers, params=params)
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 208, in do_request
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api self._handle_fault_response(status_code, replybody)
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api exception_handler_v20(status_code, des_error_body)
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 67, in exception_handler_v20
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api status_code=status_code)
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api IpAddressGenerationFailureClient: No more IP addresses available on network 31baf6a4-aa9b-442f-9034-8948feacad40.
2015-05-08 18:51:52.46 TRACE manila.network.neutron.api
2015-05-08 18:51:52.66 DEBUG oslo_concurrency.lockutils [req-9677492e-ea8d-49f5-ad2f-194178c44b81 ca779e1e8d3f46308b5d2341c7ae3e27 3e255d4c73c34d2bb1a2797cd3e8a4e1] Releasing file lock "/tmp/custom_manila_lock_path/manila-service_instance_setup_and_teardown_network_for_instance" after holding it for 0.273s from (pid=6863) release /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
2015-05-08 18:51:52.66 DEBUG oslo_concurrency.lockutils [req-9677492e-ea8d-49f5-ad2f-194178c44b81 ca779e1e8d3f46308b5d2341c7ae3e27 3e255d4c73c34d2bb1a2797cd3e8a4e1] Lock "service_instance_setup_and_teardown_network_for_instance" released by "setup_network" :: held 0.273s from (pid=6863) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
2015-05-08 18:51:52.67 WARNING manila.share.manager [req-9677492e-ea8d-49f5-ad2f-194178c44b81 ca779e1e8d3f46308b5d2341c7ae3e27 3e255d4c73c34d2bb1a2797cd3e8a4e1] Server Information in exception can not be written to db : Invalid server_details 'None'
2015-05-08 18:51:52.127 DEBUG oslo_concurrency.lockutils [req-9677492e-ea8d-49f5-ad2f-194178c44b81 ca779e1e8d3f46308b5d2341c7ae3e27 3e255d4c73c34d2bb1a2797cd3e8a4e1] Lock "share_manager_93042963-2a58-4106-b391-d2b96eb01579" released by "_provide_share_server_for_share" :: held 1.281s from (pid=6863) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
2015-05-08 18:51:52.128 ERROR manila.share.manager [req-9677492e-ea8d-49f5-ad2f-194178c44b81 ca779e1e8d3f46308b5d2341c7ae3e27 3e255d4c73c34d2bb1a2797cd3e8a4e1] Failed to get share server for share creation.
2015-05-08 18:51:52.176 ERROR oslo_messaging.rpc.dispatcher [req-9677492e-ea8d-49f5-ad2f-194178c44b81 ca779e1e8d3f46308b5d2341c7ae3e27 3e255d4c73c34d2bb1a2797cd3e8a4e1] Exception during message handling: Maximum number of ports exceeded.
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 308, in create_share
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher {'status': 'error'})
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 302, in create_share
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher context, share_network_id, share_id)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 253, in _provide_share_server_for_share
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher return _provide_share_server_for_share()
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 245, in _provide_share_server_for_share
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher share_server = self._setup_server(context, share_server)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 741, in _setup_server
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher self.driver.deallocate_network(context, share_server['id'])
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 692, in _setup_server
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher network_info, metadata=metadata)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/driver.py", line 332, in setup_server
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher return self._setup_server(*args, **kwargs)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/generic.py", line 631, in _setup_server
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher self.admin_context, network_info)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/service_instance.py", line 386, in set_up_service_instance
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher context, instance_name, network_info)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/service_instance.py", line 477, in _create_service_instance
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher network_data = self.network_helper.setup_network(network_info)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/service_instance.py", line 728, in setup_network
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher device_owner='manila')
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/network/neutron/api.py", line 170, in create_port
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher raise exception.PortLimitExceeded()
2015-05-08 18:51:52.176 TRACE oslo_messaging.rpc.dispatcher PortLimitExceeded: Maximum number of ports exceeded.

See original description

Tags:

Valeriy Ponomaryov (vponomaryov) on 2015-05-08

Changed in manila:
milestone:	none → liberty-1
description:	updated
Changed in manila:
importance:	Undecided → Medium

Valeriy Ponomaryov (vponomaryov) on 2015-05-08

tags:

added: driver generic

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-08: Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/181470

Changed in manila:
assignee:	nobody → Valeriy Ponomaryov (vponomaryov)
status:	New → In Progress

Thomas Bechtold (toabctl) on 2015-05-09

tags:

added: kilo-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-15: Fix merged to manila (master)

Reviewed: https://review.openstack.org/181470
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=c75db095a25b7e649a5f26fb16cd8855cff1fae9
Submitter: Jenkins
Branch: master

commit c75db095a25b7e649a5f26fb16cd8855cff1fae9
Author: Valeriy Ponomaryov <email address hidden>
Date: Fri May 8 20:07:51 2015 +0300

Release Neutron ports after share server deletion using generic driver

    Generic driver creates ports in Neutron for each share server but does not
    delete it with deletion of share servers. It leads to orphaning of ports and
    fixed ip leak. So, write their IDs to share server details and release ports
    after share server deletion.

Change-Id: I719c8ece9fcaf9dcc484fb6c522997d2856cc4a3
Closes-Bug: #1453191

Changed in manila:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-19: Fix proposed to manila (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/184231

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-27: Fix merged to manila (stable/kilo)

Reviewed: https://review.openstack.org/184231
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=b27132d8b4e17b739d49802e231ce352c705dc43
Submitter: Jenkins
Branch: stable/kilo

commit b27132d8b4e17b739d49802e231ce352c705dc43
Author: Valeriy Ponomaryov <email address hidden>
Date: Fri May 8 20:07:51 2015 +0300

Release Neutron ports after share server deletion using generic driver

    Change-Id: I719c8ece9fcaf9dcc484fb6c522997d2856cc4a3
    Closes-Bug: #1453191
    (cherry picked from commit c75db095a25b7e649a5f26fb16cd8855cff1fae9)

tags:

added: in-stable-kilo

Thierry Carrez (ttx) on 2015-06-24

Changed in manila:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-10-15

Changed in manila:
milestone:	liberty-1 → 1.0.0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.