Ports remain existing while deleting subnet

[Tatiana Ovchinnikova]

Steps to reproduce:
1. Create network and subnet.
2. Open network overview page.
3. Delete subnet. Port remains existing.
4. Click port overview. The folowing error is displayed:

Traceback:
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  111. response = callback(request, *callback_args, **callback_kwargs)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py" in dec
  38. return view_func(request, *args, **kwargs)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py" in dec
  54. return view_func(request, *args, **kwargs)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py" in dec
  38. return view_func(request, *args, **kwargs)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/decorators.py" in dec
  86. return view_func(request, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py" in view
  48. return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py" in dispatch
  69. return handler(request, *args, **kwargs)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/tabs/views.py" in get
  60. context = self.get_context_data(**kwargs)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/tabs/views.py" in get_context_data
  44. exceptions.handle(self.request)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/tabs/views.py" in get_context_data
  42. context["tab_group"].load_tab_data()
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/tabs/base.py" in load_tab_data
  122. exceptions.handle(self.request)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../horizon/tabs/base.py" in load_tab_data
  119. tab._data = tab.get_context_data(self.request)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/networks/ports/tabs.py" in get_context_data
  43. exceptions.handle(request, msg, redirect=redirect)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/networks/ports/tabs.py" in get_context_data
  39. port = api.quantum.port_get(self.request, port_id)
File "/opt/stack/horizon/openstack_dashboard/wsgi/../../openstack_dashboard/api/quantum.py" in port_get
  352. port = quantumclient(request).show_port(port_id, **params).get('port')
File "/opt/stack/python-quantumclient/quantumclient/v2_0/client.py" in with_params
  106. ret = self.function(instance, *args, **kwargs)
File "/opt/stack/python-quantumclient/quantumclient/v2_0/client.py" in show_port
  252. return self.get(self.port_path % (port), params=_params)
File "/opt/stack/python-quantumclient/quantumclient/v2_0/client.py" in get
  817. headers=headers, params=params)
File "/opt/stack/python-quantumclient/quantumclient/v2_0/client.py" in retry_request
  802. headers=headers, params=params)
File "/opt/stack/python-quantumclient/quantumclient/v2_0/client.py" in do_request
  747. self._handle_fault_response(status_code, replybody)
File "/opt/stack/python-quantumclient/quantumclient/v2_0/client.py" in _handle_fault_response
  728. exception_handler_v20(status_code, des_error_body)
File "/opt/stack/python-quantumclient/quantumclient/v2_0/client.py" in exception_handler_v20
  79. message=error_dict)

Exception Type: QuantumClientException at /project/networks/ports/53366faf-55ef-4a63-91d9-6d3a817d8837/detail
Exception Value: Port 53366faf-55ef-4a63-91d9-6d3a817d8837 could not be found on network None

[Akihiro Motoki (amotoki)]

I investigated the problem.

In Neutron/Quantum, a dhcp port for a subnet is deleted automatically when the subnet is deleted. This deletion of the dhcp port is done asynchronously.

As a result, when Horizon retrieves the network detail just after the subnet deletion succeeds, the dhcp port being deleted is displayed and when you click the port the port has disappeared.

IMO, we need to change both neutron side and horizon side to address this issue.
Neutron needs to introduce the new status for a port which indicates it is being deleted.
On Horizon side, we need to update the list based on its status as the instance list does.

What do you think?

[Akihiro Motoki (amotoki)]

I believe it is not a bug of neutronclient. Mark as invalid for neutronclient.

[Akihiro Motoki (amotoki)]

On Neutron side, it is better to set the status of a port being deleted to a different value other than ACTIVE (e.g. PENDING_DELETE).

[Akihiro Motoki (amotoki)]

On Horizon side, we don't have the traceback in this case. Instead the request is redirected to the network index page and we will have a error message "Error: Unable to retrieve port details."

The traceback should be avoided in this case, but the exception is handled properly in Horizon.
We can lower the priority.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/46496

[Victoria Martinez de la Cruz (vkmc)]

It looks like this has been solved in Neutron in master.

[Wed Nov 27 14:18:52 2013] [error] Failed to delete subnet 9246478d-d743-4474-a422-86ee0e467b17
[Wed Nov 27 14:18:52 2013] [error] Recoverable error: 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 9246478d-d743-
4474-a422-86ee0e467b17. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}

Maybe we could improve the Error message. Currently I get "Error: Failed to delete subnet 9246478d-d743-4474-a422-86ee0e467b17".

I'll try to reproduce in Havana.

[Akihiro Motoki (amotoki)]

Hi Victoria,

Thanks for taking care of it. The following is a summary of the current status.

The problem reported happens during deleting subnet and it is very transient. When a subnet is requested to delete, a port created by Neutron internally (such as DHCP port) will be deleted asynchronously. As a result, when Horizon retrieves a port list for a network after requesting Neutron to delete a subnet via API, the port list may contain a port being deleted. Thus a user has a chance to click a port detail link. The original bug says he/she got a traceback when clicking a port detail link. However, an exception from neutronclient is caught by Horizon properly and an error message is displayed, i.e., the original issue cannot be reproduced.

The only solution at now is to reload a network detail page.

What is discussed in #3 and #4 is how to refresh a network detail page (containing a port list) without reloading a page.
However, there is no progress about this. Adding a new status "PENDING_*" to a port is a change of Neutron v2 Core API resource, so the proposed change is stopped. In my current feeling, a discussion about "PENDING_*" status will take more time.

The original problem says "the following error is displayed" and it cannot be reproduced, so we can close the bug.
Further improvement can be discussed as a separate topic.

What do you think?

[Akihiro Motoki (amotoki)]

> Maybe we could improve the Error message. Currently I get "Error: Failed to delete subnet 9246478d-d743-4474-a422-86ee0e467b17".

Agree. It is better to show a reason of failure. It is useful to both users and operators (though I think it is a separate bug).

[Victoria Martinez de la Cruz (vkmc)]

It's not longer reproducible in Havana either.

When trying to delete the subnet

In logs,

[Fri Dec 20 13:33:19 2013] [error] Failed to delete subnet 042b41da-efbf-4247-a718-90c065d31e37
[Fri Dec 20 13:33:19 2013] [error] Recoverable error: 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 042b41da-efbf-4247-a718-90c065d31e37. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}

In Horizon, the error message "Error: Failed to delete subnet 042b41da-efbf-4247-a718-90c065d31e37" pops up.

[Victoria Martinez de la Cruz (vkmc)]

Hi Akihiro,

Sorry for the delayed response and thanks for giving me more information about the current status for this bug.
I agree that we should keep discussing about this once we have some news about the PENDING status feature for ports in Neutron. Having this bug open just generates confusion.
We could work on a better error message but, as you mention, that should be a different bug.

Thanks!

[Deep Shikha (d-shikha)]

Hi,

I am not able to reproduce this bug in devstack version '2015.1' (neutron version : 2.3.9.24 )

However the bug is getting reproduced in openstack Havana-release version '2013.2.3' (neutron version : 2.3.0 ).

Has it been fixed. Any suggestions?

[Nikunj Aggarwal (nikunj2512)]

i am working on the horizon side of this bug

[Deep Shikha (d-shikha)]

Thanks Nikunj,

But I am talking about the horizon side only.

I am using horizon to create net & subnet and reproduce the bugs (in both devstack and openstack insatallation).

[Nikunj Aggarwal (nikunj2512)]

Hi Deep,

Do you mean you are able to reproduce the bugs??

As you can see from the comments, the horizon is not breaking but when we tries to delete the subnet before deleting the ports, it gives error message.

But Horizon is not breaking.

Can you clarify which situation you are able to reproduce??

[Deep Shikha (d-shikha)]

Hi Nikunj,

Yes, I am able to reproduce the bug with error mesage "Error: Unable to retrieve port details."

After deleting the subnet, the port still shows in the list, and on clicking the port, the page redirects to the network index page and error message pops-up ("Error: Unable to retrieve port details.")

My question is I am not able to reproduce this scenario when I use devstack version '2015.1' (neutron version : 2.3.9.24 ).

[Nikunj Aggarwal (nikunj2512)]

Hi Deep,

From my investigation, You should not be able to delete the subnet. When i am trying to delete the subnet, it says "unable to delete the subnet" and the reason neutron shows - "because of ports are still there".

I think the scenario which is described in this bug is fixed in the neutron side and now i am only try to show the error message given by neutron.

To answer to Your question is that, maybe it is fixed in neutron side. Because they are not allowing user to delete a subnet before deleting the ports.

I am using the latest version of devstack.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/131704

[David Lapsley (dlapsley)]

Confirmed Nikunj's comments above. Neutron command line won't allow you to delete a subnet if there exists a port with IP allocation:

stack@u1:~/workspace/devstack$ neutron subnet-delete 5c45534d-ba7f-4a16-816d-186389e98185
Unable to complete operation on subnet 5c45534d-ba7f-4a16-816d-186389e98185. One or more ports have an IP allocation from this subnet.

Looks like the horizon patch is still in the queue (last update November 5 2014). Since the horizon patch is being tracked via Horizon project: https://bugs.launchpad.net/horizon/+bug/1131730 is there still a need to track here?

[David Lapsley (dlapsley)]

The patchset looks nice (other than test coverage). Would be great to see it get through!

[Kahou Lei (kahou82)]

I am not able to reproduce it in Liberty

vagrant@ubuntu:~/devstack$ neutron port-list
+--------------------------------------+---------+-------------------+----------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+---------+-------------------+----------------------------------------------------------------------------------+
| ab398e08-5781-4ea2-be00-1d0b541e36fc | rawport | fa:16:3e:98:52:81 | {"subnet_id": "f5aa66c4-11e9-4087-8ada-364571f8ea4a", "ip_address": "10.10.0.2"} |
+--------------------------------------+---------+-------------------+----------------------------------------------------------------------------------+
vagrant@ubuntu:~/devstack$ neutron subnet-list
+--------------------------------------+------------+--------------+----------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+------------+--------------+----------------------------------------------+
| f5aa66c4-11e9-4087-8ada-364571f8ea4a | testsubnet | 10.10.0.0/24 | {"start": "10.10.0.2", "end": "10.10.0.254"} |
+--------------------------------------+------------+--------------+----------------------------------------------+
vagrant@ubuntu:~/devstack$ neutron subnet-delete testsubnet
Unable to complete operation on subnet f5aa66c4-11e9-4087-8ada-364571f8ea4a. One or more ports have an IP allocation from this subnet.

[OpenStack Infra (hudson-openstack)]

Change abandoned by David Lyle (<email address hidden>) on branch: master
Review: https://review.openstack.org/131704
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.