FIP disassociation takes longer in non DVR test scenario

[Sonu]

Problem description:
With series of VM delete operation in openstack (4000 vms) with KVM compute nodes, the VM instance goes into ERROR state.
The error shown in Horizon UI is
"ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='192.168.0.1', port=9696): Read timed out. (read timeout=30)"

This happens because neutron takes more than 30 secs (actually around 80 secs) to delete one port, and nova sets the instance into ERROR state 'cz the default timeout of all neutron api(s) is set to 30 sec in nova.
This can be worked around, by increasing the timeout to 120 in nova.conf. But this cannot be recommended as the solution.

cat /etc/nova/nova.conf | grep url_timeout
url_timeout = 120

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/234168

[Sonu (sonu-sudhakaran)]

https://review.openstack.org/#/c/234168/
The review attempts to solve the problem for non DVR cases in Kilo.
DVR code is re-factored for better performance in L release and there is no need to fix anything in Liberty.

[Ihar Hrachyshka (ihar-hrachyshka)]

Please describe your setup, which types of routers you use, how many of them you have, etc. Please provide debug logs and config files.

[Sonu (sonu-sudhakaran)]

Neutron release tested : Juno/stable
Compute environment: KVM (100 Hosts)
Networking configuration: centralized router, no DVR in picture
Number of tenant networks: 25
Number of Floating IPs: 4000

[Sonu (sonu-sudhakaran)]

horizon error

[Sonu (sonu-sudhakaran)]

stack trace

[Sonu (sonu-sudhakaran)]

stack trace

[Ihar Hrachyshka (ihar-hrachyshka)]

Sonu, the code proposed, what makes you think it's the culprit? Have you tested it locally?

[Ryan Moats (rmoats)]

It is still unclear from the description and comment stream, whether the original test was performed for both DVR and non-DVR cases and whether the problem still exists with liberty/master for the non-DVR case - the code commit addresses the non-DVR code path in kilo, but the comment about liberty/master only discusses DVR code path refactoring without being clear that the non-DVR code path is also fixed.

[Sonu (sonu-sudhakaran)]

We have tested this fix on both DVR and non DVR case in Juno. The DVR code is refactored majorly in L release, and this function is removed. The fix addresses the performance degradation under non DVR configuration in Juno and Kilo. For DVR cases, customer will have to move to Liberty to solve the performance issue.

[Armando Migliaccio (armando-migliaccio)]

This is really a catch-all bug, in the sense that a connection timeout during heavy load can occur for all sort of reasons. Please consider going more specific on the issue you're facing and adjust the description accordingly.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sonu (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/234168
Reason: Not applicable in liberty due to major refactoring in DVR.

[Armando Migliaccio (armando-migliaccio)]

This bug is > 180 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]