neutron-agent-dhcp takes excessive amount of time when you have a lot of networks

Bug #1405477 reported by Sergii Golovatiuk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Vladimir Kuklin
6.0.x
Fix Released
High
Denis Meltsaykin

Bug Description

When you have many tenants with many networks. /usr/lib/ocf/resource.d/mirantis/neutron-agent-dhcp stop takes minutes performing clean_up of orphaned processes in every name space.

+++ for netns in '$ns_list'
+++ ip netns exec qdhcp-b556bceb-6924-4a17-8a8e-91e2e41aa24a lsof -n -i -t
+++ for netns in '$ns_list'
+++ ip netns exec qdhcp-d1617a6b-fb81-494e-bd5f-4193d2af25f0 lsof -n -i -t
+++ for netns in '$ns_list'
+++ ip netns exec qdhcp-b0a48d31-b1e2-4858-adc4-f812a17a8416 lsof -n -i -t
+++ for netns in '$ns_list'
+++ ip netns exec qdhcp-9b7d7755-9dca-48d1-aadf-76201a7320a2 lsof -n -i -t

every ip netns exec qdhcpXXXXX lsof -n -i -t takes almost a second. So if you have more than 60 networks, stop operation will be timed out leaving a lot of orphaned networks and dnsmasq processes.

Changed in fuel:
importance: Undecided → High
assignee: nobody → Fuel Library Team (fuel-library)
milestone: none → 6.1
status: New → Triaged
tags: added: scale
tags: added: release-notes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/143909

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Sergii Golovatiuk (sgolovatiuk)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/154093

Changed in fuel:
assignee: Sergii Golovatiuk (sgolovatiuk) → Stanislav Makar (smakar)
Changed in fuel:
assignee: Stanislav Makar (smakar) → Vladimir Kuklin (vkuklin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/154093
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=503eaf96cf6a0909dbb34b829b2058b61c136cb6
Submitter: Jenkins
Branch: master

commit 503eaf96cf6a0909dbb34b829b2058b61c136cb6
Author: Stanislav Makar <email address hidden>
Date: Mon Feb 9 15:41:31 2015 +0000

    Use the command *ip netns pids* in Neutron OCF scripts

    Use command *ip netns pids* instead of lsof. This decreases the time of
    getting pids inside namespace. Profit >20 times faster then with lsof.

    Change-Id: Ife3f5da3558bf4b5ddd0abbe997cd4b845a8d53c
    Closes-bug: #1405477

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Dina Belova (dbelova) wrote :

Fix verified for 6.1 release on #233 ISO.

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
Dina Belova (dbelova) wrote :

Sorry, mixed the links in the browser. This bug needs to be verified.

Changed in fuel:
status: Fix Released → Fix Committed
tags: added: release-notes-done
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-docs (stable/6.1)

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/194961

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-docs (stable/6.1)
Download full text (45.4 KiB)

Reviewed: https://review.openstack.org/194961
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=0e26e7d7cc153d179ec34985645dd23cdd239ddb
Submitter: Jenkins
Branch: stable/6.1

commit 5cc5f0c643aebecaf3bf4580535a3ea7c3334a6c
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:43:35 2015 -0700

    Removed streamlined patching backend pieces

    Change-Id: I955e76ccdbd12a9145f4e9b689f80bdf9fcaf929

commit 563c4b5c78ebfcb1f4f91047c2919f6270f9a1d4
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:30:30 2015 -0700

    Removed outdated patching guide

    Change-Id: I76180c277789ade9c5ebedd19fe2092847c0b7d9

commit 8d120c14bec1ab41d448683ad146a3053a57c4ee
Author: Irina Povolotskaya <email address hidden>
Date: Tue Jun 23 19:59:11 2015 +0300

    Add dual hypervisor ref arch into 6.1 docs

    Change-Id: I900c24c9de878eafadbfc995aa879b7f55737fac

commit feebd1592d3305b64bbdfd0bc5fe108190aef120
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:38:17 2015 +0300

    [OPs guide] Running Ceilometer section edits

    1. conf file extract is updated
    2. note is updated

    Closes-bug: 1467817
    Change-Id: I0217e164108e0ba6c1397045a5e57d13ff429223

commit 44a93f9dead7511a3461ec35248dbb689c81eafd
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:04:40 2015 +0300

    [RN6_1] Final changes

    1. capitalization
    2. 2014.2 to 2014.2.2
    3. general improvements

    Change-Id: I45057e90c90550559f66bc67ccdf97a559fd9000

commit bb41389cae58084285688853281516b659686422
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 16:45:35 2015 +0300

    Update patching decription

    Update patching description with
    the standard Linux commands.

    Change-Id: Ia1a8346639c468fdfce15a11d2430bf3a4731244

commit bf3018fae3f2e564413d33aba6cdebf8868f0b4e
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 15:55:49 2015 +0300

    [RN6_1] Clean up

    1. Rearranges sections
    2. Improves RST
    3. Changes titles order

    Change-Id: I6110bf515667d3d6ba08ad35ff5d593dbc96641e

commit 1c7e4457808e8f2d6c56fdf31252170972e444b9
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 15:26:28 2015 +0300

    Replaces VBOX screenshots

    This patch:
    - replaces VBOX screenshots
    - changes the link for Download Mirantis VirtualBox scripts
     to https://docs.mirantis.com/openstack/fuel/fuel-master/#downloads

    Change-Id: I58dede960c5c3355d39b07ff44b757403f6af02c
    Closes-Bug: #1467872

commit 0a568bf53fc0e25d1d692d5d74b4a7b4d983bbcc
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 14:01:55 2015 +0300

    6.1 --separate repos

    change wording and add links to the
    separate repos feature.

    Change-Id: Ib5d0778a0d8f1534f79ed2f553574cb69a3150b0

commit 95a188b21cbdd064d92696b7920e6a0105fe0c56
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 12:07:28 2015 +0300

    Corrects the output 'pcs status'

    Changes the example outputs to appropriate ones.

    Change-Id: Ib6d83...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/143909

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.0)

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/243135

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/6.0)

Reviewed: https://review.openstack.org/243135
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b0aee5f006031ec0a9ac938b035615d5b8b92ddf
Submitter: Jenkins
Branch: stable/6.0

commit b0aee5f006031ec0a9ac938b035615d5b8b92ddf
Author: Denis V. Meltsaykin <email address hidden>
Date: Mon Nov 9 17:15:17 2015 +0300

    Speedup gathering pids from namespaces by OCF

    Currently, neutron OCF scripts are using `lsof` to gather pids of
    processes running in particular namespace. `lsof` is extremely slow
    due to inoptimal architecture and causes significant timeouts:
    2+ minutes on 300 pids in 300 namespaces. On the other hand,
    we cannot use `ip netns pids`, as this command is not present in
    precise. The fix changes this call to the 2-stage execution inspired
    by the original code of `ip netns pids`, which uses comparison of
    the namespace inode with the `ns/net` symlink of the running
    process that points to the required namespace. This scales about
    linear: 1.7 seconds per 300 pids in 300 namespaces.

    Closes-Bug: #1464815
    Closes-Bug: #1405477

    Change-Id: Ife3f5da3558bf4b5ddd0abbe997cd4b845a8d53c

Revision history for this message
OSCI Robot (oscirobot) wrote :

NOTE: Changeset is not merged, created temporary package repository.
RPM package fuel-library6.0 has been built for project openstack/fuel-library.
Files placed in repository:
fuel-ha-utils6.0-6.0.0-6210.1.noarch.rpm
fuel-library6.0-6.0.0-6210.1.noarch.rpm
Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable/centos .

Revision history for this message
Alexander Tsamutali (astsmtl) wrote :

Disregard "NOTE" is previous comment.

Revision history for this message
Alexander Tsamutali (astsmtl) wrote :

s/is/in

Revision history for this message
OSCI Robot (oscirobot) wrote :

NOTE: Changeset is not merged, created temporary package repository.
RPM package fuel-library6.0 has been built for project openstack/fuel-library.
Files placed in repository:
fuel-ha-utils6.0-6.0.0-6210.1.noarch.rpm
fuel-library6.0-6.0.0-6210.1.noarch.rpm
Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable/centos .

Revision history for this message
OSCI Robot (oscirobot) wrote :

NOTE: Changeset is not merged, created temporary package repository.
DEB package fuel-library has been built for project openstack/fuel-library.
Files placed in repository:
fuel-ha-utils6.0_6.0.0-6210.1_all.deb
fuel-library6.0_6.0.0-6210.1_all.deb
Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable/ubuntu .

Revision history for this message
Vadim Rovachev (vrovachev) wrote :

Verified on 6.0

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.