neutron-server initiation is time-consuming with a large vxlan/gre range

[Xurong Yang]

When we configure VXLAN range [1,16M], neutron-server service costs long time and cpu rate is very high(100%) when initiation. One test base on postgresql has been verified: more than 1h when VXLAN range is [1, 1M].

[yong sheng gong (gongysh)]

How about this algorithm?
first we populate a small block of resources into db, and then have a background thread monitoring the allocation status. If the allocation meets a criteria, the thread will populate one more block of resources into db.

take the vxlan ranges: 1..100000, we divide it into 100 blocks of 1000, first one 1000 is populated into db, and then another 1000 if needed.

[Xurong Yang (idopra)]

Hi, yong sheng gong,
   Thanks for your idea, your algorithm sounds great, I'm in favor of the resource division, but I still have other considers:
   we can't predict which segmentation_id would be used immediately or latter, so I think it's more wise to allocate the required segmentation_id and have a background thread allocate the remaining ids in the same block, rather than allocation
   on the phase of initiation.
   And we need to consider how to solve on the condition that multi neturon-server services coexist in system.
   What do you think?

-----邮件原件-----
发件人: <email address hidden> [mailto:<email address hidden>] 代表 yong sheng gong
发送时间: 2014年6月3日 10:28
收件人: Yangxurong
主题: [Bug 1324875] Re: neutron-server initiation is time-consuming with a large vxlan/gre range

How about this algorithm?
first we populate a small block of resources into db, and then have a background thread monitoring the allocation status. If the allocation meets a criteria, the thread will populate one more block of resources into db.

take the vxlan ranges: 1..100000, we divide it into 100 blocks of 1000, first one 1000 is populated into db, and then another 1000 if needed.

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1324875

Title:
  neutron-server initiation is time-consuming with a large vxlan/gre
  range

Status in OpenStack Neutron (virtual network service):
  Confirmed

Bug description:
  When we configure VXLAN range [1,16M], neutron-server service costs
  long time and cpu rate is very high(100%) when initiation. One test
  base on postgresql has been verified: more than 1h when VXLAN range is
  [1, 1M].

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1324875/+subscriptions

[Eugene Nikanorov (enikanorov)]

I'm not sure it's a good option: increases complexity and can lead to lags in some responses.
Why not just to optimize db performance?

[yalei wang (yalei-wang)]

Just use tox to do a unit test, and it costs about 70s in my PC with range ([1, 1024*1024]), so most of the time is occupied by DB update operation?

[Xurong Yang (idopra)]

Yes, DB operations are time-consuming, we also have done the series of tests and noticed that sqlalchemy occupied most proportions of time eventually, so i personally think the optimization in aspect of sqlalchemy is limit.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/97774

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97774
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6e2fce563ab250d4bfe000dd2a2dbc00f094141b
Submitter: Jenkins
Branch: master

commit 6e2fce563ab250d4bfe000dd2a2dbc00f094141b
Author: Eugene Nikanorov <email address hidden>
Date: Wed Jun 4 14:40:13 2014 +0400

    Improve vxlan type driver initialization performance

    Vxlan type driver may take long time to initialize
    vxlan allocation table. Optimize db performance by issuing
    raw sql inserts coalesced into bulk statements.
    Also optimize deleting logic.

    Proposed patch gives ~2x performance gain in comparison with
    original code on Mysql and Postgesql backends

    Change-Id: I801d967e8e3c0260593f289097d17270ef0b391e
    Partial-Bug: #1324875

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/101982

[OpenStack Infra (hudson-openstack)]

Change abandoned by Cedric Brandily (<email address hidden>) on branch: master
Review: https://review.openstack.org/101982
Reason: Abandoned for the moment

[Eugene Nikanorov (enikanorov)]

Lowering the importance. Initial issue with slow initialization was partially fixed with decent speed improvement.

[Eugene Nikanorov (enikanorov)]

Also, ideally initialization is done just once for the deployment, and not be each neutron-server.