Update the processing of assigned addresses when assigning addresses

[Chengqian Liu]

openstack allinone.

16 cpus with 2.40GHz
16G memery

only one subnet in a network.

1. Neutron-server takes a long time when creating a port with multiple addresses at once

neutron port-create --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 --fixed-ip subnet_id=474b9575-b569-48dd-a3f2-0b02b752c098 66105fab-7ea9-4efc-b9c8-1839f742bdd1

When there are already 6300+ addresses, it takes 50s to allocate the address part every time the command is run.

in _generate_ips,
2s+ for list_allocations (not fixed in this fix)
2s+ for ip_allocations.add(ipallocation.ip_address)
10 times.

https://github.com/drkjam/netaddr/issues/171

- list_allocations
I found it is slow because of log in process_rows method. https://github.com/sqlalchemy/sqlalchemy/blob/master/lib/sqlalchemy/engine/result.py
I turn off 'connection_debug' in neutron.conf ，Speed increases quickly, '2 seconds , 741903 microseconds' to '711636 microseconds'.

2.When the current address pool does not have enough addresses,
  all addresses are allocated from the current pool,
  and allocations are continued from the next address pool
  until all addresses are assigned.

[Lajos Katona (lajos-katona)]

Hi could you give some details for the usecase where you add so much addresses to one port?
Am I right that you worked with master neutron?

[Chengqian Liu (liuchengqian90)]

Hi
I just want to test the performance of the neutron assigned address.
And, I worked with rocky.

[LIU Yulong (dragon889)]

Fix proposed here:
https://review.openstack.org/#/c/633406/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633406
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1746d7e0e682eab1001bfc434b73efb1cdf5f0f4
Submitter: Zuul
Branch: master

commit 1746d7e0e682eab1001bfc434b73efb1cdf5f0f4
Author: liuchengqian90 <[e-mail]>
Date: Sun Jan 27 20:47:50 2019 +0800

    Update the processing of assigned addresses when assigning addresses

    1.It is best not to use 'netaddr.IPSet.add',
      because _compact_single_network in 'IPSet.add' is quite time consuming

    2.When the current address pool does not have enough addresses,
      all addresses are allocated from the current pool,
      and allocations are continued from the next address pool
      until all addresses are assigned.

    Change-Id: I804a95fdaa3552c785e85ffab7b8ac849c634a87
    Closes-Bug: #1813253

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/736969

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/738548

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736969
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b182f9c326df8b943e840ffe3a5c5f64f535ac48
Submitter: Zuul
Branch: stable/queens

commit b182f9c326df8b943e840ffe3a5c5f64f535ac48
Author: liuchengqian90 <[e-mail]>
Date: Sun Jan 27 20:47:50 2019 +0800

    Update the processing of assigned addresses when assigning addresses

    1.It is best not to use 'netaddr.IPSet.add',
      because _compact_single_network in 'IPSet.add' is quite time consuming

    2.When the current address pool does not have enough addresses,
      all addresses are allocated from the current pool,
      and allocations are continued from the next address pool
      until all addresses are assigned.

    Change-Id: I804a95fdaa3552c785e85ffab7b8ac849c634a87
    Closes-Bug: #1813253
    (cherry picked from commit 1746d7e0e682eab1001bfc434b73efb1cdf5f0f4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738548
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e23f226d15f77a2aad6f675f7b89b987aa6ab56b
Submitter: Zuul
Branch: stable/rocky

commit e23f226d15f77a2aad6f675f7b89b987aa6ab56b
Author: liuchengqian90 <[e-mail]>
Date: Sun Jan 27 20:47:50 2019 +0800

    Update the processing of assigned addresses when assigning addresses

    1.It is best not to use 'netaddr.IPSet.add',
      because _compact_single_network in 'IPSet.add' is quite time consuming

    2.When the current address pool does not have enough addresses,
      all addresses are allocated from the current pool,
      and allocations are continued from the next address pool
      until all addresses are assigned.

    Change-Id: I804a95fdaa3552c785e85ffab7b8ac849c634a87
    Closes-Bug: #1813253
    (cherry picked from commit 1746d7e0e682eab1001bfc434b73efb1cdf5f0f4)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.