updating ipv6 allocation pool start ip address made neutron-server hang

[Jerry Zhao]

neutron subnet-update --allocation-pool start=2001:470:1f0e:cb4::20,end=2001:470:1f0e:cb4:ffff:ffff:ffff:fffe ipv6

Dec 12 04:21:14 ci-overcloud-controller0-fm6zhh6u6uwd neutron-server: 2014-12-12 04:21:14.024 21692 DEBUG neutron.api.v2.base [req-8e0c6b88-4beb-4b43-af6a-cab2824fa90c None] Request body: {u'subnet': {u'allocation_pools': [{u'start': u'2001:470:1f0e:cb4::20', u'end': u'2001:470:1f0e:cb4:ffff:ffff:ffff:fffe'}]}} prepare_request_body /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/api/v2/base.py:585
Dec 12 04:21:14 ci-overcloud-controller0-fm6zhh6u6uwd neutron-server: 2014-12-12 04:21:14.055 21692 DEBUG neutron.db.db_base_plugin_v2 [req-8e0c6b88-4beb-4b43-af6a-cab2824fa90c None] Performing IP validity checks on allocation pools _validate_allocation_pools /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py:639
Dec 12 04:21:14 ci-overcloud-controller0-fm6zhh6u6uwd neutron-server: 2014-12-12 04:21:14.058 21692 DEBUG neutron.db.db_base_plugin_v2 [req-8e0c6b88-4beb-4b43-af6a-cab2824fa90c None] Checking for overlaps among allocation pools and gateway ip _validate_allocation_pools /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py:675
Dec 12 04:21:14 ci-overcloud-controller0-fm6zhh6u6uwd neutron-server: 2014-12-12 04:21:14.061 21692 DEBUG neutron.db.db_base_plugin_v2 [req-8e0c6b88-4beb-4b43-af6a-cab2824fa90c None] Rebuilding availability ranges for subnet {'ip_version': 6L, u'allocation_pools': [{u'start': u'2001:470:1f0e:cb4::20', u'end': u'2001:470:1f0e:cb4:ffff:ffff:ffff:fffe'}], 'cidr': u'2001:470:1f0e:cb4::/64', 'id': u'5579d9bb-0d03-4d8e-ba61-9b2d8842983d'} _rebuild_availability_ranges /opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py:262

 wget 162.3.121.66:9696
--2014-12-12 04:24:18-- http://162.3.121.66:9696/
Connecting to 162.3.121.66:9696... connected.
HTTP request sent, awaiting response...

restart the neutron-server service, neutron-server got back to normal and other neutron command still worked, but neutron subnet-update allocation pool would reproduce the bug.

[Jerry Zhao (zhaoxinyu)]

Some of the agents were not alive due to an oslo.messaging bug. but that shouldn't be the reason that made neutron-server hang there.

neutron agent-list
+--------------------------------------+--------------------+-----------------------------------------+-------+----------------+---------------------------+
| id | agent_type | host | alive | admin_state_up | binary |
+--------------------------------------+--------------------+-----------------------------------------+-------+----------------+---------------------------+
| 1066b624-b739-4c12-82f3-36b111334b25 | Open vSwitch agent | ci-overcloud-novacompute0-kdu6y4wq3guu | :-) | True | neutron-openvswitch-agent |
| 1f3c8147-335d-411a-aef7-b59258566331 | Open vSwitch agent | ci-overcloud-novacompute12-avqncqm6rbte | xxx | True | neutron-openvswitch-agent |
| 20e16d37-d5ab-4520-90c0-01940d7eda04 | Open vSwitch agent | ci-overcloud-novacompute9-cjp3mz5hmjoi | :-) | True | neutron-openvswitch-agent |
| 3fa595af-84b4-43a8-87e6-6b755595a63e | Open vSwitch agent | ci-overcloud-novacompute8-xhi556pmjae4 | xxx | True | neutron-openvswitch-agent |
| 5d77d94d-6ea8-46c5-a53c-0b7d09292255 | L3 agent | ci-overcloud-controller0-fm6zhh6u6uwd | :-) | True | neutron-l3-agent |
| 7bca54c8-9d80-4af9-923a-0d5bf84b5c48 | DHCP agent | ci-overcloud-controller0-fm6zhh6u6uwd | :-) | True | neutron-dhcp-agent |
| 81de5381-a29e-4703-8838-6b6f95cdf6c3 | Metadata agent | ci-overcloud-controller0-fm6zhh6u6uwd | xxx | True | neutron-metadata-agent |
| 9ade3360-8ab9-4181-a5ff-ad08c10aa4ab | Open vSwitch agent | ci-overcloud-novacompute3-vvzr6jbys7jw | :-) | True | neutron-openvswitch-agent |
| a2f6104e-a9cf-449d-b838-bc60bf8c853e | Open vSwitch agent | ci-overcloud-novacompute11-chq5pbknaejk | :-) | True | neutron-openvswitch-agent |
| a48b919a-95fb-4658-b56f-6eead15ef0a6 | Open vSwitch agent | ci-overcloud-novacompute7-pshw5vipeo7d | xxx | True | neutron-openvswitch-agent |
| ad637423-e113-4d53-bc6d-7e1b83864a7c | Open vSwitch agent | ci-overcloud-novacompute2-k62nw5ziy6kl | :-) | True | neutron-openvswitch-agent |
| b2f6bf9f-70d8-44f8-86fa-382c2f70000a | Open vSwitch agent | ci-overcloud-novacompute1-3bcxsq5nhbtw | xxx | True | neutron-openvswitch-agent |
| b544eee2-43cf-426f-8255-55c4ca0193d9 | Open vSwitch agent | ci-overcloud-novacompute6-yejsyqkcrgrw | :-) | True | neutron-openvswitch-agent |
| b662c9c1-aadd-4022-ab25-0942274f4cb9 | Open vSwitch agent | ci-overcloud-novacompute10-zpxewqdhfu6o | :-) | True | neutron-openvswitch-agent |
| b6bf8cab-68e7-4995-8968-701252747585 | Open vSwitch agent | ci-overcloud-controller0-fm6zhh6u6uwd | xxx | True | neutron-openvswitch-agent |
| d15d05d1-54e3-412b-b2bb-6ae210b7c398 | Open vSwitch agent | ci-overcloud-novacompute5-imfc4cbe7ebe | :-) | True | neutron-openvswitch-agent |
| d20933fc-6437-4f9d-95da-b29a9b812dce | Open vSwitch agent | c...

[Sridhar Gaddam (sridhargaddam)]

Inline with our conversation, Numan I'm assigning this bug on my name and would propose a patch.

[Eugene Nikanorov (enikanorov)]

Unfortunately it's a by-design behavior of the code.

You've created ip range of 2**32 ips, when doing rebuild it operates with that many objects.

IPv6 doesn't work too well with the ipv4 allocation logic.
SLAAC addressing mode should be preferred when working with ipv6 networks.

[Eugene Nikanorov (enikanorov)]

I think it worth documenting, but I'm not sure it makes sense to do anything with this bug on neutron side.

[Jerry Zhao (zhaoxinyu)]

it doesn't make sense to let neutron-server hang there and unresponsive for, like forever.
neutron either support it with this command or return an error saying it not permitted due to so large an ip pool.

[Sridhar Gaddam (sridhargaddam)]

I agree with Jerry. I'm looking into this issue and will update my observations shortly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/141770

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141770
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b38f1bfa7b4f662182f552530b43301b214baa8d
Submitter: Jenkins
Branch: master

commit b38f1bfa7b4f662182f552530b43301b214baa8d
Author: sridhargaddam <email address hidden>
Date: Mon Dec 15 11:11:15 2014 +0000

    Fix neutron hang for IPv6 allocation pool update

    While rebuilding IPAllocationPools, Neutron is using netaddr.iter_iprange
    (a generator that produces IPAddress objects). For an IPv6 subnet, this is
    a costly operation and is causing Neutron Server to hang. This patch
    addresses the issue by replacing netaddr.iter_iprange with netaddr.IPRange
    which provides the same functionality while calculating the IPSets.

    Closes-Bug: #1401751
    Change-Id: I2287e5348aab82e39cf1b6e2884d08237a09b06d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/145242

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/145242
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=560ef8187275415e4817e2cc78ca161b0a4aa7cf
Submitter: Jenkins
Branch: stable/juno

commit 560ef8187275415e4817e2cc78ca161b0a4aa7cf
Author: sridhargaddam <email address hidden>
Date: Mon Dec 15 11:11:15 2014 +0000

    Fix neutron hang for IPv6 allocation pool update

    While rebuilding IPAllocationPools, Neutron is using netaddr.iter_iprange
    (a generator that produces IPAddress objects). For an IPv6 subnet, this is
    a costly operation and is causing Neutron Server to hang. This patch
    addresses the issue by replacing netaddr.iter_iprange with netaddr.IPRange
    which provides the same functionality while calculating the IPSets.

    Closes-Bug: #1401751
    Change-Id: I2287e5348aab82e39cf1b6e2884d08237a09b06d
    (cherry picked from commit b38f1bfa7b4f662182f552530b43301b214baa8d)

[Eran Kuris (ekuris)]

the output in bug is from this log :/var/log/neutron/server.log ?

[Sridhar Gaddam (sridhargaddam)]

Yes @Eran, the output is from the Neutron Server log. The log file can vary depending on what you are using (i.e., devstack/packstack etc).