too many subnet-create cause q-dhcp failure

[watanabe.isao]

============================[NEUTRON]============================

DHCP max port is only validated when port is creating.
But when create_port_repuest has been sent in subnet create or update, validate will not be functionally.
This results the number of total DHCP ports excess max_fixed_ips_per_port.
If so, the DHCP agent will export error, and cannot restart itself.
Also, user is not announced about that Fixed IP not been created after the subnet creation, even the "enable_dhcp" of subnet shows "True".

[reproduce]
1. neutron net create testnet
2. neutron dhcp-agent-network-add <dhcp_agent_id> testnet
3. neutron subnet-create testnet CIDR1 --name testsub1
4. neutron subnet-create testnet CIDR2 --name testsub2
5. neutron subnet-create testnet CIDR3 --name testsub3
6. neutron subnet-create testnet CIDR4 --name testsub4
7. neutron subnet-create testnet CIDR5 --name testsub5
>>>since default value of max_fixed_ips_per_port is 5, it is ok till here.
8-1. neutron subnet-create testnet CIDR6 --name testsub6
>>>error repetly occured in q-dhcp.log.

Also, confirmed the following case cause the same error
9-1. neutron subnet-create testnet CIDR6 --name testsub6 --enable_dhcp False
9-2. neutron subnet-update testsub6 --enable_dhcp True

[trace log]
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent Traceback (most recent call last):
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 112, in call_driver
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 132, in restart
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent self.enable()
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 205, in enable
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 919, in setup
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent port = self.setup_dhcp_port(network)
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 863, in setup_dhcp_port
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent 'fixed_ips': port_fixed_ips}})
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 441, in update_dhcp_port
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent port_id=port_id, port=port, host=self.host)
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent retry=self.retry)
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent timeout=timeout, retry=retry)
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 349, in send
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent retry=retry)
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 340, in _send
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent raise result
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent RemoteError: Remote error: InvalidInput Invalid input for operation: Exceeded maximim amount of fixed ips per port.
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent [u'Traceback (most recent call last):\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/opt/stack/neutron/neutron/api/rpc/handlers/dhcp_rpc.py", line 312, in update_dhcp_port\n return self._port_action(plugin, context, port, \'update_port\')\n', u' File "/opt/stack/neutron/neutron/api/rpc/handlers/dhcp_rpc.py", line 75, in _port_action\n return plugin.update_port(context, port[\'id\'], port)\n', u' File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1014, in update_port\n port)\n', u' File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1389, in update_port\n original[\'mac_address\'], port[\'device_owner\'])\n', u' File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 466, in _update_ips_for_port\n raise n_exc.InvalidInput(error_message=msg)\n', u'InvalidInput: Invalid input for operation: Exceeded maximim amount of fixed ips per port.\n'].
2015-02-28 00:31:45.548 3011 TRACE neutron.agent.dhcp.agent
2015-02-28 00:31:45.553 DEBUG oslo_concurrency.lockutils [req-41e2c225-2f9f-4e82-a18e-c79faf13cc49 admin admin] Lock "dhcp-agent" released by "subnet_update_end" :: held 0.358s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:442
2015-02-28 00:31:45.732 3011 DEBUG neutron.agent.dhcp.agent [-] resync (b682f8e6-5250-4c8c-bb83-93427cfd6185): [RemoteError(u'Remote error: InvalidInput Invalid input for operation: Exceeded maximim amount of fixed ips per port.\n[u\'Traceback (most recent call last):\\n\', u\' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\\n executor_callback))\\n\', u\' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\\n executor_callback)\\n\', u\' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\\n result = func(ctxt, **new_args)\\n\', u\' File "/opt/stack/neutron/neutron/api/rpc/handlers/dhcp_rpc.py", line 312, in update_dhcp_port\\n return self._port_action(plugin, context, port, \\\'update_port\\\')\\n\', u\' File "/opt/stack/neutron/neutron/api/rpc/handlers/dhcp_rpc.py", line 75, in _port_action\\n return plugin.update_port(context, port[\\\'id\\\'], port)\\n\', u\' File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1014, in update_port\\n port)\\n\', u\' File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1389, in update_port\\n original[\\\'mac_address\\\'], port[\\\'device_owner\\\'])\\n\', u\' File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 466, in _update_ips_for_port\\n raise n_exc.InvalidInput(error_message=msg)\\n\', u\'InvalidInput: Invalid input for operation: Exceeded maximim amount of fixed ips per port.\\n\'].',)] _periodic_resync_helper /opt/stack/neutron/neutron/agent/dhcp/agent.py:185
2015-02-28 00:31:45.733 3011 DEBUG oslo_concurrency.lockutils [-] Lock "dhcp-agent" acquired by "sync_state" :: waited 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:430
2015-02-28 00:31:45.733 3011 INFO neutron.agent.dhcp.agent [-] Synchronizing state
2015-02-28 00:31:45.734 3011 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 81477b08ea9f4328bafe8f90ef2d3f33 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:310
2015-02-28 00:31:45.735 3011 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is 0f306089fd55406bba1b5e7af7c489ce. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:226
2015-02-28 00:31:45.861 3011 DEBUG neutron.agent.dhcp.agent [-] Calling driver for network: b682f8e6-5250-4c8c-bb83-93427cfd6185 action: enable call_driver /opt/stack/neutron/neutron/agent/dhcp/agent.py:103
2015-02-28 00:31:45.862 3011 DEBUG neutron.agent.linux.utils [-] Unable to access /opt/stack/data/neutron/dhcp/b682f8e6-5250-4c8c-bb83-93427cfd6185/pid get_value_from_file /opt/stack/neutron/neutron/agent/linux/utils.py:171
2015-02-28 00:31:45.862 3011 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 2f0b3ab027e74f819d7969701ee4a414 _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:310
2015-02-28 00:31:45.862 3011 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is 8008ae78501a4269990536eb149cc6b7. _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:226
2015-02-28 00:31:45.891 3011 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for b682f8e6-5250-4c8c-bb83-93427cfd6185.

==========================[TEMPEST]============================

DHCP port 'technically' is not a tenant resource. So, the operations of DHCP port from both of admin and non-admin user shouldn't been restricted. Moreover, the best solution here is the non-admin user shouldn’t has permissions to operate with DHCP port.
Currently, the operations of DHCP port from both of admin and non-admin user are restricted by "max_fixed_ips_per_port" parameter in neutron.conf file or it's fixed to 5 by default.
https://github.com/openstack/neutron/blob/master/etc/neutron.conf#L151
On Tempest, all test cases regarding to the operations of DHCP port are ignored or tested for the cases that:
+ Both of admin and non-admin user have permissions to perform the operations with DHCP port.
+ The number of ips per DHCP port is restricted.
For example, please refer the below test case:
https://github.com/openstack/tempest/blob/master/tempest/api/network/test_ports.py#L55
>>> In this test case, we can see that Non-admin user has permissions to create/update/delete port but does NOT care about whether that’s DHCP port or not.

So now, we need to change/add test cases relate to DHCP port operations as the following:
+ Admin user has permissions to get/create/update/delete DHCP port and no restricted.
+ Non-admin user has permissions to get/delete but cannot create/update DHCP port.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/160214

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/160214
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[watanabe.isao (watanabe.isao)]

Restored this fix.

I'm very sorry about I was stopped by some other issues, and couldn't update this fix recently. My co-worker NGUYEN TUONG THANH will continue this work on neutron and tempest. Please review his next patch set, when he updated this fix.

Co-authored-by: NGUYEN TUONG THANH <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/222982

[Armando Migliaccio (armando-migliaccio)]

I don't think tempest has anything to do with this.