The issue was reproduced on following environment:
------------------------------------------------
id | name | roles | version
---+---------------------+------------+---------
3 | slave-05_controller | controller | 9.0
5 | slave-04_compute | compute | 9.0
4 | slave-03_controller | controller | 9.1 (snapshot #315)
1 | slave-02_controller | controller | 9.0
2 | slave-01_controller | controller | 9.0
-----------------------------------------------
Test 10 attempt of create-delete server failed
- 6 times on controller node-4 which is 9.1
- 1 time on node-2
- 3 times on node-3
and no fails on node-1
All fails corresponded following records in /var/log/nova/nova-compute.log:
----
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [req-76d93a7e-e469-48a7-9c0e-7efb99a9a291 952683525946431fa2e586a2116c7be9 8e72eafd176a45379104ac98a4fccf69 - - -] [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] Neutron error creating port on network 0fba810c-f21a-4f08-b0ef-f49e4a50b090
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] Traceback (most recent call last):
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 254, in _create_port
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] port = port_client.create_port(port_req_body)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] ret = self.function(instance, *args, **kwargs)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 654, in create_port
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] return self.post(self.ports_path, body=body)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363, in post
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] headers=headers, params=params)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in do_request
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] self._handle_fault_response(status_code, replybody, resp)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 273, in _handle_fault_response
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] exception_handler_v20(status_code, error_body)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] request_ids=request_ids)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] BadRequest: Unrecognized attribute(s) 'dns_name'
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] Neutron server returns request_ids: ['req-cd0c8911-ec46-4f86-9929-54f6d5401b18']
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9]
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager [req-76d93a7e-e469-48a7-9c0e-7efb99a9a291 952683525946431fa2e586a2116c7be9 8e72eafd176a45379104ac98a4fccf69 - - -] Instance failed network setup after 1 attempt(s)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager Traceback (most recent call last):
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1570, in _allocate_network_async
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager bind_host_id=bind_host_id)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 668, in allocate_for_instance
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager self.force_reraise()
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 657, in allocate_for_instance
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 295, in _create_port
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager network_id, instance=instance)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager self.force_reraise()
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 254, in _create_port
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager port = port_client.create_port(port_req_body)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager ret = self.function(instance, *args, **kwargs)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 654, in create_port
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager return self.post(self.ports_path, body=body)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363, in post
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager headers=headers, params=params)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in do_request
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager self._handle_fault_response(status_code, replybody, resp)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 273, in _handle_fault_response
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager exception_handler_v20(status_code, error_body)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager request_ids=request_ids)
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager BadRequest: Unrecognized attribute(s) 'dns_name'
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager Neutron server returns request_ids: ['req-cd0c8911-ec46-4f86-9929-54f6d5401b18']
2016-09-28 19:17:12.188 23105 ERROR nova.compute.manager
The issue was reproduced on following environment: ------- ------- ------- ------- ------- ------ ------- ------- ----+-- ------- ---+--- ------ ------- ------- ------- ------- ------- ------
-------
id | name | roles | version
---+---
3 | slave-05_controller | controller | 9.0
5 | slave-04_compute | compute | 9.0
4 | slave-03_controller | controller | 9.1 (snapshot #315)
1 | slave-02_controller | controller | 9.0
2 | slave-01_controller | controller | 9.0
------
Test 10 attempt of create-delete server failed
- 6 times on controller node-4 which is 9.1
- 1 time on node-2
- 3 times on node-3
and no fails on node-1
All fails corresponded following records in /var/log/ nova/nova- compute. log: neutronv2. api [req-76d93a7e- e469-48a7- 9c0e-7efb99a9a2 91 952683525946431 fa2e586a2116c7b e9 8e72eafd176a453 79104ac98a4fccf 69 - - -] [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] Neutron error creating port on network 0fba810c- f21a-4f08- b0ef-f49e4a50b0 90 neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] Traceback (most recent call last): neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 254, in _create_port neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] port = port_client. create_ port(port_ req_body) neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 97, in with_params neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] ret = self.function( instance, *args, **kwargs) neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 654, in create_port neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] return self.post( self.ports_ path, body=body) neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 363, in post neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] headers=headers, params=params) neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 298, in do_request neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] self._handle_ fault_response( status_ code, replybody, resp) neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 273, in _handle_ fault_response neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] exception_ handler_ v20(status_ code, error_body) neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 84, in exception_ handler_ v20 neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] request_ ids=request_ ids) neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] BadRequest: Unrecognized attribute(s) 'dns_name' neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] Neutron server returns request_ids: ['req-cd0c8911- ec46-4f86- 9929-54f6d5401b 18'] neutronv2. api [instance: 2aa8ba23- 6285-4ed4- b25e-f957faab3f a9] manager [req-76d93a7e- e469-48a7- 9c0e-7efb99a9a2 91 952683525946431 fa2e586a2116c7b e9 8e72eafd176a453 79104ac98a4fccf 69 - - -] Instance failed network setup after 1 attempt(s) manager Traceback (most recent call last): manager File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 1570, in _allocate_ network_ async manager bind_host_ id=bind_ host_id) manager File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 668, in allocate_ for_instance manager self._delete_ ports(neutron, instance, created_port_ids) manager File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 220, in __exit__ manager self.force_ reraise( ) manager File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 196, in force_reraise manager six.reraise( self.type_ , self.value, self.tb) manager File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 657, in allocate_ for_instance manager security_group_ids, available_macs, dhcp_opts) manager File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 295, in _create_port manager network_id, instance=instance) manager File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 220, in __exit__ manager self.force_ reraise( ) manager File "/usr/lib/ python2. 7/dist- packages/ oslo_utils/ excutils. py", line 196, in force_reraise manager six.reraise( self.type_ , self.value, self.tb) manager File "/usr/lib/ python2. 7/dist- packages/ nova/network/ neutronv2/ api.py" , line 254, in _create_port manager port = port_client. create_ port(port_ req_body) manager File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 97, in with_params manager ret = self.function( instance, *args, **kwargs) manager File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 654, in create_port manager return self.post( self.ports_ path, body=body) manager File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 363, in post manager headers=headers, params=params) manager File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 298, in do_request manager self._handle_ fault_response( status_ code, replybody, resp) manager File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 273, in _handle_ fault_response manager exception_ handler_ v20(status_ code, error_body) manager File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 84, in exception_ handler_ v20 manager request_ ids=request_ ids) manager BadRequest: Unrecognized attribute(s) 'dns_name' manager Neutron server returns request_ids: ['req-cd0c8911- ec46-4f86- 9929-54f6d5401b 18'] manager
----
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.173 23105 ERROR nova.network.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.
2016-09-28 19:17:12.188 23105 ERROR nova.compute.