Deploy is failing when neutron network is created after registering infoblox and then infoblox is Unregistered before deploy of Virtual Machine.
Error:
Deploy of virtual machine xyz on host xxx failed with exception: Build of instance 8b9a1d54-9417-454b-bb4f-b94d0d89da5d was re-scheduled: Subnet 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 could not be found. Neutron server returns request_ids: ['req-9bc7b6c9-bc73-4234-9854-f5d091a52616']
Although
The subnet is existing:
[root@abc# neutron subnet-list
+--------------------------------------+------+----------------+--------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+------+----------------+--------------------------------------------------+
| 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 | | 120.20.10.0/24 | {"start": "120.20.10.2", "end": "120.20.10.254"} |
Steps:
1.Registered Infoblox
2.Created network net_test.
[root@jupiter-vm951 powervc]# neutron net-show c51c8399-2b9e-4319-ab82-3dbbf46adfa8
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2016-11-11T10:15:59Z |
| description | |
| id | c51c8399-2b9e-4319-ab82-3dbbf46adfa8 |
| ipv4_address_scope | |
| ipv6_address_scope | |
| mtu | 1500 |
| name | net_test |
| project_id | 38b6750413e742db97ffa854c2752848 |
| provider:network_type | vlan |
| provider:physical_network | default |
| provider:segmentation_id | 1510 |
| revision_number | 4 |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 |
| tags | |
| tenant_id | 38b6750413e742db97ffa854c2752848 |
| updated_at | 2016-11-11T10:16:01Z |
+---------------------------+--------------------------------------+
3.Unregistered infoblox and waited.
4.Tried deploy of virtual machine xyz.
Deploy of virtual machine xyz on host xxx failed with exception: Build of instance 8b9a1d54-9417-454b-bb4f-b94d0d89da5d was re-scheduled: Subnet 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 could not be found. Neutron server returns request_ids: ['req-9bc7b6c9-bc73-4234-9854-f5d091a52616']
Exception seen:
2016-11-11 05:43:16.389 55297 WARNING nova.compute.manager [req-e43bafe4-866f-4865-866e-45ad8c5408e8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] Instance failed to spawn
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] o.start_deploy_simple()
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] r = f(*args, **kwds)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] for d in self.network_info:
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] self.wait()
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] self[:] = self._gt.wait()
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] return self._exit_event.wait()
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] current.throw(*self._exc)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] result = function(*args, **kwargs)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] return func(*args, **kwargs)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] return self.post(self.ports_path, body=body)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] ret = obj(*args, **kwargs)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 365, in post
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] headers=headers, params=params)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 98, in wrapper
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] ret = obj(*args, **kwargs)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] self._handle_fault_response(status_code, replybody, resp)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 98, in wrapper
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] ret = obj(*args, **kwargs)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] exception_handler_v20(status_code, error_body)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] request_ids=request_ids)
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] NotFound: Subnet 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 could not be found.
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] Neutron server returns request_ids: ['req-6a95977f-bb0a-4fb6-8fb4-e62ad8e957b8']
2016-11-11 05:43:16.389 55297 ERROR nova.compute.manager [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d]
2016-11-11 05:43:16.394 55297 INFO powervc_nova.compute.kvmhmc_manager [req-e43bafe4-866f-4865-866e-45ad8c5408e8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] [instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d] PC-FD7A7DA Terminating instance: 8b9a1d54-9417-454b-bb4f-b94d0d89da5d
2016-11-11 05:43:16.396 55297 DEBUG oslo_messaging._drivers.amqpdriver [req-e43bafe4-866f-4865-866e-45ad8c5408e8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] CAST unique_id: 012da4800d8a4b079b81433e34b5f38a NOTIFY exchange 'nova' topic 'notifications.info' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:432
2016-11-11 05:43:16.436 55297 DEBUG powervc_nova.compute.kvmhmc_manager [req-e43bafe4-866f-4865-866e-45ad8c5408e8 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] block_device_info: {'swap': None, 'ephemerals': [], 'block_device_mapping': [], 'root_device_name': u'/dev/sda'} _shutdown_instance /usr/lib/python2.7/site-packages/powervc_nova/compute/kvmhmc_manager.py:1516
In Infoblox ID following is the only log entry:
2016-11-11 05:16:02.136 55565 INFO networking_infoblox.neutron.common.notification_handler [-] Created subnet: {u'service_types': [], u'description': u'', u'enable_dhcp': False, u'network_id': u'c51c8399-2b9e-4319-ab82-3dbbf46adfa8', u'tenant_id': u'38b6750413e742db97ffa854c2752848', u'created_at': u'2016-11-11T10:16:01Z', u'dns_nameservers': [], u'updated_at': u'2016-11-11T10:16:01Z', u'ipv6_ra_mode': None, u'allocation_pools': [{u'start': u'120.20.10.2', u'end': u'120.20.10.254'}], u'gateway_ip': u'120.20.10.1', u'revision_number': 2, u'ipv6_address_mode': None, u'ip_version': 4, u'host_routes': [], u'cidr': u'120.20.10.0/24', u'project_id': u'38b6750413e742db97ffa854c2752848', u'id': u'0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694', u'subnetpool_id': None, u'name': u''}
Error seen in neutron log:
2016-11-14 06:57:49.059 61224 DEBUG neutron.ipam.driver [req-2ab7f283-ebe0-4d12-82bb-af1211675b13 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] Loading ipam driver: internal get_instance /usr/lib/python2.7/site-packages/neutron/ipam/driver.py:51
2016-11-14 06:57:49.067 61224 ERROR neutron.ipam.drivers.neutrondb_ipam.driver [req-2ab7f283-ebe0-4d12-82bb-af1211675b13 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] IPAM subnet referenced to Neutron subnet 0d08b30c-4eb8-4b2a-9bd0-0b4e0ba5d694 does not exist
2016-11-14 06:57:49.087 61224 INFO neutron.api.v2.resource [req-2ab7f283-ebe0-4d12-82bb-af1211675b13 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 38b6750413e742db97ffa854c2752848 - - -] create failed (client error): The resource could not be found.
According to Infoblox team:
This won't work unless you migrate from Infoblox back to the built-in IPAM. There is no automatic migration, and currently we do not offer a script that goes from Infoblox -> Builtin, only the other way.
It should be feasible to write a script or database migration that does what you want. There is enough data stored in the Neutron database to populate the default driver tables as needed. This would be part of Neutron though not the Infoblox driver, and should be entered as an RFE there.
If the intention is to go back to the internal IPAM backend once you have deployed pluggable IPAM with Infoblox, this is not a scenario that we contemplated and be willing to support. I'll defer to the infoblox team for more details.