Running latest devstack, ironic and nova, I get the following error when I request an instance:
| fault | {"message": "Node 6cc8803d-4e77-4948-b653-663d8d5e52b7 could not be found. (HTTP 404)", "code": 500, "details": " File \"/opt/stack/nova/nova/compute/manager.py\", line 1780, in _do_build_and_run_instance |
| | filter_properties) |
| | File \"/opt/stack/nova/nova/compute/manager.py\", line 2016, in _build_and_run_instance |
| | instance_uuid=instance.uuid, reason=six.text_type(e)) |
| | ", "created": "2017-02-02T13:42:01Z"} |
On ironic side, this node was indeed deleted, it is also deleted from nova.compute_nodes table:
| created_at | updated_at | deleted_at | id | service_id | vcpus | memory_mb | local_gb | vcpus_used | memory_mb_used | local_gb_used | hypervisor_type | hypervisor_version | cpu_info | disk_available_least | free_ram_mb | free_disk_gb | current_workload | running_vms | hypervisor_hostname | deleted | host_ip | supported_instances | pci_stats | metrics | extra_resources | stats | numa_topology | host | ram_allocation_ratio | cpu_allocation_ratio | uuid | disk_allocation_ratio |
...................................................
| 2017-02-02 12:20:27 | 2017-02-02 13:20:15 | 2017-02-02 13:21:15 | 2 | NULL | 1 | 1536 | 10 | 0 | 0 | 0 | ironic | 1 | | 10 | 1536 | 10 | 0 | 0 | 6cc8803d-4e77-4948-b653-663d8d5e52b7 | 2 | 192.168.122.22 | [["x86_64", "baremetal", "hvm"]] | {"nova_object.version": "1.1", "nova_object.changes": ["objects"], "nova_object.name": "PciDevicePoolList", "nova_object.data": {"objects": []}, "nova_object.namespace": "nova"} | [] | NULL | {"cpu_arch": "x86_64"} | NULL | ubuntu | 1 | 0 | 035be695-0797-44b3-930b-42349e40579e | 0 |
But in nova_api.inventories it's still there:
| created_at | updated_at | id | resource_provider_id | resource_class_id | total | reserved | min_unit | max_unit | step_size | allocation_ratio |
..........................
| 2017-02-02 13:20:14 | NULL | 13 | 2 | 0 | 1 | 0 | 1 | 1 | 1 | 16 |
| 2017-02-02 13:20:14 | NULL | 14 | 2 | 1 | 1536 | 0 | 1 | 1536 | 1 | 1 |
| 2017-02-02 13:20:14 | NULL | 15 | 2 | 2 | 10 | 0 | 1 | 10 | 1 | 1 |
nova_api.resource_providers bit:
| created_at | updated_at | id | uuid | name | generation | can_host |
.........................
| 2017-02-02 12:20:27 | 2017-02-02 13:20:14 | 2 | 035be695-0797-44b3-930b-42349e40579e | 6cc8803d-4e77-4948-b653-663d8d5e52b7 | 7 | 0 |
Waiting for resource tracker run did not help, node's been deleted for ~30 minutes already and the inventory is still there.
Code versions:
Devstack commit debc695ddfc8b7b2aeb53c01c624e15f69ed9fa2 Updated from generate-devstack-plugins-list.
Nova commit 5dad7eaef7f8562425cce6b233aed610ca2d3148 Merge "doc: update the man page entry for nova-manage db sync"
Ironic commit 5071b99835143ebcae876432e2982fd27faece10 Merge "Remove deprecated heartbeat policy check"
If it is anyhow relevant, I also run two nova-computes on the same host, I've set host=test for the second one, other than that all configs are the same. I was trying to reproduce another cell-related issue, and was creating/deleting ironic nodes, so that they map to the second nova-compute by the hash_ring.
@vdrok: Do me a favor and comment out the call to node_is_available() here:
https:/
and re-run.