InstanceInfoCache is not always updated with concurrent instance creation
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Expired
|
Undecided
|
Unassigned |
Bug Description
Hi,
I try to launch 45 instances in parallel :
nova boot --image ${image_id} --flavor ${flavor_id} --nic net-id=${net_id} --num-instances 45 jordan-internet
Sometimes, nova fails to get the IP address of one or 2 instances :
>nova list
+------
| ID | Name | Status | Task State | Power State | Networks |
+------
| 31d9c78b-
| 3301a06c-
| 357b3366-
| 35af1894-
| 2a5afa62-
| 30d14347-
| 318a9c90-
>nova show 357b3366-
+------
| Property | Value |
+------
| status | ACTIVE |
| updated | 2013-09-
| OS-EXT-
| OS-EXT-
| key_name | jordan |
| image | Ubuntu precise 12.04.2 LTS (cecc8e4c-
| hostId | bf54cba4f3fdc7b
| OS-EXT-STS:vm_state | active |
| OS-EXT-
| OS-EXT-
| flavor | m1.small (6) |
| id | 357b3366-
| security_groups | [{u'name': u'default'}] |
| user_id | bbdcffe9d3944c3
| name | jordan-
| created | 2013-09-
| tenant_id | 462d7cbd479f499
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| accessIPv4 | |
| accessIPv6 | |
| progress | 0 |
| OS-EXT-
| OS-EXT-
| config_drive | |
+------
What's weird is that nova interfast-list is "correct" :
jordan@jpi-octo:~$ nova interface-list 357b3366-
+------
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------
| ACTIVE | d5b85cae-
+------
And quantum port-show is also 'correct' :
jordan@jpi-octo:~$ quantum port-show d5b85cae-
+------
| Field | Value |
+------
| admin_state_up | True |
| binding:
| binding:vif_type | ovs |
| device_id | 357b3366-
| device_owner | compute:None |
| fixed_ips | {"subnet_id": "32ce548d-
| id | d5b85cae-
| mac_address | fa:16:3e:2f:9b:51 |
| name | |
| network_id | 7b07d545-
| security_groups | 865ca78f-
| status | ACTIVE |
| tenant_id | 462d7cbd479f499
+------
So the problem lies in the SQL column 'network_info' of the table nova.instance_info. For a problematic instance, this column remains set as '[]' whereas for a 'correct' instance, the column contains some useful JSON data related to network
I cannot reproduce this issue when I launch let's say 5 instances in parallel or if I launch 45 instances one after another.
Any help would be very much appreciated. I don't fully understand what does compute/
Thanks,
Jordan
Setup :
Grizzly 2013.1.3
Nova-api with osapi_compute_
Quantum with openvswitch-plugin
tags: | added: network |
Changed in nova: | |
status: | Incomplete → Confirmed |
tags: | added: neutron |
All nova does to generate the initial blob of json with "good" network info in it is call out to the Neutron API. It does a few different calls and if one of those fails the whole process fails. Two additional pieces of information would be helpful:
1. At the time of creation nova-compute logs from the host the instance launched on
2. Logs from the neutron server at the same time period