> "the instance has a pending task (spawning). Skip".
That's unrelated from a periodic task that runs in nova-compute to look for discrepancies between what the hypervisor reports as guests running on it vs what's in the nova database. As noted in the log message, it skips instances that have a task_state set (spawning in this case).
--
Looking at the original failed job link, nova-compute starts waiting for the network-vif-plugged event here:
Aug 20 12:52:07.626394 ubuntu-xenial-ovh-gra1-0001411231 nova-compute[27062]: DEBUG nova.compute.manager [None req-633098b6-6b29-473b-a7d1-76fbf8648e7b service nova] [instance: 9797a641-8925-4f74-abbf-c6111d47f91d] Received event network-vif-plugged-e14bdbb1-ca9a-44b8-9f3b-a296f01a71fd {{(pid=27062) external_instance_event /opt/stack/nova/nova/compute/manager.py:8077}}
And then after that yeah I don't see nova saying the VM is active...I wonder if we're dropping the event somehow with an eventlet greenthread switch or something - there were some recent eventlet updates in upper-constraints, but I wouldn't think those are also on stable branches.
Do we know when this started occurring in the DNS job?
> "the instance has a pending task (spawning). Skip".
That's unrelated from a periodic task that runs in nova-compute to look for discrepancies between what the hypervisor reports as guests running on it vs what's in the nova database. As noted in the log message, it skips instances that have a task_state set (spawning in this case).
--
Looking at the original failed job link, nova-compute starts waiting for the network-vif-plugged event here:
http:// logs.openstack. org/74/ 591074/ 4/check/ neutron- tempest- plugin- designate- scenario/ d02f171/ controller/ logs/screen- n-cpu.txt. gz#_Aug_ 20_12_52_ 04_659588
Aug 20 12:52:04.659588 ubuntu- xenial- ovh-gra1- 0001411231 nova-compute[ 27062]: DEBUG nova.compute. manager [None req-e3fd2700- 621a-4a80- b16b-03d6018efb f2 tempest- DNSIntegrationT ests-199337017 tempest- DNSIntegrationT ests-199337017] [instance: 9797a641- 8925-4f74- abbf-c6111d47f9 1d] Preparing to wait for external event network- vif-plugged- e14bdbb1- ca9a-44b8- 9f3b-a296f01a71 fd {{(pid=27062) prepare_ for_instance_ event /opt/stack/ nova/nova/ compute/ manager. py:328} }
The libvirt driver then plugs the vif on the host:
vif={"profile": {}, "ovs_interfaceid": "e14bdbb1- ca9a-44b8- 9f3b-a296f01a71 fd", "preserve_ on_delete" : true, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "10.1.0.6"}], "version": 4, "meta": {"dhcp_server": "10.1.0.2"}, "dns": [], "routes": [], "cidr": "10.1.0.0/28", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "10.1.0.1"}}], "meta": {"injected": false, "tunneled": true, "tenant_id": "df130fc6f8a64a b1b877c39e32e8a 1d6", "physical_network": null, "mtu": 1400}, "id": "f7495a16- b458-49e3- b219-b5baa539df 75", "label": "tempest- test-network- -304750879" }, "devname": "tape14bdbb1-ca", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true, "datapath_type": "system", "ovs_hybrid_plug": false}, "address": "fa:16: 3e:7a:37: d1", "active": false, "type": "ovs", "id": "e14bdbb1- ca9a-44b8- 9f3b-a296f01a71 fd", "qbg_params": null}
n-cpu receives the network-vif-plugged event here:
http:// logs.openstack. org/74/ 591074/ 4/check/ neutron- tempest- plugin- designate- scenario/ d02f171/ controller/ logs/screen- n-cpu.txt. gz#_Aug_ 20_12_52_ 07_626394
Aug 20 12:52:07.626394 ubuntu- xenial- ovh-gra1- 0001411231 nova-compute[ 27062]: DEBUG nova.compute. manager [None req-633098b6- 6b29-473b- a7d1-76fbf8648e 7b service nova] [instance: 9797a641- 8925-4f74- abbf-c6111d47f9 1d] Received event network- vif-plugged- e14bdbb1- ca9a-44b8- 9f3b-a296f01a71 fd {{(pid=27062) external_ instance_ event /opt/stack/ nova/nova/ compute/ manager. py:8077} }
And then after that yeah I don't see nova saying the VM is active...I wonder if we're dropping the event somehow with an eventlet greenthread switch or something - there were some recent eventlet updates in upper-constraints, but I wouldn't think those are also on stable branches.
Do we know when this started occurring in the DNS job?