in ostf log we can see that 10.108.0.4 is online but(actually we get this data from nailgun, and nailgun says that both computes are online 2014-06-06 04:28:50 INFO (config) Online compute ips is [u'10.108.0.4', u'10.108.0.7'] ) we fail to ssh on it with Timeout error. (SSHTimeout: Connection to the 10.108.0.4 via SSH timed out.)I verify nailgun logs, and we can see that compute with ip 10.108.0.4 is actually online and agent from this node send data with status up and etc. It is strange but seems we should to check why nailgun api says that node is online after destroy. And may be create separate issue(may be we do not fully destroy node in sys tests)
And in app/log from nailgun we can see that cthis compute is reallu online
"role": "compute", "vlan_splinters": "disabled", "online": true, "keystone": { "db_password": "SKS34eSw", "admin_token": "U9Y6ODi8" },
2014-06-06 04:27:12.494 DEBUG [7f50fabfd700] (logger) Request PUT /api/nodes/agent/ from 10.108.0.4:49567 {"manufacturer":"QEMU","os_platform":"centos","mac":"64:19:69:F9:12:4B","is_agent":true,"agent_checksum":"18a178639b9814bfeeefca8cfbd9a47b62d20f26","platform_name":"Standard PC (i440FX + PIIX, 1996)","meta":{"disks":[{"model":null,"disk":"disk/by-path/pci-0000:00:0a.0-virtio-pci-virtio7","removable":"0","size":53687091200,"extra":[],"name":"vdc"},{"model":null,"disk":"disk/by-path/pci-0000:00:09.0-virtio-pci-virtio6","removable":"0","size":53687091200,"extra":[],"name":"vdb"},{"model":null,"disk":"disk/by-path/pci-0000:00:08.0-virtio-pci-virtio5","removable":"0","size":53687091200,"extra":[],"name":"vda"}],"memory":{"devices":[{"type":"RAM","size":1610612736}],"maximum_capacity":1610612736,"slots":1,"total":1610612736},"cpu":{"total":1,"real":0,"spec":[{"model":"Intel Xeon E312xx (Sandy Bridge)","frequency":3500}]},"system":{"manufacturer":"QEMU","version":"pc-i440fx-trusty","fqdn":"node-2","product":"Standard PC (i440FX + PIIX, 1996)"},"interfaces":[{"netmask":"255.255.255.0","mac":"64:64:BF:9C:85:C4","state":"up","current_speed":null,"name":"eth4","ip":"10.108.4.3"},{"mac":"64:A8:91:64:84:BA","state":"up","current_speed":null,"name":"eth3"},{"netmask":"255.255.255.0","mac":"64:D4:07:13:ED:B6","state":"up","current_speed":null,"name":"eth2","ip":"10.108.2.4"},{"mac":"64:DD:58:34:20:B6","netmask":"255.255.255.0","state":"up","current_speed":null,"name":"eth1","ip":"10.108.1.4"},{"netmask":"255.255.255.0","mac":"64:19:69:F9:12:4B","state":"up","current_speed":null,"name":"eth0","ip":"10.108.0.4"}]},"ip":"10.108.0.4"}
So seems that issue is in out system test and we need add wait untill destroyed nodes bacome offline in nailgun and then only run ostf
in ostf log we can see that 10.108.0.4 is online but(actually we get this data from nailgun, and nailgun says that both computes are online 2014-06-06 04:28:50 INFO (config) Online compute ips is [u'10.108.0.4', u'10.108.0.7'] ) we fail to ssh on it with Timeout error. (SSHTimeout: Connection to the 10.108.0.4 via SSH timed out.)I verify nailgun logs, and we can see that compute with ip 10.108.0.4 is actually online and agent from this node send data with status up and etc. It is strange but seems we should to check why nailgun api says that node is online after destroy. And may be create separate issue(may be we do not fully destroy node in sys tests)
And in app/log from nailgun we can see that cthis compute is reallu online
" vlan_splinters" : "disabled",
" online" : true,
" keystone" : {
"db_ password" : "SKS34eSw",
"admin_ token": "U9Y6ODi8"
} ,
"role": "compute",
2014-06-06 04:27:12.494 DEBUG [7f50fabfd700] (logger) Request PUT /api/nodes/agent/ from 10.108.0.4:49567 {"manufacturer" :"QEMU" ,"os_platform" :"centos" ,"mac": "64:19: 69:F9:12: 4B","is_ agent": true,"agent_ checksum" :"18a178639b981 4bfeeefca8cfbd9 a47b62d20f26" ,"platform_ name":" Standard PC (i440FX + PIIX, 1996)", "meta": {"disks" :[{"model" :null," disk":" disk/by- path/pci- 0000:00: 0a.0-virtio- pci-virtio7" ,"removable" :"0","size" :53687091200, "extra" :[],"name" :"vdc"} ,{"model" :null," disk":" disk/by- path/pci- 0000:00: 09.0-virtio- pci-virtio6" ,"removable" :"0","size" :53687091200, "extra" :[],"name" :"vdb"} ,{"model" :null," disk":" disk/by- path/pci- 0000:00: 08.0-virtio- pci-virtio5" ,"removable" :"0","size" :53687091200, "extra" :[],"name" :"vda"} ],"memory" :{"devices" :[{"type" :"RAM", "size": 1610612736} ],"maximum_ capacity" :1610612736, "slots" :1,"total" :1610612736} ,"cpu": {"total" :1,"real" :0,"spec" :[{"model" :"Intel Xeon E312xx (Sandy Bridge) ","frequency" :3500}] },"system" :{"manufacturer ":"QEMU" ,"version" :"pc-i440fx- trusty" ,"fqdn" :"node- 2","product" :"Standard PC (i440FX + PIIX, 1996)"} ,"interfaces" :[{"netmask" :"255.255. 255.0", "mac":" 64:64:BF: 9C:85:C4" ,"state" :"up"," current_ speed": null,"name" :"eth4" ,"ip":" 10.108. 4.3"},{ "mac":" 64:A8:91: 64:84:BA" ,"state" :"up"," current_ speed": null,"name" :"eth3" },{"netmask" :"255.255. 255.0", "mac":" 64:D4:07: 13:ED:B6" ,"state" :"up"," current_ speed": null,"name" :"eth2" ,"ip":" 10.108. 2.4"},{ "mac":" 64:DD:58: 34:20:B6" ,"netmask" :"255.255. 255.0", "state" :"up"," current_ speed": null,"name" :"eth1" ,"ip":" 10.108. 1.4"},{ "netmask" :"255.255. 255.0", "mac":" 64:19:69: F9:12:4B" ,"state" :"up"," current_ speed": null,"name" :"eth0" ,"ip":" 10.108. 0.4"}]} ,"ip":" 10.108. 0.4"}
So seems that issue is in out system test and we need add wait untill destroyed nodes bacome offline in nailgun and then only run ostf