Two clusters (1 controller + 4 compute) was created, one with CentOS and another with Ubuntu.
Cluster with CentOS was failed with the same "Cobbler error", all nodes was marked as 'failed'.
==== astute.log
2014-11-20T10:22:07 warning: [416] Cobbler problem. Try to repeat: 1 attempt
...
2014-11-20T10:22:51 warning: [416] Cobbler problem. Try to repeat: 2 attempt
2014-11-20T10:23:14 err: [413] Error occured while provisioning: #<XMLRPC::FaultException: <type 'exceptions.OSError'>:[Errno 2] No such file or directory: '/var/lib/tftpboot/images/bootstrap/initramfs.img'>
2014-11-20T10:23:14 info: [413] Casting message to Nailgun: {"method"=>"provision_resp", "args"=>{"task_uuid"=>"a2510311-eccb-4c01-a562-70efa7b54b82", "status"=>"error", "error"=>"Cobbler error", "progress"=>100}}
2014-11-20T10:23:16 debug: [413] Unlock discovery for failed nodes. Result: []
2014-11-20T10:23:16 err: [413] Error running provisioning: <type 'exceptions.OSError'>:[Errno 2] No such file or directory: '/var/lib/tftpboot/images/bootstrap/initramfs.img', trace:
["/usr/lib64/ruby/2.1.0/xmlrpc/client.rb:271:in `call'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler.rb:108:in `sync'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler_manager.rb:131:in `sync'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler_manager.rb:83:in `reboot_nodes'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:77:in `provision'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:41:in `provision'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:142:in `dispatch_message'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:103:in `block in dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `call'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `block in each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `each_with_index'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:85:in `block in perform_main_job'"]
2014-11-20T10:23:16 debug: [413] Dispatching aborted by provision
===========================
See the attached diagnostic snapshot. Please note that the snapshot contains several tries of start deploy/stop provisioning before the bug appeared.
On another environment I faced the same bug with another 'no such file or directory':
===========================
2014-11-19T13:50:41 err: [405] Error running provisioning: <type 'exceptions.OSError'>:[Errno 2] No such file or directory: '/var/lib/tftpboot/images/centos-x86_64/vmlinuz', trace:
["/usr/lib64/ruby/2.1.0/xmlrpc/client.rb:271:in `call'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler.rb:108:in `sync'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler_manager.rb:131:in `sync'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler_manager.rb:50:in `add_nodes'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:63:in `provision'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:41:in `provision'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:142:in `dispatch_message'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:103:in `block in dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `call'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `block in each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `each_with_index'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `dispatch'",
"/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:85:in `block in perform_main_job'"]
Reproduced on the custom ISO built with the proposed path: http:// jenkins- product. srt.mirantis. net:8080/ view/custom_ iso/job/ custom_ 6_0_iso/ 502/
Two clusters (1 controller + 4 compute) was created, one with CentOS and another with Ubuntu.
Cluster with CentOS was failed with the same "Cobbler error", all nodes was marked as 'failed'.
==== astute.log :FaultException : <type 'exceptions. OSError' >:[Errno 2] No such file or directory: '/var/lib/ tftpboot/ images/ bootstrap/ initramfs. img'> =>"provision_ resp", "args"= >{"task_ uuid"=> "a2510311- eccb-4c01- a562-70efa7b54b 82", "status"=>"error", "error"=>"Cobbler error", "progress"=>100}} OSError' >:[Errno 2] No such file or directory: '/var/lib/ tftpboot/ images/ bootstrap/ initramfs. img', trace: lib64/ruby/ 2.1.0/xmlrpc/ client. rb:271: in `call'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ cobbler. rb:108: in `sync'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ cobbler_ manager. rb:131: in `sync'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ cobbler_ manager. rb:83:in `reboot_nodes'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ orchestrator. rb:77:in `provision'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ dispatcher. rb:41:in `provision'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:142: in `dispatch_ message' ", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:103: in `block in dispatch'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ task_queue. rb:64:in `call'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ task_queue. rb:64:in `block in each'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ task_queue. rb:56:in `each'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ task_queue. rb:56:in `each'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:101: in `each_with_index'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:101: in `dispatch'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:85:in `block in perform_main_job'"]
2014-11-20T10:22:07 warning: [416] Cobbler problem. Try to repeat: 1 attempt
...
2014-11-20T10:22:51 warning: [416] Cobbler problem. Try to repeat: 2 attempt
2014-11-20T10:23:14 err: [413] Error occured while provisioning: #<XMLRPC:
2014-11-20T10:23:14 info: [413] Casting message to Nailgun: {"method"
2014-11-20T10:23:16 debug: [413] Unlock discovery for failed nodes. Result: []
2014-11-20T10:23:16 err: [413] Error running provisioning: <type 'exceptions.
["/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
2014-11-20T10:23:16 debug: [413] Dispatching aborted by provision ======= ======= ======
=======
See the attached diagnostic snapshot. Please note that the snapshot contains several tries of start deploy/stop provisioning before the bug appeared.
On another environment I faced the same bug with another 'no such file or directory': ======= ======= ====== OSError' >:[Errno 2] No such file or directory: '/var/lib/ tftpboot/ images/ centos- x86_64/ vmlinuz' , trace: lib64/ruby/ 2.1.0/xmlrpc/ client. rb:271: in `call'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ cobbler. rb:108: in `sync'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ cobbler_ manager. rb:131: in `sync'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ cobbler_ manager. rb:50:in `add_nodes'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ orchestrator. rb:63:in `provision'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ dispatcher. rb:41:in `provision'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:142: in `dispatch_ message' ", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:103: in `block in dispatch'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ task_queue. rb:64:in `call'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ task_queue. rb:64:in `block in each'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ task_queue. rb:56:in `each'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ task_queue. rb:56:in `each'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:101: in `each_with_index'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:101: in `dispatch'", lib64/ruby/ gems/2. 1.0/gems/ astute- 0.0.2/lib/ astute/ server/ server. rb:85:in `block in perform_main_job'"]
=======
2014-11-19T13:50:41 err: [405] Error running provisioning: <type 'exceptions.
["/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/
"/usr/