Comment 8 for bug 1387699

Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :

Reproduced on the custom ISO built with the proposed path: http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_6_0_iso/502/

Two clusters (1 controller + 4 compute) was created, one with CentOS and another with Ubuntu.
Cluster with CentOS was failed with the same "Cobbler error", all nodes was marked as 'failed'.

==== astute.log
2014-11-20T10:22:07 warning: [416] Cobbler problem. Try to repeat: 1 attempt
...
2014-11-20T10:22:51 warning: [416] Cobbler problem. Try to repeat: 2 attempt
2014-11-20T10:23:14 err: [413] Error occured while provisioning: #<XMLRPC::FaultException: <type 'exceptions.OSError'>:[Errno 2] No such file or directory: '/var/lib/tftpboot/images/bootstrap/initramfs.img'>
2014-11-20T10:23:14 info: [413] Casting message to Nailgun: {"method"=>"provision_resp", "args"=>{"task_uuid"=>"a2510311-eccb-4c01-a562-70efa7b54b82", "status"=>"error", "error"=>"Cobbler error", "progress"=>100}}
2014-11-20T10:23:16 debug: [413] Unlock discovery for failed nodes. Result: []
2014-11-20T10:23:16 err: [413] Error running provisioning: <type 'exceptions.OSError'>:[Errno 2] No such file or directory: '/var/lib/tftpboot/images/bootstrap/initramfs.img', trace:
["/usr/lib64/ruby/2.1.0/xmlrpc/client.rb:271:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler.rb:108:in `sync'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler_manager.rb:131:in `sync'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler_manager.rb:83:in `reboot_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:77:in `provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:41:in `provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

2014-11-20T10:23:16 debug: [413] Dispatching aborted by provision
===========================

See the attached diagnostic snapshot. Please note that the snapshot contains several tries of start deploy/stop provisioning before the bug appeared.

On another environment I faced the same bug with another 'no such file or directory':
===========================
2014-11-19T13:50:41 err: [405] Error running provisioning: <type 'exceptions.OSError'>:[Errno 2] No such file or directory: '/var/lib/tftpboot/images/centos-x86_64/vmlinuz', trace:
["/usr/lib64/ruby/2.1.0/xmlrpc/client.rb:271:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler.rb:108:in `sync'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler_manager.rb:131:in `sync'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/cobbler_manager.rb:50:in `add_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:63:in `provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:41:in `provision'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:85:in `block in perform_main_job'"]