2014-07-31 17:49:09 |
Dmitry Borodaenko |
description |
Environment deployment failed on Fuel v5.1 build 269 with ceph-deploy-1.5.2-0.mira1
Deployment failed on both enviroments with errors.
Fuel:
"build_id": "2014-06-24_14-57-29",
"mirantis": "yes",
"build_number": "269",
"ostf_sha": "4d2efa822344b6ca022ec4086b6f083c07d90e14",
"nailgun_sha": "b8713a4026239e7b1769e64239e54950ef7c739b",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "b8c33af2da36f81828e68b77da4025e59aee3283",
"astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48",
"release": "5.1",
"fuellib_sha": "8ec900a2bc96a463a72405a674e41f52fc0d3e14"
Environment 1:
OS: CentOS
Mode: Multi-node with HA (1 controller node)
Storage: Cinder Ceph and Glance Ceph
Nodes: 1 Controller, 1 Compute, 2 Ceph OSD
Puppet log from Controller node:
=============================
2014-06-24 15:32:25 ERR (/Stage[main]/Ceph::Mon/Exec[ceph-deploy mon create]/returns) change from notrun to 0 failed: ceph-deploy mon create node-1:192.168.0.3 returned 1 instead of one of [0]
2014-06-24 15:32:25 ERR /usr/bin/puppet:4
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
2014-06-24 15:32:25 ERR /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
2014-06-24 15:32:25 ERR /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:18:in `evaluate'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `perform_changes'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `each'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:82:in `perform_changes'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:193:in `sync'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:120:in `sync'
2014-06-24 15:32:25 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:97:in `fail'
2014-06-24 15:32:25 ERR ceph-deploy mon create node-1:192.168.0.3 returned 1 instead of one of [0]
2014-06-24 15:24:54 ERR (/Stage[main]/Ceph::Mon/Exec[ceph-deploy mon create]/returns) change from notrun to 0 failed: ceph-deploy mon create node-1:192.168.0.3 returned 1 instead of one of [0]
2014-06-24 15:24:54 ERR /usr/bin/puppet:4
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
2014-06-24 15:24:54 ERR /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
2014-06-24 15:24:54 ERR /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:18:in `evaluate'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `perform_changes'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `each'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:82:in `perform_changes'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:193:in `sync'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:120:in `sync'
2014-06-24 15:24:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:97:in `fail'
2014-06-24 15:24:54 ERR ceph-deploy mon create node-1:192.168.0.3 returned 1 instead of one of [0]
2014-06-24 15:12:54 ERR (/Stage[main]/Ceph::Mon/Exec[ceph-deploy mon create]/returns) change from notrun to 0 failed: ceph-deploy mon create node-1:192.168.0.3 returned 1 instead of one of [0]
2014-06-24 15:12:54 ERR /usr/bin/puppet:4
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
2014-06-24 15:12:54 ERR /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
2014-06-24 15:12:54 ERR /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:18:in `evaluate'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `perform_changes'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:81:in `each'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:82:in `perform_changes'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:193:in `sync'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:120:in `sync'
2014-06-24 15:12:54 ERR /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:97:in `fail'
2014-06-24 15:12:54 ERR ceph-deploy mon create node-1:192.168.0.3 returned 1 instead of one of [0]
=============================
Environment 2:
OS: CentOS
Mode: Multi-node
Storage: Cinder Ceph and Glance Ceph
Nodes: 1 Controller, 1 Compute, 2 Ceph OSD
Puppet log from Controller node for this environment is the same. |
Environment deployment failed on Fuel v5.1 build 269 with ceph-deploy-1.5.2-0.mira1
Deployment failed on both enviroments with errors.
Fuel:
"build_id": "2014-06-24_14-57-29",
"mirantis": "yes",
"build_number": "269",
"ostf_sha": "4d2efa822344b6ca022ec4086b6f083c07d90e14",
"nailgun_sha": "b8713a4026239e7b1769e64239e54950ef7c739b",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "b8c33af2da36f81828e68b77da4025e59aee3283",
"astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48",
"release": "5.1",
"fuellib_sha": "8ec900a2bc96a463a72405a674e41f52fc0d3e14"
Environment 1:
OS: CentOS
Mode: Multi-node with HA (1 controller node)
Storage: Cinder Ceph and Glance Ceph
Nodes: 1 Controller, 1 Compute, 2 Ceph OSD
Puppet log from Controller node:
=============================
2014-06-24 15:32:25 ERR (/Stage[main]/Ceph::Mon/Exec[ceph-deploy mon create]/returns) change from notrun to 0 failed: ceph-deploy mon create node-1:192.168.0.3 returned 1 instead of one of [0]
(useless ruby stack trace removed)
=============================
Environment 2:
OS: CentOS
Mode: Multi-node
Storage: Cinder Ceph and Glance Ceph
Nodes: 1 Controller, 1 Compute, 2 Ceph OSD
Puppet log from Controller node for this environment is the same.
(update)
The root cause of this bug is ceph-deploy failing due to mismatching ceph.conf in /root and /etc/ceph (indicative error message is "config file /etc/ceph/ceph.conf exists with different content"). Please do not reopen due to any other failure of ceph-deploy, they will have a different root cause. |
|