Deployment with Ceph in HA failed on task ceph-mon

[Nastya Urlapova]

Deployment failed on third controller with error:
(/Stage[main]/Osnailyfacter::Ceph::Mon/Ceph::Key[client.admin]/Exec[ceph-key-client.admin]/returns) change from notrun to 0 failed: Command exceeded timeout
2017-03-13 00:50:54 ERR /usr/bin/puppet:8:in `<main>'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
2017-03-13 00:50:54 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'

full log is http://paste.openstack.org/show/602457/

Scenario:
            1. Create cluster
            2. Add 3 nodes with controller and ceph OSD roles
            3. Add 1 node with ceph OSD roles
            4. Add 2 nodes with compute and ceph OSD roles
            5. Deploy the cluster <<< failed here

Version:
10.0 ISO #1455
fuel-nailgun-10.0.0-1.mos9084.noarch
fuel-ostf-10.0.0-1.mos970.noarch
python-fuelclient-10.0.0-1.mos411.noarch
fuel-notify-10.0.0-1.mos8985.noarch
fuel-10.0.0-1.mos6384.noarch
fuel-utils-10.0.0-1.mos8985.noarch
fuel-agent-10.0.0-1.mos345.noarch
fuel-ui-10.0.0-1.mos3007.noarch
fuel-setup-10.0.0-1.mos6384.noarch
fuel-release-10.0.0-1.mos6384.noarch
fuel-bootstrap-cli-10.0.0-1.mos345.noarch
fuel-misc-10.0.0-1.mos8985.noarch
fuelmenu-10.0.0-1.mos300.noarch
fuel-openstack-metadata-10.0.0-1.mos9084.noarch
fuel-migrate-10.0.0-1.mos8985.noarch
fuel-library10.0-10.0.0-1.mos8985.noarch

[Oleksiy Molchanov (omolchanov)]

It seems to be related to host's network.

[Nastya Urlapova (aurlapova)]

@Oleksiy if it is related to host's network, should we assign this one to infra team?

[Andrey Nikitin (heos)]

Could you please provide more additional information about where that job or installation was running? Also you can provide a link to the job.

We can't investigate the problem without this informatiion.

[Roman Vyalov (r0mikiam)]

Please provide more information about the problem with network.
>It seems to be related to host's network.
what the problem ? with internet , network between VM's etc

[Oleksiy Molchanov (omolchanov)]

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.thread_3/208/testReport/(root)/ceph_ha/

node-6 had no connectivity with controllers, nothing suspicious in logs was found. On reverted env was not reproduced.