[LCM][Task idempotency] Task "primary-ceph-mon" is not idempotent

[Sergey Novikov]

Steps to reproduce:
 1. deploy env with 3 controllers and 3 compute+ceph-osd
 2. Rerun task "primary-ceph-mon" against primary controller node

 # Run task via fuel CLI client
 >fuel node --node-id <controller_id> --tasks primary-ceph-mon
 #or the direct run of corresponding puppet manifest on controller node
 >puppet apply --noop /etc/puppet/modules/osnailyfacter/modular/ceph/mon.pp

Expected results: No resource changes
Actual result: The following resources were changed
    - Exec[ceph-deploy gatherkeys]
Reproducibility:
 100%
Description of the environment:
env was built from ISO MOS 9.0 #448
Additional information:
puppet's output: http://paste.openstack.org/show/508761/

issue is caught by https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.idempotency/43/testReport/(root)/idempotency_3_ctrl_3_cmp_ceph_sahara/idempotency_3_ctrl_3_cmp_ceph_sahara/

[Maciej Relewicz (rlu)]

This is not a problem with indepotency, but with ceph. Resource Exec[ceph-deploy gatherkeys] is idempotent, but from time to time, keyrings are not created correctly. I have done a few deployments, and sometime mds, sometime ods keyring, sometime both wasn't created. Log when both are not created below:

root@node-1:~# grep "gatherkeys" /var/log/puppet.log
2016-06-09 08:27:21 +0000 Puppet (debug): Adding relationship from Exec[Wait for Ceph quorum] to Exec[ceph-deploy gatherkeys] with 'before'
2016-06-09 08:27:21 +0000 Exec[ceph-deploy gatherkeys] (debug): Adding default for path
2016-06-09 08:27:21 +0000 Exec[ceph-deploy gatherkeys] (debug): Adding default for cwd
2016-06-09 08:27:23 +0000 /Stage[main]/Ceph::Mon/Exec[Wait for Ceph quorum]/before (debug): requires Exec[ceph-deploy gatherkeys]
2016-06-09 08:27:38 +0000 Exec[ceph-deploy gatherkeys](provider=posix) (debug): Executing check 'test -f /root/ceph.bootstrap-mds.keyring \
2016-06-09 08:27:38 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (debug): Exec try 1/6
2016-06-09 08:27:38 +0000 Exec[ceph-deploy gatherkeys](provider=posix) (debug): Executing 'ceph-deploy gatherkeys node-1'
2016-06-09 08:27:38 +0000 Puppet (debug): Executing 'ceph-deploy gatherkeys node-1'
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [ceph_deploy.cli][INFO ] Invoked (1.5.20): /usr/bin/ceph-deploy gatherkeys node-1
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [ceph_deploy.gatherkeys][DEBUG ] Checking node-1 for /etc/ceph/ceph.client.admin.keyring
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] connected to host: node-1
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] detect platform information from remote host
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] detect machine type
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] fetch remote file
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from node-1.
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [ceph_deploy.gatherkeys][DEBUG ] Checking node-1 for /var/lib/ceph/bootstrap-osd/ceph.keyring
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] connected to host: node-1
2016-06-09 08:27:39 +0000 /Stage[main]/Ceph::Mon/Exec[ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] detect platform information from remote host
2016-06-09 08:27:39 +0000 /...

[Kyrylo Galanov (kgalanov)]

Lowering priority because cluster health is not affected.

[Kyrylo Galanov (kgalanov)]

Upstream puppet-ceph is used in 10.0, there is no such step.

[Sergey Novikov (snovikov)]

I propose to move the issue to 9.0-updates as there is no cluster outage.