[LCM][Task idempotency] Task "primary-ceph-mon" is not idempotent
Bug #1590134 reported by
Sergey Novikov
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Invalid
|
Medium
|
Kyrylo Galanov | ||
Mitaka |
Confirmed
|
Medium
|
Fuel Sustaining |
Bug Description
Steps to reproduce:
1. deploy env with 3 controllers and 3 compute+ceph-osd
2. Rerun task "primary-ceph-mon" against primary controller node
# Run task via fuel CLI client
>fuel node --node-id <controller_id> --tasks primary-ceph-mon
#or the direct run of corresponding puppet manifest on controller node
>puppet apply --noop /etc/puppet/
Expected results: No resource changes
Actual result: The following resources were changed
- Exec[ceph-deploy gatherkeys]
Reproducibility:
100%
Description of the environment:
env was built from ISO MOS 9.0 #448
Additional information:
puppet's output: http://
Changed in fuel: | |
importance: | Undecided → High |
description: | updated |
tags: | added: area-library |
no longer affects: | fuel/newton |
Changed in fuel: | |
assignee: | Fuel Sustaining (fuel-sustaining-team) → Kyrylo Galanov (kgalanov) |
Changed in fuel: | |
importance: | High → Medium |
tags: | added: team-bugfix |
To post a comment you must log in.
This is not a problem with indepotency, but with ceph. Resource Exec[ceph-deploy gatherkeys] is idempotent, but from time to time, keyrings are not created correctly. I have done a few deployments, and sometime mds, sometime ods keyring, sometime both wasn't created. Log when both are not created below:
root@node-1:~# grep "gatherkeys" /var/log/puppet.log main]/Ceph: :Mon/Exec[ Wait for Ceph quorum]/before (debug): requires Exec[ceph-deploy gatherkeys] (provider= posix) (debug): Executing check 'test -f /root/ceph. bootstrap- mds.keyring \ main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (debug): Exec try 1/6 (provider= posix) (debug): Executing 'ceph-deploy gatherkeys node-1' main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [ceph_deploy. conf][DEBUG ] found configuration file at: /root/. cephdeploy. conf main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [ceph_deploy. cli][INFO ] Invoked (1.5.20): /usr/bin/ ceph-deploy gatherkeys node-1 main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [ceph_deploy. gatherkeys] [DEBUG ] Checking node-1 for /etc/ceph/ ceph.client. admin.keyring main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] connected to host: node-1 main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] detect platform information from remote host main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] detect machine type main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] fetch remote file main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [ceph_deploy. gatherkeys] [DEBUG ] Got ceph.client. admin.keyring key from node-1. main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [ceph_deploy. gatherkeys] [DEBUG ] Have ceph.mon.keyring main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [ceph_deploy. gatherkeys] [DEBUG ] Checking node-1 for /var/lib/ ceph/bootstrap- osd/ceph. keyring main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] connected to host: node-1 main]/Ceph: :Mon/Exec[ ceph-deploy gatherkeys]/returns (notice): [node-1][DEBUG ] detect platform information from remote host
2016-06-09 08:27:21 +0000 Puppet (debug): Adding relationship from Exec[Wait for Ceph quorum] to Exec[ceph-deploy gatherkeys] with 'before'
2016-06-09 08:27:21 +0000 Exec[ceph-deploy gatherkeys] (debug): Adding default for path
2016-06-09 08:27:21 +0000 Exec[ceph-deploy gatherkeys] (debug): Adding default for cwd
2016-06-09 08:27:23 +0000 /Stage[
2016-06-09 08:27:38 +0000 Exec[ceph-deploy gatherkeys]
2016-06-09 08:27:38 +0000 /Stage[
2016-06-09 08:27:38 +0000 Exec[ceph-deploy gatherkeys]
2016-06-09 08:27:38 +0000 Puppet (debug): Executing 'ceph-deploy gatherkeys node-1'
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /Stage[
2016-06-09 08:27:39 +0000 /...