Detailed bug description:
When Ceilometer connects to MongoDB it issues a TTL update command. Rollback for this command is case of Mongo replica recovery is not possible. The following error will appear:
replSet error cannot rollback a collMod command: { collMod: "event", index: { expireAfterSeconds: 172800, keyPattern: { timestamp: -1 } } }
Steps to reproduce:
1. Set up an environment with Ceilometer and MongoDB
2. Restart Ceilometer services on one of the controllers.
3. Disconnect one Mongo node from the network (or just stop Mongo on it) for 10 minutes.
4. Restore connection to the Mongo nodes.
5. It will not be able to restore the replica set.
Expected results:
Mongo cluster restores all replica sets.
Actual result:
The reconnected Mongo node will have a FATAL status.
Reproducibility:
Every time
Workaround:
Make a full sync for a FATAL node manually
Description of the environment:
- Operation system: any
- Versions of components: MOS 9.2
- Network model: any
- Related projects installed: Ceilometer
Additional information:
Updating TTL every time a connection is established is not actually necessary. In the newer releases of Ceilometer the TTL update mechanics was moved to CLI rather then Ceilometer itself.
sla2 for 9.0-updates