docker puppet concurrency is problematic

[Michele Baldessari]

We currently spawn docker containers for puppet in a concurrent fashion:
https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-puppet.py#L82

There are two problems with this:
1) We are trying to to configure HA containers by making pcs calls and creating properies, resources and constraints for the pacemaker cluster running on the host. Neither pacemaker nor pcs were ever designed with command concurrency from the same node. Once we set the PROCESS_COUNT=1 we do not see any issues in having pcs commands manage resources from the container

2) Debugging becomes incredibly painful. It is already non-trivial to debug when something goes wrong with containers. If we add concurrency in the mix, it becomes really daunting to rebuild the exact sequence of events that caused a problem.

The proposal would be to leave the change that brought in concurrency "I083d302b8cf6538569e4d165221c21df152266bc" but to default it to 1. So a developer can override it and let it be the number of CPUs of the system in order to speed development up, but we keep a more linear development approach and avoid concurrency issues.

[Steve Baker (steve-stevebaker)]

I'm going to mark this as invalid, docker-puppet.py exists only to generate config files which are used later by other containers, so its main criteria is that it runs fast and as concurrent as possible.

If you're using puppet to modify the host state then this should be done with a script or puppet deployment.

If you're using puppet to modify container state then this should be done in a docker-cmd deployment, which has more control over ordering.