The refresh-triggered puppet exec for manage-projects got reenabled in https://review.openstack.org/72178 and then logoutput was turned on for it in https://review.openstack.org/72158 so we could test in context.

Next, I merged https://review.openstack.org/70761 and https://review.openstack.org/73036 to see what happened. As luck would have it, things mostly worked as designed... the only problem I encountered was that the gerrit replicate call happened before git03 and git04 ran create-cgitrepos, so the resultant repos on those servers were empty. Manually retriggering gerrit replicate for the two projects involved solved this immediately.

After that, I got bolder and merged https://review.openstack.org/66521 and https://review.openstack.org/71293 and that's when the real fun started. New project creation failed, reporting these tracebacks through puppet into the syslog:

ERROR:manage_projects:Exception creating stackforge/savanna-ci-config in Gerrit.
Traceback (most recent call last):  
  File "/usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py", line 478, in create_gerrit_project
    gerrit.createProject(project)  
  File "/usr/local/lib/python2.7/dist-packages/gerritlib/gerrit.py", line 131, in createProject
    out, err = self._ssh(cmd)  
  File "/usr/local/lib/python2.7/dist-packages/gerritlib/gerrit.py", line 228, in _ssh
    raise Exception("Gerrit error executing %s" % command)  
Exception: Gerrit error executing gerrit create-project --require-change-id --name stackforge/savanna-ci-config
ERROR:manage_projects:Problems creating stackforge/savanna-ci-config, moving on.
Traceback (most recent call last):  
  File "/usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py", line 625, in main
    project, project_list, gerrit)  
  File "/usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py", line 478, in create_gerrit_project  
    gerrit.createProject(project)
  File "/usr/local/lib/python2.7/dist-packages/gerritlib/gerrit.py", line 131, in createProject
    out, err = self._ssh(cmd)
  File "/usr/local/lib/python2.7/dist-packages/gerritlib/gerrit.py", line 228, in _ssh
    raise Exception("Gerrit error executing %s" % command)
Exception: Gerrit error executing gerrit create-project --require-change-id --name stackforge/savanna-ci-config
ERROR:manage_projects:Exception creating stackforge/savanna-guestagent in Gerrit.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py", line 478, in create_gerrit_project
    gerrit.createProject(project)
  File "/usr/local/lib/python2.7/dist-packages/gerritlib/gerrit.py", line 131, in createProject
    out, err = self._ssh(cmd)
  File "/usr/local/lib/python2.7/dist-packages/gerritlib/gerrit.py", line 228, in _ssh
    raise Exception("Gerrit error executing %s" % command)
Exception: Gerrit error executing gerrit create-project --require-change-id --name stackforge/savanna-guestagent
ERROR:manage_projects:Problems creating stackforge/savanna-guestagent, moving on.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py", line 625, in main
    project, project_list, gerrit)
  File "/usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py", line 478, in create_gerrit_project
    gerrit.createProject(project)
  File "/usr/local/lib/python2.7/dist-packages/gerritlib/gerrit.py", line 131, in createProject
    out, err = self._ssh(cmd)
  File "/usr/local/lib/python2.7/dist-packages/gerritlib/gerrit.py", line 228, in _ssh
    raise Exception("Gerrit error executing %s" % command)
Exception: Gerrit error executing gerrit create-project --require-change-id --name stackforge/savanna-guestagent

Hunting in the Gerrit error log turned up these corresponding log entries:

ERROR com.google.gerrit.server.git.PushReplication : Failed to replicate project locally: /var/lib/git/stackforge/savanna-ci-config.git  
ERROR com.google.gerrit.server.git.PushReplication : Cannot replicate to file:///var/lib/git/stackforge/savanna-ci-config.git; repository not found  
ERROR com.google.gerrit.server.git.PushReplication : Cannot replicate to file:///var/lib/git/stackforge/savanna-ci-config.git; repository not found  
ERROR com.google.gerrit.server.git.PushReplication : Failed to replicate project locally: /var/lib/git/stackforge/savanna-guestagent.git  
ERROR com.google.gerrit.server.git.PushReplication : Cannot replicate to file:///var/lib/git/stackforge/savanna-guestagent.git; repository not found  
ERROR com.google.gerrit.server.git.PushReplication : Cannot replicate to file:///var/lib/git/stackforge/savanna-guestagent.git; repository not found

Looking in the filesystem on review.openstack.org, the /home/gerrit2/review_site/git/stackforge/savanna-ci-config.git and .../savanna-guestagent.git directories had empty refs/heads (as if they were freshly git init'ed with no first commit). There was no base repository created for either of them in /var/lib/jeepyb/stackforge nor any local replicas for them in /var/lib/git/stackforge either. Initial project creation in Gerrit seems to have happened, but the corresponding ACLs were not applied.