DependentPipeline merge detection and job restart race

[Clark Boylan]

There is a potential race in the merge detection of change 21569,6 in the following DependentPipeline queue
 Project openstack/nova change 21569,6
  gate-nova-docs: SUCCESS
  gate-nova-pep8: SUCCESS
  gate-nova-python26: SUCCESS
  gate-nova-python27: SUCCESS
  gate-tempest-devstack-vm-full: SUCCESS
  gate-tempest-devstack-vm-postgres-full: SUCCESS
  gate-tempest-devstack-vm-quantum: SUCCESS
Followed by:
  Project openstack/python-novaclient change 21098,4
    gate-python-novaclient-docs: SUCCESS
    gate-python-novaclient-pep8: SUCCESS
    gate-python-novaclient-python26: SUCCESS
    gate-python-novaclient-python27: SUCCESS
    gate-tempest-devstack-vm-full: None
    gate-tempest-devstack-vm-postgres-full: FAILURE
    gate-tempest-devstack-vm-quantum: SUCCESS
  Followed by:
    Project openstack/nova change 21573,6
      gate-nova-docs: SUCCESS
      gate-nova-pep8: SUCCESS
      gate-nova-python26: ABORTED
      gate-nova-python27: SUCCESS
      gate-tempest-devstack-vm-full: ABORTED
      gate-tempest-devstack-vm-postgres-full: ABORTED
      gate-tempest-devstack-vm-quantum: ABORTED

21569,6 merges because all jobs are successful. After 21569,6 merges 21098,4 becomes a severed head because one of its jobs has failed but at least one other job is running. This puts 21573,6 at the head of the queue. Zuul should restart all of its jobs and all of the jobs for changes behind it in the queue. Instead, 21573,6 depends on 21569,6 in gerrit, but zuul has not received notification from gerrit that 21569,6 has merged so zuul reports 21573,6 as unmergable as its depdencies are unmerged.

This race is more clearly seen in this captured zuul debug log http://paste.openstack.org/show/31104/.