Promotion jobs fail on timeout, ControllerServiceChain takes too long

[Emilien Macchi]

Promotions to Pike master fails, OVB jobs timout at step 5:
http://logs.openstack.org/15/359215/76/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/db6d3e6/

More details to come later in this bug report (I just noticed it this week-end).

[Emilien Macchi (emilienm)]

Set to critical because we haven't promoted TripleO CI for 2 weeks.

[Emilien Macchi (emilienm)]

Based on my research and log analysis, something made ControllerServiceChain very slow (from 10 min to 50 min) between March 17th and March 25th.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/450481

[Alfredo Moralejo (amoralej)]

I'm not sure if it's related, but i've observed problems when registering nodes waiting for messages in zaqar queues that i suspect may be related to https://review.openstack.org/#/c/442482/ , it's reported in https://bugs.launchpad.net/tripleo/+bug/1675384

[Emilien Macchi (emilienm)]

Moved bug to High (not Critical anymore, since we found the reason why we wouldn't get a promotion).
Moved the bug to tripleoclient, since we know it's related to the zaqar events that might consume too much CPU.
Unassigned the bug from myself, since I'm not going to work on it anymore.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/450481
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=f7c032fb583edfe127572644fd76f10a7189a079
Submitter: Jenkins
Branch: master

commit f7c032fb583edfe127572644fd76f10a7189a079
Author: Emilien Macchi <email address hidden>
Date: Mon Mar 27 22:48:49 2017 +0000

    Revert "Use a Zaqar queue to get stack events"

    It seems like we have a ton of events in zaqar
    logs that might slow down the overcloud deployment
    runtime.

    This reverts commit f4f0e92d75e635f08bdcbe14327a2956ad647f22.

    Change-Id: I1f262666b565ec64084603d341169f51c6100dca
    Related-Bug: #1676250

[Emilien Macchi (emilienm)]

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

[Ben Nemec (bnemec)]

It looks like this was fixed by the revert.