[devops] zabbix trigger for fuel ci failures

[Aleksandra Fedorova]

We need to monitor jobs on Fuel CI and trigger the alert if certain job $JOB_NAME fails $COUNT times on particular node $NODE_NAME.

I.e. if master_fuellib_review_systest_centos job fails 3 times in a row on fuel-jenkins3, we should probably take fuel-jenkins3 out of the test node pool.

[Mike Scherbakov (mihgen)]

Idea is great, however:
> I.e. if master_fuellib_review_systest_centos job fails 3 times in a row on fuel-jenkins3, we should probably take fuel-jenkins3 out of the test node pool.
it can happen with patchsets which fail. So to determine if server is bad, we need to run such a patch or code on it, which is known to work well (or recheck one of recently passed patches, but I'm in a favor of first option)

[Aleksandra Fedorova (bookwar)]

Right now to test the environment I trigger the test on master branch with no patches applied: see https://fuel-jenkins.mirantis.com/view/devops/job/try_test_pull_request/

Thus if we have consequent failures on a slave we can trigger this probing job, and if it fails - block the node from running tests till the problem resolved.

[Igor Shishkin (teran)]

In the light of email notifications set is it still actual?

[Igor Shishkin (teran)]

Fixed with verify-* jobs/checks