Comment 2 for bug 1510556

At the start in this documentation we should show:
- how to install basic zabbix server from manifests, if we already have working roles in manifests then it is enough to describe how to use it to get simple zabbix installation, it could be simple installation, one server without proxy
- how to configure zabbix server, only the required steps, we dont need to teach how to use all zabbix functions, but we must to describe what steps are needed to get worked solution after end of puppet apply
- how to install zabbix agent, it is probably done just by setting variables in hiera, but it should be described to show what end user need to do/check to get agent installed and propertly configured
- how to add zabbix aggent to server and how to enable tests on it

After this steps we should get simple installation of zabbix, with monitoring of basic elements of system. Then we require to add monitoring of CI elements:
- we already have some number of tests for CI, (number of executors, no slave process, etc), maybe you could export it to json and put in repository, in documentation we could describe how to import it and enable on selected servers
- we have also tests for failed jobs, it will be very usefull to describe how to use it, and i think it is also required to describe

About zabbix tests i can only guess what we have configured and we can describe. I dont have access to our zabbix configs and i only show examples which i remember :)
It will be good to add all our CI related tests, of course some of tests could be not easy to export and describe, at the start we can describe the simpler part.