Snapshot should contain Zabbix logs

[Tomasz 'Zen' Napierala]

Diagnostic snapshot does not contain Zabbix logs. For all zabbix related issues they are priceless

[Ihor Kalnytskyi (ikalnytskyi)]

In 5.1 we pack to snapshop entire /var/log of master node. So if zabbix logs are in this folder the snapshots will contain them.

[Tomasz 'Zen' Napierala (tzn)]

There are no zabbix logs in this one:
https://bugs.launchpad.net/fuel/+bug/1361903

Looks like quite fresh 5.1

[Ihor Kalnytskyi (ikalnytskyi)]

Tomasz, could you please tell me where are zabbix logs located? On Zabbix node?

[Tomasz 'Zen' Napierala (tzn)]

Ubuntu: /var/log/zabbix-server/zabbix_server.log
CentOS: /var/log/zabbix/zabbix_server.log

[Ihor Kalnytskyi (ikalnytskyi)]

Well, as I see, they are on zabbix node and _didn't locate on master node_, right?

If so, we need to configure rsyslog to sending those logs to master node too. That's exactly what other service do..

[Tomasz 'Zen' Napierala (tzn)]

Sorry, I was not precise enough. Yes, those are remote logs, on zabbix server

[Ihor Kalnytskyi (ikalnytskyi)]

Very well then. There're two possible solutions:

1. Add hack to DumpTask:

    https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/task/task.py#L692

The hack should search for zabbix nodes and dynamically add rules for retrieving logs from them.

2. Configure rsyslog on zabbix node. so it will send logs on some remote host (master node in our case).

Personally, I like the second solution, because that's about how it works for other services now. Moreover, it's a kind of backup too.

[Tomasz 'Zen' Napierala (tzn)]

Let's go with syslog, I also liked it more, it's easier and more elegant

[Mike Scherbakov (mihgen)]

Why did you move the priority to High?
If it is high, Tomasz, can you confirm that you are working on the bug?

[Tomasz 'Zen' Napierala (tzn)]

This is definately not high, as Zabbix itself is considered experimental and this bug does not cause any distruption to deployment

[Roman Prykhodchenko (romcheg)]

Moving it to 6.1 due to code freeze.

[Stanislaw Bogatkin (sbogatkin)]

I don't actually understand what is this bug about. I just deployed ISO #248, setup Zabbix, Controller, Compute. On zabbix node:
[root@node-8 rsyslog.d]# cat /etc/rsyslog.d/70-zabbix-server.conf
# file is managed by puppet

### collect zabbix server logs of all levels in /var/log/zabbix_server.log
if ($programname == 'zabbix_server') \
then -/var/log/zabbix_server.log
### stop further processing for the matched entries
& ~
[root@node-8 rsyslog.d]#

on Zabbix node:
[root@node-8 rsyslog.d]# tail -n2 /var/log/zabbix_server.log
<27>Apr 2 14:06:57 node-8 zabbix_server[23956]: [Z3005] .....<intentionally omitted for brevity>......
<27>Apr 2 14:07:54 node-8 zabbix_server[23952]: [Z3005] .....<intentionally omitted for brevity>......
[root@node-8 rsyslog.d]#

On master node:
[root@nailgun node-8.test.domain.local]# tail -n2 /var/log/docker-logs/remote/node-8.test.domain.local/zabbix_server.log
2015-04-02T14:06:54.275041+00:00 err: [Z3005] .....<intentionally omitted for brevity>......
2015-04-02T14:07:51.431969+00:00 err: [Z3005] .....<intentionally omitted for brevity>......
[root@nailgun node-8.test.domain.local]#

So, log of zabbix-server is transferred to master node. Actually, I see that zabbix-agent logs not transferred, but it should be in another bug, I think, cause all this thread dedicated to zabbix-server logs. So I will create another one for zabbix-agent.

[Tomasz 'Zen' Napierala (tzn)]

Did you confirme, that those logs go to diagnostic snapshot? This is the root issue