log_level case inconsistencies and Hostname tag missing in fluentd filter

[Jack Heskett]

Hi,

I thought I'd flag this to see if anyone had already thought about the best way to fix some inconsistencies in the logs reaching elasticsearch. The case of the log_level key value is often inconsistent between different fluentd inputs because of the source (e.g. kolla_logs log_level is 'INFO' and syslog.local1 is 'info'). This becomes a problem when you attempt to filter by the Hostname or log_level tags in Kibana and it filters case sensitively.

This seems to apply to at least the master, rocky and stein branches. A small change such as below could help people out.

kolla logs log_level appears to be consistently uppercase before input into fluentd, however I think it might be prudent to ensure this is the case by also modifying the key for existing filters.

Further, there is no Hostname tag on syslog entries which can also lead to the haproxy/swift logs being excluded from simple searches filtered by Hostname. There is a host field.

I think this might also apply to the 'Timestamp' key as the format is different.

E.g.

<filter *.var.log.kolla.*.*.log>
    @type record_transformer
    enable_ruby true
    <record>
        Hostname ${hostname}
        Logger openstack.${tag_parts[4]}
        programname ${tag_parts[5]}
        log_level ${record["log_level"].upcase}
    </record>
</filter>

<filter syslog.local1.**>
    @type record_transformer
    enable_ruby true
    <record>
        Hostname ${hostname}
        programname haproxy
        log_level ${record["log_level"].upcase}
    </record>
</filter>

Thanks,
Jack.

[Mark Goddard (mgoddard)]

Hi Jack, thanks for raising this. Consistency of log messages is important to allow them to be indexed and searched easily. The proposal of upper-casing the log level seems sound to me.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/660359