Rsyslog in tcp mode degrades/crashes deploy of cloud.

[Dmitriy Stremkovskiy]

Detail:
If one
1) uses remote rsyslog server for log collecting
2) this server is slow performing
3) server set up in fuel ui as tcp remote rsyslog server

deploy may hang with unsynced galera cluster.

Steps to reproduce:
Deploy MOS9.1 cloud with remote rsyslog server set up in tcp mode.
Remote rsyslog server should be slow performing (not enought memory/cpu/bandwidth to collect logs)

Expected:
deploy succeeded regardless of log delivery

Actual:
deploy speed degrades or deploy hangs with unsynced galera

Workaround:
use udp mode for remote rsyslog server

Impact:
Clouds with 15+ hw nodes

[Oleksiy Molchanov (omolchanov)]

Please attach diagnostic snapshot or if you logging server is very slow - use udp.

[Dmitriy Stremkovskiy (dstremkouski)]

Cannot attach snapshot, bug was found during real deployment.

I'll try to reproduce it, but anyway, we should aware users of using remote rsyslog/tcp until they really need it, adding warning info, that slow performing rsyslog server may degrade or crash deploy.

[Rudy McComb (rmccomb)]

this bug also affects me

[Oleksiy Molchanov (omolchanov)]

Please use better hardware for master node or you can use 9.2 feature with sending logs to remote syslog server via udp.