deployment fails if nodes unable to send logs to fuel master using rsyslog with tcp

[Vlad Naboichenko]

Fuel 9 deployment. 378 nodes.

Deployment failed, after fail i was unable to login on slave nodes via ssh until restart rsyslog

/etc/rsyslog.d/00-remote.conf
*.* @@(o)172.29.218.14:514;RemoteLog # == use tcp

Strace ssh process

[pid 30381] read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 118
[pid 30381] lseek(4, -62, SEEK_CUR) = 56
[pid 30381] read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 62
[pid 30381] close(4) = 0
[pid 30381] munmap(0x7f27b0fd3000, 4096) = 0
[pid 30381] socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 4
[pid 30381] connect(4, {sa_family=AF_LOCAL, sun_path="/dev/log"}, 110) = 0
[pid 30381] sendto(4, "<38>Sep 17 19:25:21 sshd[30381]:"..., 98, MSG_NOSIGNAL, NULL, 0

It stuck on trying to send log in /dev/log, after restart rsyslog - i was abble to login on node. Puppet apply failed because of the same reason.

Rsyslog configured to use both tcp and udp, in logging.pp hardcoded to send logs via tcp to fuel-master. It would be usefull to choose desired transport protocol to prevent such situations in future.

[Oleksiy Molchanov (omolchanov)]

Switching to UDP should help, but from my point of view it is just a workaround, not a fix.

[Oleksiy Molchanov (omolchanov)]

What hardware did you have? ram/cpu?

[Oleksiy Molchanov (omolchanov)]

Marking as Invalid because of no update for more than a month.