all controllers node hang in MOS 9.2 env which have 300 nodes

[Xiwen Deng]

We face a grievous issue that all controller nodes hang in MOS 9.2 env. Cann't ssh and all openstack service block.

Env describe is below:
1. Three controller nodes、three network nodes、three rabbitmq nodes. We separate network and rabbitmq service from controller nodes.
2. We have 300 openstack nodes and about 2700 VMs.

Issue describe is below:
1. all controller、network、rabbitmq node hang and cann't ssh. ALL services include openstack service and ubuntu service block.
2. use cmd "strace" ssh service at the controller node, we find ssh service block at "connect(7, {sa_family=AF_LOCAL, such_path="/dev/log"}, 110) = 0"
3. if we stop rsyslog service all nodes can access and all openstack service re-work.

So what is the root cause? Do anyone face the same issue?

[Denis Meltsaykin (dmeltsaykin)]

Is there enough free disk space on nodes? On Fuel Master node?

[Xiwen Deng (deng-xiwen)]

@Denis
    Yes, all nodes have enough free disk space. We find the issue key is that all nodes will send some service logs to Fuel node. If env have too many nodes, Fuel node can not process so many service logs.