all controllers node hang in MOS 9.2 env which have 300 nodes

Bug #1744689 reported by Xiwen Deng on 2018-01-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Undecided
Xiwen Deng

Bug Description

We face a grievous issue that all controller nodes hang in MOS 9.2 env. Cann't ssh and all openstack service block.

Env describe is below:
1. Three controller nodes、three network nodes、three rabbitmq nodes. We separate network and rabbitmq service from controller nodes.
2. We have 300 openstack nodes and about 2700 VMs.

Issue describe is below:
1. all controller、network、rabbitmq node hang and cann't ssh. ALL services include openstack service and ubuntu service block.
2. use cmd "strace" ssh service at the controller node, we find ssh service block at "connect(7, {sa_family=AF_LOCAL, such_path="/dev/log"}, 110) = 0"
3. if we stop rsyslog service all nodes can access and all openstack service re-work.

So what is the root cause? Do anyone face the same issue?

Xiwen Deng (deng-xiwen) on 2018-01-22
summary: - all controllers node hang in MOS 9.2 env
+ all controllers node hang in MOS 9.2 env that have 300 nodes
summary: - all controllers node hang in MOS 9.2 env that have 300 nodes
+ all controllers node hang in MOS 9.2 env which have more than 300 nodes
summary: - all controllers node hang in MOS 9.2 env which have more than 300 nodes
+ all controllers node hang in MOS 9.2 env which have 300 nodes
Denis Meltsaykin (dmeltsaykin) wrote :

Is there enough free disk space on nodes? On Fuel Master node?

Changed in fuel:
status: New → Incomplete
assignee: nobody → Xiwen Deng (deng-xiwen)
Xiwen Deng (deng-xiwen) wrote :

@Denis
    Yes, all nodes have enough free disk space. We find the issue key is that all nodes will send some service logs to Fuel node. If env have too many nodes, Fuel node can not process so many service logs.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers