HA tests failed after filling main controller file system

[Vladimir]

Steps to reproduce:

1. Deploy any environment with 3 controllers and NeutronTUN or NeutronVLAN
2. Fill root filesystem on primary controller "dd if=/dev/zero of=/root/file bs=32KB"
3. Wait for ~5 min
4. Verify networks
5. Run OSTF tests

HA Tests failed:

* Check state of haproxy backends on controllers
Some haproxy backend has down state.. Please refer to OpenStack logs for more details.
Target Service: HA haproxy

Failed action:
2. Check backend state for availability

* Check data replication over mysql
Can not connect to mysql. Please check that mysql is running and there is connectivity by management network Please refer to OpenStack logs for more details.
Target Service: HA mysql

Failed action:
1. Check that mysql is running on all controller or database nodes.

* Check if amount of tables in databases is the same on each node
Can list tables Please refer to OpenStack logs for more details.

Failed action:
2. Request list of tables for os databases on each node.

* Check galera environment state
Actual value - 3,

Failed action:
3. For each node check cluster size

* Check pacemaker status
Resource sysinfo_node-1.test.domain.local allowed to start on the nodes ['node-1.test.domain.local'], but it is not started on any node. Please refer to OpenStack logs for more details.

Failed action:
7. Check controllers that pcs resources are started on the same nodes

* RabbitMQ availability
Cannot retrieve cluster nodes Please refer to OpenStack logs for more details.

Failed action:
1. Retrieve cluster status for each controller.

* RabbitMQ replication
Failed to establish AMQP connection to 5673/tcp port on 10.109.1.5 from controller node! Please refer to OpenStack logs for more details.

Failed action:
1. Check rabbitmq connections.

Diagnostic snapshot is available in attachment.

[Aleksander Mogylchenko (amogylchenko)]

All kinds of problems might occur if one fills filesystem. It is recommended to have at least 10% free.

[Nastya Urlapova (aurlapova)]

If you know that all kinds of problems might occur, please do notification for users about free space at least.

[Aleksander Mogylchenko (amogylchenko)]

It is that kind of notification that says "Do not shoot yourself in the leg".

And anyway that is not a Fuel bug (or even a feature request). If UX is better with some kind of notifications - please create a separate feature request to appropriate team. And notification won't fix the behavior - thus the status remains the same.