Increased RAM usage leads to floating galera-mysql errors during swarm runs

[Dmitry Kalashnik]

During 9.1 cycle we have faced floating issue with mysql-galera cluster caused by increased RAM usage.
Issue could appears during deploy, during ostf run, etc.

Making new ticket to keep all investigation details in the one place.

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive_vlan/84/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/86/testReport/(root)/ha_neutron_mysql_termination/ha_neutron_mysql_termination/

[Stanislaw Bogatkin (sbogatkin)]

We solved this problem for smoke_neutron by increasing target nodes RAM volume. So, I wanna ask QA team if we want to go this way for other tests and if no, then why.

[Stanislaw Bogatkin (sbogatkin)]

This question first need to be addressed to CI team also, so I assign this bug to them.

[Roman Vyalov (r0mikiam)]

how much memory should be on VMs ? you are proposing to increase memory only for 2 threads in the swarm ?

[Alexandra (aallakhverdieva)]

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.mixed_os_components/14/testReport/(root)/mixed_components_murano_sahara_ceilometer/

[Stanislaw Bogatkin (sbogatkin)]

Roman, how much do we have now? 2Gb? Can we raise it to one GB more?

[Roman Vyalov (r0mikiam)]

only for 2 swarm threads ?

[Roman Vyalov (r0mikiam)]

now we have 3 GB

[Stanislaw Bogatkin (sbogatkin)]

Ok, sounds good. Let's look to test results.

[Roman Vyalov (r0mikiam)]

@Stas now we have 3 Gb of RAM, we can increase to 4 Gb if it necessary
Also we should to increase memory only for 2 swarm threads or in the all ones ?

[Sergii Golovatiuk (sgolovatiuk)]

@Rvyalov. The problem is not in Galera. It uses around 500-700MB which is ok. The problem is in services such neutron that uses 100Mb per process, but 16 processes spawned.

Run

ps -C neutron-server -orss= | awk '{ count ++; size += $1 }; END {print "Number of processes =",count; print "Memory usage per process =",size/1024/count, "MB"; print "Total memory usage =", size/1024, "MB"}'

next time to find a victim.

[Sergii Golovatiuk (sgolovatiuk)]

It would be nice to increase RAM for all threads which spawns neutron related jobs

[Ivan (iremizov)]

Adding global setter of fuel-qa environ stuff:
https://review.fuel-infra.org/27237

[Roman Vyalov (r0mikiam)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive_vlan/99/console
SLAVE_NODE_MEMORY=4096

[ElenaRossokhina (esolomina)]

RAM is increased, but the following issues have been occuring from time to time on different test scenarios, for example, the latest 9.x swarm:
https://product-ci.infra.mirantis.net/view/9.x_swarm/job/9.x.system_test.ubuntu.thread_7/99/testReport/(root)/deploy_neutron_tun_ha_nodegroups/deploy_neutron_tun_ha_nodegroups/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_public/96/testReport/(root)/deploy_neutron_tun_ha_with_public_network/deploy_neutron_tun_ha_with_public_network/

Error Message
Cluster is not deployed: some nodes are in the Error state

Such errors are due to mysql could not start during deploy
SLAVE_NODE_MEMORY=4096

[Alexandra (aallakhverdieva)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ha_neutron_destructive/102/testReport/(root)/ha_neutron_delete_vips/

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #549.

Actual results:
SLAVE_NODE_MEMORY=3968
No failures with this simptoms were found.