Fuel-QA: OSTF tests fails after replace cluster repositories

[Tatyana Kuterina]

Detailed bug description:
Found on CI:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.multirole/14/console

Test: Deploy cluster with multiple services using local mirror
Test Group: deploy_multiple_services_local_mirror

Steps to reproduce:
        1. Revert snapshot 'prepare_slaves_5' with default set of mirrors
        2. Run 'fuel-mirror' to create mirror repositories
        3. Create cluster with many components to check as many
           packages in local mirrors have correct dependencies
        4. Run 'fuel-mirror' to replace cluster repositories
           with local mirrors
        5. Check that repositories are changed
        6. Deploy cluster
        7. Check running services with OSTF
Expected results:
   All OSTF tests are finished without errors
Actual result:
   Failed 2 OSTF tests with error 'Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.'

- Launch instance, create snapshot, launch instance from snapshot
- Create user and authenticate with it

Description of the environment:
9.1 snapshot #81

FUEL_QA_COMMIT=bfb750898b0f5ef196eb0c8a295cc29279487ade
UBUNTU_MIRROR_ID=ubuntu-2016-07-31-170655
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-08-01-154321
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-08-01-154321
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-07-18-162958
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNMHVlSU1nbDdLVmc/view?usp=sharing

[Georgy Kibardin (gkibardin)]

Keystone has returned 500 due to mysql:

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=1
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 76505 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f89667b5770
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f896400ee88 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x7f8964848b7c]
/usr/sbin/mysqld(handle_fatal_signal+0x3c2)[0x7f89645995c2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f896328e340]
/usr/sbin/mysqld(thd_get_ha_data+0xc)[0x7f89645e654c]
/usr/sbin/mysqld(_Z20thd_binlog_trx_resetP3THD+0x2e)[0x7f89647f379e]
/usr/sbin/mysqld(_Z17wsrep_post_commitP3THDb+0xcc)[0x7f89646d332c]
/usr/sbin/mysqld(_Z12trans_commitP3THD+0x6f)[0x7f89646b9dcf]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x38d1)[0x7f8964627851]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x3c8)[0x7f896462b9d8]
/usr/sbin/mysqld(+0x508c24)[0x7f896462bc24]
/usr/sbin/mysqld(_Z19do_handle_bootstrapP3THD+0x111)[0x7f896462bff1]
/usr/sbin/mysqld(+0x509060)[0x7f896462c060]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f8963286182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f89629a947d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f893c0009f0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

[Georgy Kibardin (gkibardin)]

It turned out that the crash above happens all the time at startup and, actually, not the crash at all, i.e. mysql is not dead by the time.
The problem is that mysql has been killed by OOM.

<3>Aug 2 02:31:24 node-4 kernel: [ 4150.801226] Out of memory: Kill process 22915 (mysqld) score 38 or sacrifice child
<3>Aug 2 02:31:24 node-4 kernel: [ 4150.802603] Killed process 22915 (mysqld) total-vm:2559656kB, anon-rss:107812kB, file-rss:0kB

[Georgy Kibardin (gkibardin)]

The seem to be no outstanding memory hog so that we may fix it. By now the simplest solution looks most reasonable - add up some memory.