[7.0] 7.0.system_test.ubuntu.plugins.thread_db_separate_services fails

[Denis Meltsaykin]

When we're running big swarm tests for Fuel 7.0 we are getting from `7.0.system_test.ubuntu.plugins.thread_db_separate_services` the following error:

======================================================================
FAIL: Deploy cluster with 3 separate database roles
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/tests/tests_separate_services/test_separate_db.py", line 105, in separate_db_service
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/helpers/decorators.py", line 460, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/helpers/decorators.py", line 445, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/helpers/decorators.py", line 496, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/helpers/decorators.py", line 503, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/helpers/decorators.py", line 387, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/models/fuel_web_client.py", line 877, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/__init__.py", line 58, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.plugins.thread_db_separate_services/fuelweb_test/models/fuel_web_client.py", line 307, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Tasks 'setup_repositories' can't be in requires|required_for|groups|tasks for [primary-standalone-database, standalone-database] because they don't exist in the graph'

----------------------------------------------------------------------
Ran 8 tests in 3824.907s

https://patching-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.plugins.thread_db_separate_services/

If this failure is fixable, please fix it, otherwise the test should be excluded from pool.

[Denis Meltsaykin (dmeltsaykin)]

Snapshot is attached.

[Denis Meltsaykin (dmeltsaykin)]

FYI, also 7.0.system_test.ubuntu.plugins.thread_keystone_separate_services and 7.0.system_test.ubuntu.plugins.thread_2_separate_services behave the same.

https://patching-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.plugins.thread_2_separate_services/4//console
https://patching-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.plugins.thread_keystone_separate_services/4//console

[Denis Meltsaykin (dmeltsaykin)]

https://patching-ci.infra.mirantis.net/view/All/job/7.0.system_test.ubuntu.plugins.thread_db_separate_services
https://patching-ci.infra.mirantis.net/view/All/job/7.0.system_test.ubuntu.plugins.thread_2_separate_services
https://patching-ci.infra.mirantis.net/view/All/job/7.0.system_test.ubuntu.plugins.thread_keystone_separate_services

These tests are still failed, but the reason is new:
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method granular_deploy. Deployment failed on nodes 2. Inspect Astute logs for the details'
Could you please investigate?

[slava valyavskiy (slava-val-al)]

Hello,

https://patching-ci.infra.mirantis.net/view/All/job/7.0.system_test.ubuntu.plugins.thread_db_separate_services

Task has been failed:
Task name - "database_hiera_override.pp"
2016-02-04 11:29:36 +0000 Puppet (err): Could not set 'file' on ensure: No such file or directory - /etc/hiera/plugins/detach-database.yaml20160204-15452-nh6z5p.lock at 112:/etc/fuel/plugins/detach-database-1.1/database_hiera_override.pp

[Vadim Rovachev (vrovachev)]

Hello, Slava. Do you want to say that bug must have confirmed state again?

[slava valyavskiy (slava-val-al)]

Job fails for an other reason than previously. Looking into it.

[Vadim Rovachev (vrovachev)]

Thank you.

[slava valyavskiy (slava-val-al)]

https://patching-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.plugins.thread_keystone_separate_services/7/console
https://patching-ci.infra.mirantis.net/view/All/job/7.0.system_test.ubuntu.plugins.thread_2_separate_services/7/console

Following jobs have been failed due to jenkins related issues(Slave went offline during the build).

[slava valyavskiy (slava-val-al)]

Regression for https://patching-ci.infra.mirantis.net/view/All/job/7.0.system_test.ubuntu.plugins.thread_db_separate_services was introduced by https://review.openstack.org/#/c/275791/ and it's not connected with previous problem somehow. Please, create separate bug to track an issue.