MOS 5.1 CI RPM tests are failing

[Alexey Stupnikov]

Bulid team, please investigate the issue with RPM packages testing process in 5.1 CI tests. All tests are failing with the following errors in logs:

2016-02-12 08:12:21,111 - INFO regenerate_repo.py:291 -- Local "CentOS" repository /var/www/nailgun/centos/fuelweb/x86_64 has been updated successfuly.
2016-02-12 08:12:21,409 - INFO fuel_web_client.py:404 -- Deploy cluster 1
2016-02-12 08:12:21,409 - INFO fuel_web_client.py:422 -- Launch deployment of a cluster #1
2016-02-12 08:12:21,978 - INFO fuel_web_client.py:233 -- Assert task {u'status': u'running', u'name': u'deploy', u'cluster': 1, u'result': {}, u'progress': 0, u'message': None, u'id': 3, u'uuid': u'07159263-da1c-4059-9539-45829f0909f4'} is success
2016-02-12 08:12:21,978 - INFO fuel_web_client.py:591 -- Wait for task {u'status': u'running', u'name': u'deploy', u'cluster': 1, u'result': {}, u'progress': 0, u'message': None, u'id': 3, u'uuid': u'07159263-da1c-4059-9539-45829f0909f4'} 7800 seconds
2016-02-12 08:23:53,050 - INFO regenerate_repo.py:305 -- Error: Package: openstack-nova-common-2014.1.3-fuel5.1.1.mira14.git.5c8fa76.7ad20ad.noarch (nailgun)
 has unmet dependencies:
2016-02-12 08:23:53,050 - INFO regenerate_repo.py:307 -- Requires: lshell
2016-02-12 08:23:53,096 - INFO fuel_web_client.py:591 -- Wait for task {u'status': u'running', u'name': u'dump', u'cluster': None, u'result': {}, u'progress': 0, u'message': None, u'id': 8, u'uuid': u'c6220bf0-525e-441d-848a-376854bf5a14'} 300 seconds
2016-02-12 08:24:18,241 - INFO decorators.py:33 -- Saving logs to "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/logs/fail_error_deploy_simple_flat-2016_02_12__08_24_18.tar.gz" file
2016-02-12 08:25:12,322 - INFO decorators.py:117 -- <<<<<****************************************************************************************************>>>>>
2016-02-12 08:25:12,322 - INFO decorators.py:118 -- Failed in method 'deploy_simple_flat'. Make snapshot: error_deploy_simple_flat
2016-02-12 08:25:12,323 - INFO decorators.py:127 -- You could revert this snapshot using [dos.py revert package_testing-centos-5_1 --snapshot-name error_deploy_simple_flat && dos.py resume package_testing-centos-5_1 && virsh net-dumpxml package_testing-centos-5_1_admin | grep -P "(\d+\.){3}" -o | awk '{print "Admin node IP: "$0"2"}']
2016-02-12 08:25:12,323 - INFO decorators.py:132 -- <<<<<****************************************************************************************************>>>>>
FAIL

======================================================================
FAIL: Deploy cluster in simple mode with flat nova-network
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/helpers/decorators.py", line 52, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/tests/test_simple.py", line 110, in deploy_simple_flat
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/helpers/decorators.py", line 209, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/models/fuel_web_client.py", line 406, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_rpm_updates/fuel-main/fuelweb_test/models/fuel_web_client.py", line 239, in assert_task_success
    task['status'], 'ready', name=task["name"]
AssertionError: Task 'deploy' has incorrect status. error != ready

----------------------------------------------------------------------
Ran 4 tests in 974.529s

FAILED (SKIP=3, failures=1)

Example jobs:
http://osci-jenkins.srt.mirantis.net:8080/job/package_testing_rpm_updates/687/
http://osci-jenkins.srt.mirantis.net:8080/job/package_testing_rpm_updates/689/

Affected reviews:
https://review.fuel-infra.org/#/c/16893/
https://review.fuel-infra.org/#/c/16894/
https://review.fuel-infra.org/#/c/16895/

[Sergey Otpuschennikov (sotpuschennikov)]

Alexander, i see you copied lshell from http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-updates-stable/centos/ repository to http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-updates-stable-16893/centos/ for system test.
What do you think about add update repository (e.g http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-updates-stable/centos/) for system test ?
We could test work updated package with another updated packages - not only with packages from GA.

[Alexander Tsamutali (astsmtl)]

We already thought about this and came to conclusion that adding updates repository to tests in every request may solve some potential issues but introduce some others. One example is when user didn't want to install all updates but only selected, this will again lead to situations where his environment differs from our test environment. Taking into account that old releases receive only security updates, we decided to keep CI as it is and apply manual workarounds if necessary.