[system-test] Not enough 600 sec to collect log snapshot on huge environments

[Dmitry Tyzhnenko]

Found on CI - https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.huge_ha_neutron/77/console

Fuel couldn't collect diagnostic snapshot on huge environment for a 600 sec and the system tests aborts task by timeout

2015-09-02 03:41:29,440 - ERROR decorators.py:93 -- Fetching of diagnostic snapshot failed: Traceback (most recent call last):
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/helpers/decorators.py", line 90, in wrapper
    "fail", name)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/helpers/decorators.py", line 343, in create_diagnostic_snapshot
    task = env.fuel_web.task_wait(env.fuel_web.client.generate_logs(), 60 * 10)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/models/fuel_web_client.py", line 1189, in task_wait
    "was exceeded: ".format(task=task["name"], timeout=timeout))
TimeoutError: Waiting task "dump" timeout 600 sec was exceeded:

2015-09-02 06:23:00,609 - ERROR decorators.py:122 -- Fetching of diagnostic snapshot failed: Traceback (most recent call last):
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/helpers/decorators.py", line 119, in wrapper
    func.__name__)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/helpers/decorators.py", line 343, in create_diagnostic_snapshot
    task = env.fuel_web.task_wait(env.fuel_web.client.generate_logs(), 60 * 10)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/models/fuel_web_client.py", line 1189, in task_wait
    "was exceeded: ".format(task=task["name"], timeout=timeout))
TimeoutError: Waiting task "dump" timeout 600 sec was exceeded:

We should increase timeout in tests with huge environments

Fuel ISO 7.0-262

ISO version: {
 "build_id": "262",
 "build_number": "262",
 "auth_required": true,
 "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4",
 "fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0",
 "nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468",
 "openstack_version": "2015.1.0-7.0",
 "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
 "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
 "api": "1.0",
 "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512",
 "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25",
 "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535",
 "feature_groups": [
  "mirantis"
 ],
 "release": "7.0",
 "release_versions": {
  "2015.1.0-7.0": {
   "VERSION": {
    "build_id": "262",
    "build_number": "262",
    "fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0",
    "nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468",
    "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4",
    "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
    "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
    "api": "1.0",
    "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512",
    "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25",
    "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535",
    "feature_groups": [
     "mirantis"
    ],
    "release": "7.0",
    "openstack_version": "2015.1.0-7.0",
    "production": "docker"
   }
  }
 },
 "production": "docker"
}

[Dmitry Tyzhnenko (dtyzhnenko)]

Not reproduced. It may be has happened because CI slave had huge load.

Moved to invalid.

[Denis Meltsaykin (dmeltsaykin)]

This is actually a valid bug. We have recently increased the count of files included into a diagnostic snapshot and are getting this problem now almost every run of SWARM. I'm going to increase the timeout to 30 mins.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable-mu/8.0
Review: https://review.openstack.org/380362

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/380362
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=c58f84c2c03725e63ebd54ebfc852fd6c846f043
Submitter: Jenkins
Branch: stable-mu/8.0

commit c58f84c2c03725e63ebd54ebfc852fd6c846f043
Author: Denis V. Meltsaykin <email address hidden>
Date: Fri Sep 30 17:50:47 2016 +0300

    Increase the timeout of diag snapshot generation

    As we have recently added new files to diagnostic snapshots the snapshot
    generation procedure might time out on big environments.
    The commit increases the timeout for the diagnostic snapshot generation
    procedure from 10 to 30 minutes. It helps avoid growing amount of
    timed outs during snapshot compressing.

    Change-Id: Iab1347e1af68e5e67fca819ef00e7741df4cb2ae
    Closes-Bug: #1491356

[Ekaterina Shutova (eshutova)]

Reproduced in: https://patching-ci.infra.mirantis.net/view/8.0.swarm/job/8.0.system_test.ubuntu.huge_ha_neutron/61/console

2017-01-26 14:43:53,004 - ERROR decorators.py:95 -- Fetching of diagnostic snapshot failed: Traceback (most recent call last):
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/helpers/decorators.py", line 92, in wrapper
    create_diagnostic_snapshot(args[0].env, "fail", name)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.huge_ha_neutron/fuelweb_test/helpers/decorators.py", line 332, in create_diagnostic_snapshot
    "Generation of diagnostic snapshot failed: {}".format(task))
  File "/home/jenkins/qa-venv-8.0/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Generation of diagnostic snapshot failed: {u'status': u'error', u'name': u'dump', u'cluster': None, u'result': {}, u'progress': 100, u'message': u'Dump is timed out', u'id': 5, u'uuid': u'080cd096-4faf-47ea-8584-7e1836d8dd29'}

[Ekaterina Shutova (eshutova)]

Verified latest runs on 8.0 mu4:
https://patching-ci.infra.mirantis.net/view/8.0.swarm/job/8.0.system_test.ubuntu.huge_ha_neutron/63/
No problems with diagnostic snapshots generation in specified timeout 1800sec.