[testrail] Reporter fails if API returned "429 Too Many Requests"

[Artem Panchenko]

Sometimes TestRail API returns "429 Too Many Requests" error while uploading tests results from CI. Reporter script doesn't handle such cases, so it fails:

2015-11-10 10:23:28,264 - DEBUG - Looking for previous tests runs on "Ubuntu 14.04" using tests suite "Swarm 8.0"...
Traceback (most recent call last):
  File "fuelweb_test/testrail/report.py", line 571, in <module>
    main()
  File "fuelweb_test/testrail/report.py", line 551, in main
    results=tests_results[os['distro']]
  File "fuelweb_test/testrail/report.py", line 232, in publish_results
    config_id=config_id)
  File "/home/jenkins/workspace/8.0-test-reports-Testrail/fuelweb_test/testrail/testrail_client.py", line 238, in get_previous_runs
    for plan in self.get_plans_by_milestone(milestone_id=milestone_id):
  File "/home/jenkins/workspace/8.0-test-reports-Testrail/fuelweb_test/testrail/testrail_client.py", line 189, in get_plans_by_milestone
    if plan['milestone_id'] == milestone_id]
  File "/home/jenkins/workspace/8.0-test-reports-Testrail/fuelweb_test/testrail/testrail_client.py", line 184, in get_plan
    return self.client.send_get(plan_uri)
  File "/home/jenkins/workspace/8.0-test-reports-Testrail/fuelweb_test/testrail/testrail.py", line 53, in send_get
    return self.__send_request('GET', uri, None)
  File "/home/jenkins/workspace/8.0-test-reports-Testrail/fuelweb_test/testrail/testrail.py", line 98, in __send_request
    (e.code, error))
testrail.APIError: TestRail API returned HTTP 429 (No additional error message received)

According to TestRail documentation 'Such a response also includes a Retry-After header indicating how many seconds to wait before you are allowed to submit the next request'. That means we can try to add handler for such responses to the TestRail client which will print warning to log and wait for N seconds before sending new request.

[Timur Nurlygayanov (tnurlygayanov)]

This issue is blocker for MOS QA team because we can't upload results of Tempest tests to TestRail.

[Nastya Urlapova (aurlapova)]

Timur, from your comment it isn't clear why you faced issue exactly today, there was change with your jobs or other?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/244288

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/244288
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=0debb2816160b27660449cf16561191e75539d82
Submitter: Jenkins
Branch: master

commit 0debb2816160b27660449cf16561191e75539d82
Author: Artem Panchenko <email address hidden>
Date: Wed Nov 11 20:39:56 2015 +0200

    Add retries to TestRail client

    In order to workaround API limitations (such as
    requests rate), adding mechanism of requests
    retries based on predefined error codes.

    Try to resend request 3 times if '429' code is
    received from API.

    Closes-bug: #1514797
    Change-Id: I723ef7886c86b7e418627962a1ff8aa7a4281c0a

[Oleksiy Butenko (obutenko)]

http://paste.openstack.org/show/478736/
still reproduced

[Artem Panchenko (apanchenko-8)]

@Oleksiy,

please look carefully at the traceback you attached, it contains the following line:

---> 53 return self.__send_request('GET', uri, None)

It refers to line #53 of fuel-qa/fuelweb_test/testrail/testrail.py script, here is how that line currently looks at 'fuel-qa' master branch:

https://github.com/openstack/fuel-qa/blob/56f80160c7af56a13275d9bfc3d5af03b58aa472/fuelweb_test/testrail/testrail.py#L53

So obviously either you are using non-master branch of 'fuel-qa' repo or your version of code is outdated.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/244882

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Timur Nurlygayanov <email address hidden>
Review: https://review.fuel-infra.org/13903

[Timur Nurlygayanov (tnurlygayanov)]

Fix for Tempest tests execution on mosi jenkins: https://review.fuel-infra.org/13903

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13903
Submitter: Alexander Evseev <email address hidden>
Branch: master

Commit: 2daf1228494fc81dc04430e75a10ebd3ac0e2a4a
Author: Timur Nurlygayanov <email address hidden>
Date: Fri Nov 13 10:41:23 2015

Fixed issue with incorrect repository links

Change-Id: I466ef86b6228bad4f20dd4fee40744fe772087ae
Closes-Bug: #1514797

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/244882
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=227f6b1acd99129dddee4d1ae79586fcae2e3b87
Submitter: Jenkins
Branch: stable/7.0

commit 227f6b1acd99129dddee4d1ae79586fcae2e3b87
Author: Artem Panchenko <email address hidden>
Date: Wed Nov 11 20:39:56 2015 +0200

    Add retries to TestRail client

    In order to workaround API limitations (such as
    requests rate), adding mechanism of requests
    retries based on predefined error codes.

    Try to resend request 3 times if '429' code is
    received from API.

    Closes-bug: #1514797
    Change-Id: I723ef7886c86b7e418627962a1ff8aa7a4281c0a
    (cherry picked from commit 0debb2816160b27660449cf16561191e75539d82)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/245819

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/245819
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=ad12ece81e5249a26e0647c17bbfcfe3da26276d
Submitter: Jenkins
Branch: master

commit ad12ece81e5249a26e0647c17bbfcfe3da26276d
Author: Timur Nurlygayanov <email address hidden>
Date: Mon Nov 16 18:12:03 2015 +0300

    Fixed issue with 429 code for Tempest results

    We fixed issue with TestRail API 429 errors for
    SWARM tests, but for Tempest results we need to
    use minimum API requests and upload all results
    with only one bulk API request.

    I sent support request to TestRail support team
    and they provided me the information how we can
    change our API calls to TestRail to avoid any
    issues in the future.

    Change-Id: I7d8ffc3238eefe186f9ec904d8dca153527261ab
    Related-Bug: #1514797