[Testrail] universal tool for publishing test results to TestRail

[Denis Deryabin]

It's necessary to report plugin tests results from Jenkins or other source (i.e. xunit reports) to TestRail.
Current sollution doesn't allow to use another sources of test results instead of Jenkins jobs.

[Nastya Urlapova (aurlapova)]

@Dennis, very hard to understand that exactly you want to do, what current solution doesn't allow?

[Denis Deryabin (ehles)]

For example, we could run automated test on local test environment and get nosetests.xml as test results.
Current sollution doesn't allow to use nosetests.xml as source for test results.

[okosse (okosse)]

Nastya, On the other hand our teams need to manipulate variables for testrail (milestones, configurations etc) via external config file or environment variables

[Artem Panchenko (apanchenko-8)]

Guys,

it's a very good initiative to create universal tool for reporting, so it will cover requirements of all QA teams and provide unified interface for working with TestRail. But there are 2 important points which you need to consider while developing it:

1) new tool must fully replace currently used customized scripts [0]. It means you should agreed with at least Fuel QA and MOS QA teams on what functionality it must provide. For example, you mentioned above that you want to add an ability to fetch tests results from XML, but other teams also need to use one (or pool) of Jenkins job(s) as source or raw JSON. All such stuff should be discussed and documented somewhere.

2) fuel-qa is under big tent now, so we can't just contribute all the stuff we use in our daily work there. Generally it should contain only things which are useful for community ( automatic tests for Fuel and its components). That's why I propose to create a separate internal repository for new testrail reporter and remove testrail related code from fuel-qa. I think we can make it public and probably publish on PyPI in the future.

[0] https://github.com/openstack/fuel-qa/tree/master/fuelweb_test/testrail

[Denis Deryabin (ehles)]

Artem, yes that is what I want.

1) Current sollution support only jenkins xUnit test results and
    doesn't support testrail configuration options.
    I see this sollution like:
    [Jenkins xUnit]--\ /--[TestRail]
                      |--[ITRR]--|
    [nosetests.xml]--/ \--[TestLink, etc]

    The script may convert test result into Internal
    Test Result Represenation (ITRR) and pass that to the destionation
    tool like TestRail, TestLink, etc.

    so, on the firt step we could add support only TestRail as a
    destination for testresults.

2) Yes, aggree that unified sollution shoud be in a separate
    repository.

I've posted the draft for this sollution as pull-request here [0]
    and as a different repository here [1].

[0] https://review.openstack.org/#/c/273015/
[1] https://github.com/ehles/trep

[Tatyanka (tatyana-leontovich)]

Guys, cool initiative with detailed comments, but it is not bug, it something more, so please move your cool idea into spec or blueprint

[OpenStack Infra (hudson-openstack)]

Change abandoned by Denis Deryabin (<email address hidden>) on branch: master
Review: https://review.openstack.org/273015
Reason: Moved to BP https://blueprints.launchpad.net/fuel/+spec/testrail-testreporter