Devstack tests fail from time to time

[Julien Danjou]

Example:

http://logs.openstack.org/17/156617/14/check/gate-gnocchi-dsvm-functional-ceph-mysql/6c198b9/console.html

This is random.

[Chris Dent (cdent)]

This is happening because the devstack plugin.sh creates archive policies and add some meters here:

https://github.com/stackforge/gnocchi/blob/master/devstack/plugin.sh#L293

Then the functional jobs go on to run the gabbi tests against the live gnocchi server, using the database that plugin.sh gave data to.

This leads to the gabbi tests (which were written to use a different database per file) having a different starting point than what they were expecting.

In the specific case of the tests I've seen, the problem is that a test which is using the first result in a list to generate a comparison and then a subsequent request. In the original testing situation there was only one result. When in the functional context there is at least two and the ordering is not predictable.

One possible fix is to get rid of (or make optional) the sample data created in plugin.sh. It would be possible to make the gabbi tests robust in this context if we use known ids or other ways of filtering in the tests but that will make them a bit more noisy and will make it harder to keep the tests consistent.

In fact, once there are several yaml files, it may get quite hard to guarantee the datasets in the functional context.

The underlying problem here is in the gabbi config fixture. It uses a different storage engine per file, but the same indexer. I'll explore changing that.

[Chris Dent (cdent)]

sorry, meant different indexer, same storage

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/167298

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/167298
Committed: https://git.openstack.org/cgit/stackforge/gnocchi/commit/?id=e5493bfee523190b0a3e8cb09a83854288cb34ae
Submitter: Jenkins
Branch: master

commit e5493bfee523190b0a3e8cb09a83854288cb34ae
Author: Chris Dent <email address hidden>
Date: Tue Mar 24 16:21:45 2015 +0000

    Correct handling of fixtures and server in gate tests

    There are multiple issues with the functional tests that this
    patchset attempts to resolve:

    * the gabbi yaml files are intended to be run with separate data
      stores, but against a live api server this is not straightforward
      so there can be collisions
    * conditional handling for the null:// indexer was busted
    * the plugin.sh checks for gnocchi-api startup by requesting a url
      that is returns 403 because of policy.json
    * the jenkins user is unable to write to the gnocchi data dir

    These are resolved by:

    * separating the py27-gate tox target tests into using live services
      for everything including web server and everything but web server
      by having two test generation files:

      * one generates tests from a gabbits-live directory only if a web
        server is running. These test use no fixtures, and thus rely only
        on the configuration of the api server. Only a very small number
        of tests are used (to be increased later) all from the same yaml
        file, so that they are all in the same test process.

      * the other uses existing gabbi yaml files with live storage and
        indexer backends, but using the fixtures to insure there are no
        collisions with the indexer (each yaml file gets a different
        indexer store)

    * Quieting some misleading curl output which suggests that the api
      is not responding approrpiately during the devstack plugin run

    * A bit of adjustment in the ConfigFixture to account for these
      changes including choosing to not override config settings if we
      know we are in a devstack gate setup.

    * Run the tests as the stack user not the jenkins user so that the
      already running services read/write permissions don't get
      confused. This is the advice from #openstack-qa when functional
      tests are invasive (which these are).

    When using a local devstack the py27-gate test can be checked with

        GNOCCHI_SERVICE_HOST=localhost GNOCCHI_SERVICE_PORT=8041 \
            DEVSTACK_GATE_TEMPEST=1 tox -epy27-gate

    Change-Id: I70897b32376b6cf0e87d1baba1537b95e78c2b4e
    Closes-Bug: #1435180