resource-class-in-use_create_a_resource_class fails with "MismatchError: '204' not in ['201']" since 7/22

[Matt Riedemann]

Seen here:

http://logs.openstack.org/02/485602/6/check/gate-nova-tox-functional-ubuntu-xenial/edf4c41/console.html#_2017-07-24_18_52_05_651293

2017-07-24 18:52:05.638101 | nova.tests.functional.api.openstack.placement.test_placement_api.resource-class-in-use_delete_resource_class.test_request
2017-07-24 18:52:05.638130 | -------------------------------------------------------------------------------------------------------------------------
2017-07-24 18:52:05.638135 |
2017-07-24 18:52:05.638143 | Captured traceback:
2017-07-24 18:52:05.638152 | ~~~~~~~~~~~~~~~~~~~
2017-07-24 18:52:05.638164 | Traceback (most recent call last):
2017-07-24 18:52:05.638201 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/gabbi/case.py", line 93, in wrapper
2017-07-24 18:52:05.638209 | func(self)
2017-07-24 18:52:05.638246 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/gabbi/case.py", line 141, in test_request
2017-07-24 18:52:05.638256 | self._run_test()
2017-07-24 18:52:05.638293 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/gabbi/case.py", line 499, in _run_test
2017-07-24 18:52:05.638304 | self._assert_response()
2017-07-24 18:52:05.638342 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/gabbi/case.py", line 183, in _assert_response
2017-07-24 18:52:05.638362 | self._test_status(self.test_data['status'], self.response['status'])
2017-07-24 18:52:05.638399 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/gabbi/case.py", line 539, in _test_status
2017-07-24 18:52:05.638419 | self.assert_in_or_print_output(observed_status, statii)
2017-07-24 18:52:05.638459 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/gabbi/case.py", line 600, in assert_in_or_print_output
2017-07-24 18:52:05.638469 | self.fail(msg)
2017-07-24 18:52:05.651245 | File "/home/jenkins/workspace/gate-nova-tox-functional-ubuntu-xenial/.tox/functional/local/lib/python2.7/site-packages/unittest2/case.py", line 690, in fail
2017-07-24 18:52:05.651275 | raise self.failureException(msg)
2017-07-24 18:52:05.651293 | AssertionError: '404' not found in ['204'], response:
2017-07-24 18:52:05.651299 | {
2017-07-24 18:52:05.651308 | "errors": [
2017-07-24 18:52:05.651315 | {
2017-07-24 18:52:05.651325 | "status": 404,
2017-07-24 18:52:05.651336 | "title": "Not Found",
2017-07-24 18:52:05.651358 | "detail": "The resource could not be found.\n\n Resource could not be found. ",
2017-07-24 18:52:05.651377 | "request_id": "req-15235c69-224a-4030-bf4e-5b16c3922fbd"
2017-07-24 18:52:05.651383 | }
2017-07-24 18:52:05.651390 | ]
2017-07-24 18:52:05.651396 | }
2017-07-24 18:52:05.651402 |
2017-07-24 18:52:05.651407 |
2017-07-24 18:52:05.651416 | Captured pythonlogging:
2017-07-24 18:52:05.651425 | ~~~~~~~~~~~~~~~~~~~~~~~
2017-07-24 18:52:05.651462 | 2017-07-24 18:30:03,444 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "DELETE /resource_classes/CUSTOM_GOLD" status: 404 len: 190 microversion: 1.10

This fails when nova.tests.functional.api.openstack.placement.test_placement_api.resource-class-in-use_create_a_resource_class.test_request fails with:

testtools.matchers._impl.MismatchError: '204' not in ['201']
2017-07-24 18:52:05.638002 |
2017-07-24 18:52:05.638007 |
2017-07-24 18:52:05.638016 | Captured pythonlogging:
2017-07-24 18:52:05.638026 | ~~~~~~~~~~~~~~~~~~~~~~~
2017-07-24 18:52:05.638061 | 2017-07-24 18:30:03,314 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "PUT /resource_classes/CUSTOM_GOLD" status: 204 len: 0 microversion: 1.10

So the PUT is expecting a 201 but it's getting a 204, and then the DELETE fails with a 404 when it's expecting a 204.

Looks like the failures would correlate with this change being merged on 7/23:

https://review.openstack.org/#/c/484154/

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22INFO%20%5Bnova.api.openstack.placement.requestlog%5D%20127.0.0.1%20%5C%5C%5C%22PUT%20%2Fresource_classes%2FCUSTOM_GOLD%5C%5C%5C%22%20status%3A%20204%20len%3A%200%20microversion%3A%201.10%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

19 hits in 7/22, check and gate, all failures.

[Matt Riedemann (mriedem)]

In the failed run, this class creates the same custom resource class:

2017-07-24 18:30:02.936984 | {3} nova.tests.functional.api.openstack.placement.test_placement_api.allocation-bad-class_create_a_resource_class.test_request [0.010374s] ... ok

And they are running on the same process. They should be separate databases in sqlite given the APIFixture used in the tests, but maybe eventlet thread goofiness messed up the responses.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/486782

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/486805

[Chris Dent (cdent)]

The second fix (in comment #3) is probably better: the root of the problem is re-use of the RC_CACHE.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/486782
Reason: Functional tests passed on https://review.openstack.org/#/c/486805/ and that's a cleaner solution so let's just go with that.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/486805
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=997ba03094062799349e8275026462c549d3989a
Submitter: Jenkins
Branch: master

commit 997ba03094062799349e8275026462c549d3989a
Author: Chris Dent <email address hidden>
Date: Tue Jul 25 01:59:52 2017 +0100

    [placement] Flush RC_CACHE after each gabbit sequence

    Each gabbi yaml file is supposed to be an isolated sequence. The
    fixtures were already making sure that the database was wiped, and the
    traits sync was reset, but not making sure the _RC_CACHE used to cache
    mappings between resource class strings and ids was flushed.

    This change clears the _RC_CACHE in the APIFixture stop_fixture method.

    For sake of information:
    Before fixing the issue it was confirmed by running the two gabbi files
    indicated in the bug report, in sequence, in a single process:

        .tox/functional/bin/python -m subunit.run discover \
            nova.tests.functional.api.openstack.placement \
            --load-list /tmp/runthese |subunit-trace

    Where runthese contains (with fully qualified package name prefixes):

    test_placement_api.allocation-bad-class_allocate_some_of_it_standard.test_request
    test_placement_api.resource-class-in-use_delete_resource_class.test_request

    Change-Id: Icb3beef7964887f29bf8cca95ac7dbae1511876d
    Closes-Bug: #1706207

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b3 development milestone.