tempest.lib.exceptions.PreconditionFailed: Precondition Failed on standalone-full-tempest-api-master

[Bhagyashri Shewale]

Error logs:

ft1.7: neutron_tempest_plugin.api.test_revisions.TestRevisions.test_update_network_constrained_by_revision[id-4a26a4be-9c53-483c-bc50-b11111113333]testtools.testresult.real._StringException: pythonlogging:'': {{{
2020-12-15 19:33:06,577 211730 INFO [tempest.lib.common.rest_client] Request (TestRevisions:test_update_network_constrained_by_revision): 201 POST http://192.168.24.3:9696/v2.0/networks 0.749s
2020-12-15 19:33:06,577 211730 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"network": {"name": "tempest-test-network--2021099203"}}
    Response - Headers: {'content-type': 'application/json', 'content-length': '626', 'x-openstack-request-id': 'req-1548254b-2055-4bad-a701-967ac66d821e', 'date': 'Tue, 15 Dec 2020 19:33:06 GMT', 'connection': 'close', 'status': '201', 'content-location': 'http://192.168.24.3:9696/v2.0/networks'}
        Body: b'{"network":{"id":"cef0d923-c7a0-44e3-a796-d17fd6b6d895","name":"tempest-test-network--2021099203","tenant_id":"2e6b43d3504c424a9740166604033168","admin_state_up":true,"mtu":1442,"status":"ACTIVE","subnets":[],"shared":false,"project_id":"2e6b43d3504c424a9740166604033168","qos_policy_id":null,"port_security_enabled":true,"dns_domain":"","router:external":false,"availability_zone_hints":[],"is_default":false,"availability_zones":[],"ipv4_address_scope":null,"ipv6_address_scope":null,"description":"","l2_adjacency":true,"tags":[],"created_at":"2020-12-15T19:33:05Z","updated_at":"2020-12-15T19:33:05Z","revision_number":1}}'
2020-12-15 19:33:06,893 211730 INFO [tempest.lib.common.rest_client] Request (TestRevisions:test_update_network_constrained_by_revision): 412 PUT http://192.168.24.3:9696/v2.0/networks/cef0d923-c7a0-44e3-a796-d17fd6b6d895 0.314s
2020-12-15 19:33:06,893 211730 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'If-Match': 'revision_number=0', 'X-Auth-Token': '<omitted>'}
        Body: {"network": {"name": "newnet"}}
    Response - Headers: {'content-length': '132', 'content-type': 'application/json', 'x-openstack-request-id': 'req-6892d0e6-7425-4eb3-ad04-c6791f805047', 'date': 'Tue, 15 Dec 2020 19:33:06 GMT', 'connection': 'close', 'status': '412', 'content-location': 'http://192.168.24.3:9696/v2.0/networks/cef0d923-c7a0-44e3-a796-d17fd6b6d895'}
        Body: b'{"NeutronError": {"type": "RevisionNumberConstraintFailed", "message": "Constrained to 0, but current revision is 1", "detail": ""}}'
2020-12-15 19:33:07,060 211730 INFO [tempest.lib.common.rest_client] Request (TestRevisions:test_update_network_constrained_by_revision): 200 GET http://192.168.24.3:9696/v2.0/networks/cef0d923-c7a0-44e3-a796-d17fd6b6d895 0.166s
2020-12-15 19:33:07,060 211730 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'content-type': 'application/json', 'content-length': '607', 'x-openstack-request-id': 'req-15563484-9fca-4cd0-9252-effc646c70f1', 'date': 'Tue, 15 Dec 2020 19:33:07 GMT', 'connection': 'close', 'status': '200', 'content-location': 'http://192.168.24.3:9696/v2.0/networks/cef0d923-c7a0-44e3-a796-d17fd6b6d895'}
        Body: b'{"network":{"id":"cef0d923-c7a0-44e3-a796-d17fd6b6d895","name":"tempest-test-network--2021099203","tenant_id":"2e6b43d3504c424a9740166604033168","admin_state_up":true,"mtu":1442,"status":"ACTIVE","subnets":[],"shared":false,"availability_zone_hints":[],"availability_zones":[],"ipv4_address_scope":null,"ipv6_address_scope":null,"router:external":false,"description":"","qos_policy_id":null,"port_security_enabled":true,"dns_domain":"","l2_adjacency":true,"tags":[],"created_at":"2020-12-15T19:33:05Z","updated_at":"2020-12-15T19:33:05Z","revision_number":1,"project_id":"2e6b43d3504c424a9740166604033168"}}'
2020-12-15 19:33:07,520 211730 INFO [tempest.lib.common.rest_client] Request (TestRevisions:test_update_network_constrained_by_revision): 412 PUT http://192.168.24.3:9696/v2.0/networks/cef0d923-c7a0-44e3-a796-d17fd6b6d895 0.459s
2020-12-15 19:33:07,521 211730 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'If-Match': 'revision_number=1', 'X-Auth-Token': '<omitted>'}
        Body: {"network": {"name": "newnet"}}
    Response - Headers: {'content-type': 'application/json', 'content-length': '132', 'x-openstack-request-id': 'req-d10865d0-75aa-4136-94b4-f7ff3a2059fb', 'date': 'Tue, 15 Dec 2020 19:33:07 GMT', 'connection': 'close', 'status': '412', 'content-location': 'http://192.168.24.3:9696/v2.0/networks/cef0d923-c7a0-44e3-a796-d17fd6b6d895'}
        Body: b'{"NeutronError": {"type": "RevisionNumberConstraintFailed", "message": "Constrained to 1, but current revision is 2", "detail": ""}}'
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/test_revisions.py", line 59, in test_update_network_constrained_by_revision
    headers={'If-Match': 'revision_number=%s' % current})
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 158, in _update
    resp, body = self.put(uri, post_data, headers=headers)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 362, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 702, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 828, in _error_checker
    raise exceptions.PreconditionFailed(resp_body, resp=resp)
tempest.lib.exceptions.PreconditionFailed: Precondition Failed
Details: {'type': 'RevisionNumberConstraintFailed', 'message': 'Constrained to 1, but current revision is 2', 'detail': ''}

[1]: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main&job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master

[2]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master/08b1a26/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Bhagyashri Shewale (bhagyashri-shewale)]

Hi All,

Related review: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/768204

[Takashi Kajinami (kajinamit)]

This is currently blocking TripleO job for puppet-neutron (puppet-neutron-tripleo-standalone).

It seems that the change in openstack-tempest-skiplist doesn't work for that job, so I submitted a patch to disable neutron_tempest_plugin in TripleO job .
 https://review.opendev.org/c/openstack/puppet-neutron/+/769071

[chandan kumar (chkumar246)]

Added the proper skip here: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/769137

[Slawek Kaplonski (slaweq)]

Patch which broke it is https://review.opendev.org/c/openstack/neutron/+/715315

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

Neutron service "revisions" is checking the OVO revision number any time a transaction is done in the DB. When OVN backend is used, more than one DB transaction is done to the DB with the same session.

Because of the transaction constraint (OVO standard attributes revision number), the specific requested OVO (in this case a network register) is retrieved from the DB context session. If this OVO is not present, a query to the DB is done [1].

When the network is updated, the revision number is updated too. During the second DB transaction, the network OVO (not present in the session) is retrieved from the DB but at this time the OVO revision number is +1 compared to the requested revision number.

A possible proposed solution [2] is to check the transaction constraint only once and the discard it. This proposal is valid because the transaction constraint refers only to a single register. If the constraint is validated once, there is no need to recheck it again.

Regards.

[1]https://github.com/openstack/neutron/blob/b2389a31a0871e22244232e0db9aba395cb35b49/neutron/services/revisions/revision_plugin.py#L241-L251
[2]https://review.opendev.org/c/openstack/neutron/+/769390

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

[Ronelle Landy (rlandy)]

Fixed in neutron - no trace in skiplist - closing out tripleo branch