Rally error "Failed to delete network for tenant" with scenario VMTasks.boot_runcommand_delete

[Alexander Evseev]

# fuel --fuel-version
api: '1.0'
astute_sha: 1be5b9b827f512d740fe907c7ff72486d4030938
auth_required: true
build_id: 2015-03-02_14-00-04
build_number: '154'
feature_groups:
- mirantis
fuellib_sha: b17e3810dbca407fca2a231c26f553a46e166343
fuelmain_sha: baf24424a4e056c6753913de5f8c94851903f718
nailgun_sha: f034fbb4b68be963e4dc5b5d680061b54efbf605
ostf_sha: 103d6cf6badd57b791cfaf4310ec8bd81c7a8a46
production: docker
python-fuelclient_sha: 3ebfa9c14a192d0298ff787526bf990055a23694
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1be5b9b827f512d740fe907c7ff72486d4030938
      build_id: 2015-03-02_14-00-04
      build_number: '154'
      feature_groups:
      - mirantis
      fuellib_sha: b17e3810dbca407fca2a231c26f553a46e166343
      fuelmain_sha: baf24424a4e056c6753913de5f8c94851903f718
      nailgun_sha: f034fbb4b68be963e4dc5b5d680061b54efbf605
      ostf_sha: 103d6cf6badd57b791cfaf4310ec8bd81c7a8a46
      production: docker
      python-fuelclient_sha: 3ebfa9c14a192d0298ff787526bf990055a23694
      release: '6.1'

Ubuntu, HA, neutron + VLAN, Ceph for volumes, images, ephemeral, objects
Controllers: 3, computes: 47

Scenario VMTasks.boot_runcommand_delete fail with timeout:

Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 77, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/vm/vmtasks.py", line 108, in boot_runcommand_delete
    password, interpreter, script)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/vm/utils.py", line 63, in run_command
    self.wait_for_ping(server_ip)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 254, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/vm/utils.py", line 51, in wait_for_ping
    timeout=120
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 109, in wait_for
    raise exceptions.TimeoutException()
TimeoutException: Timeout exceeded.

Rally's log contain errors like:

2015-03-10 22:15:55.181 32756 DEBUG neutronclient.client [-] RESP:409 {'date': 'Tue, 10 Mar 2015 22:15:55 GMT', 'connection': 'close', 'content-type': 'application/json; charset=UTF-8', 'content-length': '204', 'x-openstack-request-id': 'req-5f0bce57-ed0c-4fff-904d-a71ba37a30d6'} {"NeutronError": {"message": "Unable to complete operation on subnet ef8cae84-92d1-4c1d-ac2d-9c48a8af6d58. One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}}
 http_log_resp /opt/stack/.venv/lib/python2.7/site-packages/neutronclient/common/utils.py:139
2015-03-10 22:15:55.181 32756 DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"message": "Unable to complete operation on subnet ef8cae84-92d1-4c1d-ac2d-9c48a8af6d58. One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}} _handle_fault_response /opt/stack/.venv/lib/python2.7/site-packages/neutronclient/v2_0/client.py:173
2015-03-10 22:15:55.183 32756 ERROR rally.benchmark.context.network [-] Failed to delete network for tenant 01c5e2e04cad4ec586ac736f4e9c1433
 reason: Unable to complete operation on subnet ef8cae84-92d1-4c1d-ac2d-9c48a8af6d58. One or more ports have an IP allocation from this subnet.

[Alexander Evseev (aevseev)]

Diagnostic snapshot will be later

[Leontii Istomin (listomin)]

DG is here https://drive.google.com/a/mirantis.com/file/d/0Bx4ptZV1Jt7heVMteUJpdjdDeTg/view?usp=sharing

[Eugene Nikanorov (enikanorov)]

This is most probably a manifestation of this bug: https://bugs.launchpad.net/tempest/+bug/1357055

It's relatively rare condition that happens during fast-running API tests. It makes sense to not push the fix in 6.1 and wait until L.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/9714

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/9714
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 45b02339966c2b1e76a832aa705d41d3a91cc30d
Author: Oleg Bondarev <email address hidden>
Date: Tue Jul 21 15:05:27 2015

Handle race condition on subnet-delete

This fix targets quite rare case of race condition between
port creation and subnet deletion. This usually happens
during API tests that do things quickly.
DHCP port is being created after delete_subnet checks for
DHCP ports, but before it checks for IPAllocations on subnet.
The solution is to apply retrying logic, which is really necessary
as we can't fetch new IPAllocations with the same query and within
the active transaction in mysql because of REPEATABLE READ
transaction isolation.

The fix is composed from 2 upstream reviews:
https://review.openstack.org/171848
https://review.openstack.org/175125

Closes-Bug: #1430832
Change-Id: Ib9da018e654cdee3b64aa38de90f171c92ee28ee

[Timur Nurlygayanov (tnurlygayanov)]

I have executed this scenario on MOS 7.0 ISO #265 and it works fine without any errors during networks deletion, looks like issue was successfully fixed. Status changed to Fix Released.