Rally scen. CinderVolumes.create_and_delete_snapshot failed

[Sergey Galkin]

Rally scenario for cinder failed
---
  CinderVolumes.create_and_delete_snapshot:
    -
      args:
          force: false
      runner:
        type: "constant"
        times: {{ compute }}
        concurrency: {{ concurrency }}
      context:
        users:
          tenants: 2
          users_per_tenant: 2
        volumes:
          size: 1

errors:

03:17:01.527 CinderVolumes.create_and_delete_snapshot [97 iterations, 5 threads]
03:17:01.558 Traceback (most recent call last):
03:17:01.558 _StringException: iteration-0051-stderr: {{{
03:17:01.558 <class 'rally.exceptions.TimeoutException'>
03:17:01.558 Timeout exceeded.
03:17:01.558 Traceback (most recent call last):
03:17:01.558 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 73, in _run_scenario_once
03:17:01.559 method_name)(**kwargs) or scenario_output
03:17:01.559 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/volumes.py", line 86, in create_and_delete_snapshot
03:17:01.560 self._delete_snapshot(snapshot)
03:17:01.560 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 255, in func_atomic_actions
03:17:01.561 f = func(self, *args, **kwargs)
03:17:01.561 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/utils.py", line 145, in _delete_snapshot
03:17:01.562 check_interval=CONF.benchmark.cinder_volume_delete_poll_interval
03:17:01.563 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 147, in wait_for_delete
03:17:01.563 raise exceptions.TimeoutException()
03:17:01.563 TimeoutException: Timeout exceeded.
03:17:01.563 }}}
03:17:01.563
03:17:01.563 iteration-0094-stderr: {{{
03:17:01.563 <class 'rally.exceptions.TimeoutException'>
03:17:01.563 Timeout exceeded.
03:17:01.564 Traceback (most recent call last):
03:17:01.564 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 73, in _run_scenario_once
03:17:01.564 method_name)(**kwargs) or scenario_output
03:17:01.564 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/volumes.py", line 86, in create_and_delete_snapshot
03:17:01.564 self._delete_snapshot(snapshot)
03:17:01.564 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 255, in func_atomic_actions
03:17:01.565 f = func(self, *args, **kwargs)
03:17:01.565 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/utils.py", line 145, in _delete_snapshot
03:17:01.565 check_interval=CONF.benchmark.cinder_volume_delete_poll_interval
03:17:01.566 File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 147, in wait_for_delete
03:17:01.566 raise exceptions.TimeoutException()
03:17:01.566 TimeoutException: Timeout exceeded.
03:17:01.566 }}}
03:17:01.566
03:17:01.566 scenario-config: {{{
03:17:01.566 {
03:17:01.566 "kw": {
03:17:01.566 "runner": {
03:17:01.566 "type": "constant",
03:17:01.566 "concurrency": 5,
03:17:01.566 "times": 97
03:17:01.566 },
03:17:01.566 "args": {
03:17:01.566 "force": false
03:17:01.567 },
03:17:01.567 "context": {
03:17:01.567 "users": {
03:17:01.567 "project_domain": "default",
03:17:01.567 "concurrent": 30,
03:17:01.567 "users_per_tenant": 2,
03:17:01.567 "tenants": 2,
03:17:01.567 "user_domain": "default"
03:17:01.567 },
03:17:01.567 "volumes": {
03:17:01.567 "size": 1
03:17:01.567 }
03:17:01.567 }
03:17:01.567 },
03:17:01.567 "name": "CinderVolumes.create_and_delete_snapshot",
03:17:01.567 "pos": 0
03:17:01.567 }
03:17:01.568 }}}
03:17:01.568 summary: {{{
03:17:01.568 +------------------------+--------+--------+---------+----------+-------+--------+
03:17:01.568 | Action | min | avg | max | failures | total | result |
03:17:01.568 +------------------------+--------+--------+---------+----------+-------+--------+
03:17:01.568 | cinder.create_snapshot | 2.735 | 3.189 | 4.855 | 0 | 97 | :) |
03:17:01.568 | cinder.delete_snapshot | 20.795 | 46.222 | 52.753 | 2 | 97 | :( |
03:17:01.569 | ~total | 23.974 | 60.871 | 605.504 | 2 | 97 | :( |
03:17:01.569 +------------------------+--------+--------+---------+----------+-------+--------+
03:17:01.570 }}}

[Roman Podoliaka (rpodolyaka)]

May be a duplicate of https://bugs.launchpad.net/mos/+bug/1391852

[Sergey Galkin (sgalkin)]

Reproduced on
astute_sha: 8e05026cf17aeb92e2fb028b3b95ba34803508ed
auth_required: true
build_id: 2014-11-16_22-00-23
build_number: '110'
feature_groups:
- mirantis
fuellib_sha: 4c080377d5d1f4397a257e26c836055889fa3138
fuelmain_sha: 137d944d4a658e0358349db0daa2b9bc21ae8218
nailgun_sha: 3e98ea8a042b43be8c00b9f852e19b4c0b853404
ostf_sha: 82465a94eed4eff1fc8d8e1f2fb7e9993c22f068
production: docker
release: '6.0'

On scenario create-and-attach-volume
config
{
  "kw": {
    "runner": {
      "type": "constant",
      "concurrency": 5,
      "times": 77
    },
    "args": {
      "volume_size": 10,
      "flavor": {
        "name": "m1.nano"
      },
      "image": {
        "name": "TestVM"
      }
    },
    "context": {
      "quotas": {
        "cinder": {
          "volumes": -1
        }
      },
      "users": {
        "project_domain": "default",
        "users_per_tenant": 2,
        "user_domain": "default",
        "tenants": 2,
        "resource_management_workers": 30
      }
    }
  },
  "name": "CinderVolumes.create_and_attach_volume",
  "pos": 0
}
results
+----------------------+--------+---------+---------+----------+-------+--------+
| Action | min | avg | max | failures | total | result |
+----------------------+--------+---------+---------+----------+-------+--------+
| cinder.create_volume | 2.309 | 3.214 | 8.902 | 0 | 77 | :) |
| cinder.delete_volume | 62.441 | 236.594 | 390.664 | 1 | 77 | :( |
| nova.attach_volume | 4.829 | 5.418 | 16.022 | 0 | 77 | :) |
| nova.boot_server | 3.021 | 3.281 | 3.868 | 0 | 77 | :) |
| nova.delete_server | 2.363 | 2.484 | 2.656 | 1 | 77 | :( |
| nova.detach_volume | 2.577 | 2.660 | 2.815 | 0 | 77 | :) |
| ~total | 82.578 | 258.356 | 618.957 | 1 | 77 | :( |
+----------------------+--------+---------+---------+----------+-------+--------+
stderr-0067
<class 'rally.exceptions.TimeoutException'>
Timeout exceeded.
Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 73, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/volumes.py", line 115, in create_and_attach_volume
    self._delete_volume(volume)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 256, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/utils.py", line 100, in _delete_volume
    check_interval=CONF.benchmark.cinder_volume_delete_poll_interval
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 133, in wait_for_delete
    raise exceptions.TimeoutException()
TimeoutException: Timeout exceeded.

[Ivan Kolodyazhny (e0ne)]

Reproduced on:
root@node-78:~# logout
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "39"
  build_id: "2014-12-04_18-22-07"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "605ac6deffd601c1da809eb114cd315f670029d8"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "0df508c3247a06f8a636ded8fb04733d5aed7732"
  fuelmain_sha: "7ab330b4958ab20955372e85de05e8732e8f6df2"

 'creating' and command 'lvs' and 'vgs' also hang on nodes with snapshot in 'creating' status

[Aleksandr Shaposhnikov (alashai8)]

As far as I recall lvs hangs only if lvm manager lost control and/or get into spin lock on some procedure.
Example: delete volume and if by some reason it stuck on deleting then retying attempt to delete it will lead to described state.

[Ivan Kolodyazhny (e0ne)]

To fix it we need to get updated lvm2 package