error during create-nested-snapshots-and-attach-volume rally light test

Bug #1413647 reported by Leontii Istomin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
Pavel Boldin

Bug Description

[root@fuel ~]# fuel --fuel-version
api: '1.0'
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
auth_required: true
build_id: 2015-01-21_08-13-56
build_number: '40'
feature_groups:
- mirantis
fuellib_sha: 9aa913096fb93ea4847ee14bfaf33597326886f3
fuelmain_sha: 1ee1766a51bdb5bed75d5c2efdcaaa318118e439
nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
ostf_sha: 3d2f44dcfa32d6ce0372cc64695e9edcc1913ea7
production: docker
release: 6.0.1
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
      build_id: 2015-01-21_08-13-56
      build_number: '40'
      feature_groups:
      - mirantis
      fuellib_sha: 9aa913096fb93ea4847ee14bfaf33597326886f3
      fuelmain_sha: 1ee1766a51bdb5bed75d5c2efdcaaa318118e439
      nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
      ostf_sha: 3d2f44dcfa32d6ce0372cc64695e9edcc1913ea7
      production: docker
      release: 6.0.1

baremetal, Centos, HA, neutron-gre, Ceilometer, debug, Ceph for volumes,images,ephemerals,objects
controllers: 3
computes: 95

rally-scenarios.cinder.create-nested-snapshots-and-attach-volume [4 iterations, 1 threads]
Traceback (most recent call last):
_StringException: config: {{{
{
  "kw": {
    "runner": {
      "type": "constant",
      "concurrency": 1,
      "times": 4
    },
    "args": {
      "nested_level": {
        "max": 3,
        "min": 2
      },
      "size": {
        "max": 5,
        "min": 1
      }
    },
    "context": {
      "users": {
        "project_domain": "default",
        "users_per_tenant": 1,
        "user_domain": "default",
        "tenants": 2,
        "resource_management_workers": 30
      },
      "quotas": {
        "cinder": {
          "gigabytes": -1,
          "volumes": -1,
          "snapshots": -1
        }
      },
      "servers": {
        "servers_per_tenant": 2,
        "flavor": {
          "name": "m1.tiny"
        },
        "image": {
          "name": "TestVM"
        }
      }
    }
  },
  "name": "CinderVolumes.create_nested_snapshots_and_attach_volume",
  "pos": 0
}
}}}

results: {{{
+----------------------------+--------+---------+---------+----------+-------+--------+
| Action | min | avg | max | failures | total | result |
+----------------------------+--------+---------+---------+----------+-------+--------+
| cinder.create_snapshot | 2.352 | 2.352 | 2.352 | 2 | 4 | :( |
| cinder.create_snapshot (2) | 2.236 | 4.481 | 6.727 | 2 | 4 | :( |
| cinder.create_snapshot (3) | 2.264 | 2.264 | 2.264 | 3 | 4 | :( |
| cinder.create_volume | 2.531 | 2.556 | 2.580 | 2 | 4 | :( |
| cinder.create_volume (2) | 2.524 | 2.604 | 2.684 | 2 | 4 | :( |
| cinder.create_volume (3) | 4.444 | 4.444 | 4.444 | 3 | 4 | :( |
| nova.attach_volume | 5.552 | 6.490 | 7.427 | 2 | 4 | :( |
| nova.attach_volume (2) | 3.125 | 4.096 | 5.067 | 2 | 4 | :( |
| nova.attach_volume (3) | 4.882 | 4.882 | 4.882 | 3 | 4 | :( |
| nova.detach_volume | 2.707 | 2.753 | 2.798 | 2 | 4 | :( |
| ~total | 600 | 616.177 | 640.196 | 4 | 4 | :( |
+----------------------------+--------+---------+---------+----------+-------+--------+
}}}

stderr-0000: {{{
TimeoutError

Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/constant.py", line 88, in _run_scenario
    result = iter_result.next(timeout)
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 652, in next
    raise TimeoutError
TimeoutError
}}}

stderr-0001: {{{
TimeoutException
Timeout exceeded.
Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 77, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/volumes.py", line 304, in create_nested_snapshots_and_attach_volume
    self._delete_snapshot(snapshot)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 254, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/utils.py", line 169, in _delete_snapshot
    check_interval=CONF.benchmark.cinder_volume_delete_poll_interval
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 135, in wait_for_delete
    raise exceptions.TimeoutException()
TimeoutException: Timeout exceeded.
}}}

stderr-0002: {{{
TimeoutError

Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/constant.py", line 88, in _run_scenario
    result = iter_result.next(timeout)
  File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 652, in next
    raise TimeoutError
TimeoutError
}}}

stderr-0003: {{{
TimeoutException
Timeout exceeded.
Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 77, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/volumes.py", line 304, in create_nested_snapshots_and_attach_volume
    self._delete_snapshot(snapshot)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 254, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/cinder/utils.py", line 169, in _delete_snapshot
    check_interval=CONF.benchmark.cinder_volume_delete_poll_interval
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 135, in wait_for_delete
    raise exceptions.TimeoutException()
TimeoutException: Timeout exceeded.
}}}

no longer affects: fuel
Changed in mos:
assignee: nobody → Ivan Kolodyazhny (e0ne)
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

Looks like an issue in rally scenario. i've got other error on my local lab.

Changed in mos:
importance: Undecided → Medium
tags: added: cinder
Changed in mos:
status: New → Confirmed
milestone: none → 6.1
Ivan Kolodyazhny (e0ne)
Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

Some kernel logs and related thread about the same problem in CentOS w/o OpenStack:
Feb 12 14:06:42 node-3 kernel: INFO: task lvremove:19034 blocked for more than 120 seconds.
Feb 12 14:06:42 node-3 kernel: Not tainted 2.6.32-504.1.3.el6.x86_64 #1
Feb 12 14:06:42 node-3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 12 14:06:42 node-3 kernel: lvremove D 0000000000000006 0 19034 19033 0x00000080
Feb 12 14:06:42 node-3 kernel: ffff88086a023a18 0000000000000086 0000000000000000 ffffffff8126b3e4
Feb 12 14:06:42 node-3 kernel: ffff88086a097ec0 ffff880873369500 00004fa672027db8 ffffffffa000461c
Feb 12 14:06:42 node-3 kernel: ffff88086a0239d8 000000010532a906 ffff8807d0471af8 ffff88086a023fd8
Feb 12 14:06:42 node-3 kernel: Call Trace:
Feb 12 14:06:42 node-3 kernel: [<ffffffff8126b3e4>] ? blk_unplug+0x34/0x70
Feb 12 14:06:42 node-3 kernel: [<ffffffffa000461c>] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Feb 12 14:06:42 node-3 kernel: [<ffffffff81529643>] io_schedule+0x73/0xc0
Feb 12 14:06:42 node-3 kernel: [<ffffffff811ce7dd>] __blockdev_direct_IO_newtrunc+0xb7d/0x1270
Feb 12 14:06:42 node-3 kernel: [<ffffffff811ca140>] ? blkdev_get_block+0x0/0x20
Feb 12 14:06:42 node-3 kernel: [<ffffffff811cef47>] __blockdev_direct_IO+0x77/0xe0
Feb 12 14:06:42 node-3 kernel: [<ffffffff811ca140>] ? blkdev_get_block+0x0/0x20
Feb 12 14:06:42 node-3 kernel: [<ffffffff811cb1c7>] blkdev_direct_IO+0x57/0x60
Feb 12 14:06:42 node-3 kernel: [<ffffffff811ca140>] ? blkdev_get_block+0x0/0x20
Feb 12 14:06:42 node-3 kernel: [<ffffffff8112616b>] generic_file_aio_read+0x6bb/0x700
Feb 12 14:06:42 node-3 kernel: [<ffffffff811cbd10>] ? blkdev_get+0x10/0x20
Feb 12 14:06:42 node-3 kernel: [<ffffffff811cbd20>] ? blkdev_open+0x0/0xc0
Feb 12 14:06:42 node-3 kernel: [<ffffffff8118b17f>] ? __dentry_open+0x23f/0x360
Feb 12 14:06:42 node-3 kernel: [<ffffffff811ca581>] blkdev_aio_read+0x51/0x80
Feb 12 14:06:42 node-3 kernel: [<ffffffff8118de9a>] do_sync_read+0xfa/0x140
Feb 12 14:06:42 node-3 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
Feb 12 14:06:42 node-3 kernel: [<ffffffff811ca4dc>] ? block_ioctl+0x3c/0x40
Feb 12 14:06:42 node-3 kernel: [<ffffffff811a3772>] ? vfs_ioctl+0x22/0xa0
Feb 12 14:06:42 node-3 kernel: [<ffffffff811a3914>] ? do_vfs_ioctl+0x84/0x580
Feb 12 14:06:42 node-3 kernel: [<ffffffff8122d466>] ? security_file_permission+0x16/0x20
Feb 12 14:06:42 node-3 kernel: [<ffffffff8118e855>] vfs_read+0xb5/0x1a0
Feb 12 14:06:42 node-3 kernel: [<ffffffff8118e991>] sys_read+0x51/0x90
Feb 12 14:06:42 node-3 kernel: [<ffffffff810e5a7e>] ? __audit_syscall_exit+0x25e/0x290
Feb 12 14:06:42 node-3 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

http://forum.proxmox.com/threads/14285-lvremove-error-on-a-backup-and-hight-IO-Delays-load-average

Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

mos-linux team,
please take a look on it.

Changed in mos:
assignee: Ivan Kolodyazhny (e0ne) → MOS Linux (mos-linux)
importance: Medium → High
Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

Ivan, how can we reproduce that error without OpenStack (question is related to comment #2 in this bug)?

Changed in mos:
status: In Progress → Incomplete
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

Alexander, you could try to reproduce it using provided link in comment above. For now, it's reproduced only on a real hardware at Scale lab with Rally. I can't reproduce it on a virtual environment.

The same issue on ubuntu we fixed by updating lvm2 package: https://bugs.launchpad.net/mos/+bug/1369524

Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

Moving to confirmed to get attention of MOS Linux team

Changed in mos:
status: Incomplete → Confirmed
Revision history for this message
Michael Semenov (msemenov) wrote :
Changed in mos:
assignee: MOS Linux (mos-linux) → Pavel Boldin (pboldin)
Revision history for this message
Pavel Boldin (pboldin) wrote :

Can you please provide me with an exact kernel version?

Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

Pavel, it should be the same version as in MOS 6.1

tags: added: linux
tags: added: non-release
Revision history for this message
Pavel Boldin (pboldin) wrote :

Seems to be fixed by the LVM update. Unable to reproduce anymore.

Changed in mos:
status: Confirmed → Invalid
Pavel Boldin (pboldin)
Changed in mos:
status: Invalid → Incomplete
Pavel Boldin (pboldin)
Changed in mos:
milestone: 6.1 → 7.0
Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

Changing the status, since it was not reproduced.

Changed in mos:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.