[QA] can not create snapshot

[Stanislav Makar]

Can not create snapshot libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'

Before all worked fine.

Sometimes snapshot for one or more VMs are done but not for all VMs:

       SNAPSHOT NODES-NAME
  failed_snapshot admin, slave-01, slave-02

Upgraded devops
pip list | grep devops
devops (2.5.3)

OS. Ubuntu 14.04 (x86_64)

log:
dos.py snapshot --snapshot-name failed_snapshot env_grand
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
Traceback (most recent call last):
  File "/home/smakar/virt_env/devops-venv/bin/dos.py", line 22, in <module>
    Shell().execute()
  File "/home/smakar/virt_env/devops-venv/local/lib/python2.7/site-packages/devops/shell.py", line 30, in execute
    self.commands.get(self.params.command)(self)
  File "/home/smakar/virt_env/devops-venv/local/lib/python2.7/site-packages/devops/shell.py", line 79, in do_snapshot
    self.params.snapshot_name)
  File "/home/smakar/virt_env/devops-venv/local/lib/python2.7/site-packages/devops/models.py", line 138, in snapshot
    node.snapshot(name=name, description=description, force=force)
  File "/home/smakar/virt_env/devops-venv/local/lib/python2.7/site-packages/devops/models.py", line 360, in snapshot
    node=self, name=name, description=description)
  File "/home/smakar/virt_env/devops-venv/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
File "/home/smakar/virt_env/devops-venv/local/lib/python2.7/site-packages/devops/driver/libvirt/libvirt_driver.py", line 365, in node_create_snapshot
    domain.snapshotCreateXML(xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2290, in snapshotCreateXML
    if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
libvirt.libvirtError: operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'

[Igor Shishkin (teran)]

Where you've got it?

[Stanislav Makar (smakar)]

on my workstation
What information would you like to get ?
I can provide ssh to my ws if you need.

[Igor Shishkin (teran)]

Fuel QA guys are working on python-devops right now, so I believe they could help to investigate it.
DevOps team is not supporting on problems with workstations software.

[Artem Panchenko (apanchenko-8)]

@Stanislav,

do you get the same error while creating VM snapshot using another tool, e.g. virsh (virsh snapshot-create-as --domain env_grand_admin --name snapshot1) ? It seems there is something wrong with libvirt or python-libvirt library on your PC. Please check logs in /var/log/libvirt/ for errors and try to update python-libvirt package to the latest version.

[Stanislav Makar (smakar)]

I tried and the same
virsh -d0 snapshot-create-as --domain env_grand_slave-02 --name test_snap
snapshot-create-as: domain(optdata): env_grand_slave-02
snapshot-create-as: name(optdata): test_snap
snapshot-create-as: found option <domain>: env_grand_slave-02
snapshot-create-as: <domain> trying as domain NAME
error: operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'

There is one interesting thing: if VM is down -- all is ok
This error appears only if VM is up

And second - with admin VM all is ok in all states.
It looks likes that snapshot creation always stucks on second drive.

[Oleksandr Martsyniuk (omartsyniuk)]

Im expreriencing this bug in my lab
omartsyniuk@tpi86:~$ dos.py snapshot --snapshot-name=test 6.1-97_Ubuntu_setup
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
libvirt: QEMU Driver error : operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'
Traceback (most recent call last):
  File "/usr/bin/dos.py", line 22, in <module>
    Shell().execute()
  File "/usr/lib/python2.7/dist-packages/devops/shell.py", line 30, in execute
    self.commands.get(self.params.command)(self)
  File "/usr/lib/python2.7/dist-packages/devops/shell.py", line 71, in do_snapshot
    self.params.snapshot_name)
  File "/usr/lib/python2.7/dist-packages/devops/models.py", line 138, in snapshot
    node.snapshot(name=name, description=description, force=force)
  File "/usr/lib/python2.7/dist-packages/devops/models.py", line 356, in snapshot
    node=self, name=name, description=description)
  File "/usr/lib/python2.7/dist-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/devops/driver/libvirt/libvirt_driver.py", line 365, in node_create_snapshot
    domain.snapshotCreateXML(xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2290, in snapshotCreateXML
    if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
libvirt.libvirtError: operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'

[Oleksandr Martsyniuk (omartsyniuk)]

virsh # snapshot-create 6.1-97_Ubuntu_setup_slave-09
error: operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'

[Tatyanka (tatyana-leontovich)]

Actually I faced with this very often in the last time. And the same errors happens when I try to create snapshot manually. So I see next things, this issue happens when some I/O error happens on gues os like read(23, "d\277.\357\rGd\260\265a\322u\10\0E\0\5\334\220\353@\0?\6\220Q\nm\0\2\nm"..., 69632) = 1514
read(23, "d\277.\357\rGd\260\265a\322u\10\0E\0\0004\220\354@\0?\6\225\370\nm\0\2\nm"..., 69632) = 66
read(23, "d\277.\357\rGd\260\265a\322u\10\0E\0\5\334\220\354@\0?\6\220P\nm\0\2\nm"..., 69632) = 1514
read(23, "d\277.\357\rGd\260\265a\322u\10\0E\0\5\334\220\355@\0?\6\220O\nm\0\2\nm"..., 69632) = 1514
read(23, "d\277.\357\rGd\260\265a\322u\10\0E\0\0004\220\356@\0?\6\225\366\nm\0\2\nm"..., 69632) = 66
read(23, "d\277.\357\rGd\260\265a\322u\10\0E\0\5\334\220\356@\0?\6\220N\nm\0\2\nm"..., 69632) = 1514
read(23, "d\277.\357\rGd\260\265a\322u\10\0E\0\5\334\220\357@\0?\6\220M\nm\0\2\nm"..., 69632) = 1514
read(23, "d\277.\357\rGd\260\265a\322u\10\0E\0\0004\220\360@\0?\6\225\364\nm\0\2\nm"..., 69632) = 66
read(23, "d\277.\357\rGd\260\265a\322u\10\0E\0\5\224\220\360@\0?\6\220\224\nm\0\2\nm"..., 69632) = 1442
read(23, 0x7f3adf8d47ec, 69632) = -1 EAGAIN (Resource temporarily unavailable)

after this guest become paused and we try to create snapshot(and fail)
I am still do not know what was happens with i/o . So you can reproduce the issue by:
- start vm - show i/o error in guest - create snapshot and receive exactly the Error while creating snapshot on 'drive-virtio-disk2'

[Oleksiy Molchanov (omolchanov)]

Moved to invalid, as this issue was not updated for more than 3 weeks.

[Dennis Dmitriev (ddmitriev)]

Reproduced on CI: http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.ha_neutron_tun/9/:

======================================================================
ERROR: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_tun/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_tun/fuelweb_test/tests/test_ceph.py", line 499, in ceph_rados_gw
    self.env.make_snapshot("ceph_rados_gw")
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.ha_neutron_tun/fuelweb_test/models/environment.py", line 238, in make_snapshot
    self.d_env.snapshot(snapshot_name, force=True)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 150, in snapshot
    node.snapshot(name=name, description=description, force=force)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/node.py", line 142, in snapshot
    node=self, name=name, description=description)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/driver/libvirt/libvirt_driver.py", line 401, in node_create_snapshot
    domain.snapshotCreateXML(xml, 0)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/libvirt.py", line 2292, in snapshotCreateXML
    if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
libvirtError: operation failed: Failed to take snapshot: Error while creating snapshot on 'drive-virtio-disk1'
Error while creating snapshot on 'drive-virtio-disk2'

[Dennis Dmitriev (ddmitriev)]

Marking this bug as a dublicate of https://bugs.launchpad.net/fuel/+bug/1484080, so the issue will be handled there.