XenAPIVMTestCase.test_instance_snapshot_fails_with_no_primary_vdi sometimes fails

[Christopher Yeoh]

Sometimes XenAPIVMTestCase.test_instance_snapshot_fails_with_no_primary_vdi fails. It always seems to succeed on its own, but ocassionally fails on a full test run.

======================================================================
FAIL: nova.tests.test_xenapi.XenAPIVMTestCase.test_instance_snapshot_fails_with_no_primary_vdi
tags: worker-3
----------------------------------------------------------------------
Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
Loading network driver 'nova.network.linux_net'
Loading network driver 'nova.network.linux_net'
Fast cloning is only supported on default local SR of type ext. SR on this system was found to be of type lvm. Ignoring the cow flag.
No agent build found for xen/linux/x86-64
Instance agent version: 1.0
}}}

Traceback (most recent call last):
  File "nova/tests/test_xenapi.py", line 501, in test_instance_snapshot_fails_with_no_primary_vdi
    lambda *args, **kwargs: None)
MismatchError: <bound method XenAPIDriver.snapshot of <nova.virt.xenapi.driver.XenAPIDriver object at 0x9f7b610>> returned None
======================================================================

[Matt Riedemann (mriedem)]

I sent this email to John Garbutt last night about this issue:

Hey John,

I'm working on this review:

https://review.openstack.org/#/c/29005/

Which started failing today after rebasing my topic branch onto master once this commit was merged:

https://github.com/openstack/nova/commit/72e75dbcea66e6e4e2b0f5a931dcd647b2032ad6

This is the failure:

mriedem@ubuntu:/opt/stack/nova$ python -m unittest nova.tests.test_xenapi
....................................................................................................................................F.........................................
======================================================================
FAIL: test_instance_snapshot_fails_with_no_primary_vdi (nova.tests.test_xenapi.XenAPIVMTestCase)
nova.tests.test_xenapi.XenAPIVMTestCase.test_instance_snapshot_fails_with_no_primary_vdi
----------------------------------------------------------------------
_StringException: pythonlogging:'': {{{
Loading network driver 'nova.network.linux_net'
Loading network driver 'nova.network.linux_net'
Fast cloning is only supported on default local SR of type ext. SR on this system was found to be of type lvm. Ignoring the cow flag.
No agent build found for xen/linux/x86-64
Instance agent version: 1.0
}}}

Traceback (most recent call last):
  File "nova/tests/test_xenapi.py", line 501, in test_instance_snapshot_fails_with_no_primary_vdi
    lambda *args, **kwargs: None)
MismatchError: <bound method XenAPIDriver.snapshot of <nova.virt.xenapi.driver.XenAPIDriver object at 0xaf61fcc>> returned None

----------------------------------------------------------------------
Ran 174 tests in 19.900s

FAILED (failures=1)

Looking at history, I see the test_instance_snapshot_fails_with_no_primary_vdi test case was changed to pass in the None function with this commit:

https://github.com/openstack/nova/commit/0df60e98790c722aef59d0015c209ea0944e62c0

Looking at the code, vm_utils.snapshot_attached_here checks if the passed in function is None and opts out of using it so I'm not sure how the test case ever passed (by raising an exception). This is the assertion that fails:

        self.assertRaises(exception.NovaException, self.conn.snapshot,
                          self.context, instance, image_id,
                          lambda *args, **kwargs: None)

When I change it to this, it passes but seems to defeat the purpose of the test:

        self.assertIsNone(self.conn.snapshot(self.context, instance, image_id,
                          lambda *args, **kwargs: None))

In IRC it was pointed out that there was another unrelated change with the same failure:

http://logs.openstack.org/29257/3/check/gate-nova-python26/21604/console.html

I'm wondering if you know what's going on here since you were the last to touch the xenapi driver code?

Thanks,

Matt Riedemann

[John Garbutt (johngarbutt)]

I think this was fixed by a change by Rick Haris around stubout and mox interactions, but needs further investigation.

[John Garbutt (johngarbutt)]

This seems to have been fixed by a different change I can't track down, removing from bug list

[Hans Lindgren (hanlind)]

Just happend on the gate at http://logs.openstack.org/12/52312/1/check/gate-nova-python27/571e82b/console.html

A quick search with logstash confirms that this has started to happen frequently over the last couple of days.
Logstash query: @message:"FAIL: nova.tests.virt.xenapi.test_xenapi.XenAPIVMTestCase.test_instance_snapshot_fails_with_no_primary_vdi"

[Soren Hansen (soren)]

https://review.openstack.org/#/c/52094/ will fix this.

[Soren Hansen (soren)]

I believe this is fixed now. Please reopen if this still happens.

[Hans Lindgren (hanlind)]

Checking logstash again after the above change merged verifies that this is now gone.