instance destroy needs better guards for zfs ephemeral storage volumes

[James Page]

If instance creation fails for any reason, then its tricky to delete instances on a ZFS storage backend with ephemeral storage volumes; the destroy call does not check whether the volume has actually been created, resulting in a stacktrace and failure to delete the volume:

2017-04-11 07:58:07.400 26969 DEBUG oslo_concurrency.processutils [req-b18af233-416b-4567-8b33-6875a6800bcb 9a68aab1a4764787acd6df2fc3cb8dcd 96c08cea38fb4daeb74f021e17f3c04f - - -] u'sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy lxd/instance-0000086b-ephemeral' failed. Not Retrying. execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:433
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [req-b18af233-416b-4567-8b33-6875a6800bcb 9a68aab1a4764787acd6df2fc3cb8dcd 96c08cea38fb4daeb74f021e17f3c04f - - -] [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Instance failed to spawn
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Traceback (most recent call last):
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2125, in _build_resources
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] yield resources
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] block_device_info=block_device_info)
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 392, in spawn
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] context, instance, network_info, block_device_info)
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 482, in destroy
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] context, instance, network_info, block_device_info)
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 496, in cleanup
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] storage.detach_ephemeral(block_device_info, lxd_config, instance)
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/storage.py", line 109, in detach_ephemeral
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] run_as_root=True)
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 297, in execute
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] return RootwrapProcessHelper().execute(*cmd, **kwargs)
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 180, in execute
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] return processutils.execute(*cmd, **kwargs)
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 400, in execute
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] cmd=sanitized_cmd)
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] ProcessExecutionError: Unexpected error while running command.
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy lxd/instance-0000086b-ephemeral
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Exit code: 1
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Stdout: u''
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686] Stderr: u"cannot open 'lxd/instance-0000086b-ephemeral': dataset does not exist\n"
2017-04-11 07:58:07.401 26969 ERROR nova.compute.manager [instance: 3613ef3f-5920-4830-aebf-5bf22a471686]

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: python-nova-lxd 15.0.0-0ubuntu1~cloud0 [modified: usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py usr/lib/python2.7/dist-packages/nova/virt/lxd/storage.py] [origin: Canonical]
ProcVersionSignature: Ubuntu 4.4.0-72.93-generic 4.4.49
Uname: Linux 4.4.0-72-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Tue Apr 11 10:18:33 2017
PackageArchitecture: all
ProcEnviron:
 TERM=screen-256color-bce
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: nova-lxd
UpgradeStatus: No upgrade log present (probably fresh install)