Dear James, i scratched the hypervisor and redeployed this morning using: nova-lxd 14.0.5 active 1 nova-compute jujucharms 269 ubuntu lxd 2.0.9 active 1 lxd jujucharms 11 ubuntu nova-compute-lxd: Installed: 14.2.2-0ubuntu0.16.10.2~cloud0 Candidate: 14.2.2-0ubuntu0.16.10.2~cloud0 Version table: *** 14.2.2-0ubuntu0.16.10.2~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton/main amd64 Packages 100 /var/lib/dpkg/status 13.3.0-0ubuntu1 500 500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages 13.2.0-0ubuntu1.16.04.1 500 500 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages 13.0.0-0ubuntu3 500 500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages I changed from ZFS to LVM backend. juju deploy nova-compute nova-lxd --to=spare002 juju config nova-lxd virt-type=lxd juju config nova-lxd openstack-origin=cloud:xenial-newton enable-live-migration=true enable-resize=true migration-auth-type=ssh juju deploy lxd juju config lxd block-devices=/dev/sdb storage-type=lvm overwrite=true juju add-relation lxd nova-lxd juju add-relation nova-lxd rabbitmq-server juju add-relation nova-lxd glance juju add-relation nova-lxd neutron-openvswitch juju add-relation ceph-mon nova-lxd juju add-relation ntp nova-lxd juju add-relation nova-cloud-controller nova-lxd juju add-relation ceilometer-agent nova-lxd when i try to deploy from horizon (no volume creation) 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [req-ada8fb6a-9153-4c4f-a44f-d3e438ed5b9d 87ea33bede84421a94de0c9bb7bb49bb d554493bcaea40d5a16a3a7e7d14e590 - - -] [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Instance failed to spawn 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Traceback (most recent call last): 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2083, in _build_resources 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] yield resources 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1924, in _build_and_run_instance 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] block_device_info=block_device_info) 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 317, in spawn 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] self._add_ephemeral(block_device_info, lxd_config, instance) 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 1061, in _add_ephemeral 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] utils.execute(*cmd, run_as_root=True, attempts=3) 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 295, in execute 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] return RootwrapProcessHelper().execute(*cmd, **kwargs) 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 178, in execute 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] return processutils.execute(*cmd, **kwargs) 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] cmd=sanitized_cmd) 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] ProcessExecutionError: Unexpected error while running command. 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf lvcreate -L 10G -n instance-0000039f-ephemeral0 lxd_vg 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Exit code: 5 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Stdout: u'' 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Stderr: u' Volume group "lxd_vg" has insufficient free space (0 extents): 2560 required.\n' 2017-06-19 07:47:40.528 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] 2017-06-19 07:47:41.341 21769 WARNING nova.compute.manager [req-ada8fb6a-9153-4c4f-a44f-d3e438ed5b9d 87ea33bede84421a94de0c9bb7bb49bb d554493bcaea40d5a16a3a7e7d14e590 - - -] Could not clean up failed build, not rescheduling. Error: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf umount /dev/lxd_vg/instance-0000039f-ephemeral0 Exit code: 32 Stdout: u'' Stderr: u'umount: /dev/lxd_vg/instance-0000039f-ephemeral0: mountpoint not found\n' 2017-06-19 07:47:41.655 21769 WARNING nova.scheduler.client.report [req-ada8fb6a-9153-4c4f-a44f-d3e438ed5b9d 87ea33bede84421a94de0c9bb7bb49bb d554493bcaea40d5a16a3a7e7d14e590 - - -] No authentication information found for placement API. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading. 2017-06-19 07:47:41.656 21769 WARNING nova.scheduler.client.report [req-ada8fb6a-9153-4c4f-a44f-d3e438ed5b9d 87ea33bede84421a94de0c9bb7bb49bb d554493bcaea40d5a16a3a7e7d14e590 - - -] Unable to refresh my resource provider record 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [req-ada8fb6a-9153-4c4f-a44f-d3e438ed5b9d 87ea33bede84421a94de0c9bb7bb49bb d554493bcaea40d5a16a3a7e7d14e590 - - -] [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Build of instance 29ce4c98-77eb-4675-9150-1d58e228e143 aborted: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf lvcreate -L 10G -n instance-0000039f-ephemeral0 lxd_vg Exit code: 5 Stdout: u'' Stderr: u' Volume group "lxd_vg" has insufficient free space (0 extents): 2560 required.\n' 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Traceback (most recent call last): 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1783, in _do_build_and_run_instance 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] filter_properties) 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1943, in _build_and_run_instance 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] 'create.error', fault=e) 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] self.force_reraise() 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] six.reraise(self.type_, self.value, self.tb) 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1927, in _build_and_run_instance 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] instance=instance) 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] self.gen.throw(type, value, traceback) 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2110, in _build_resources 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] reason=six.text_type(exc)) 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] BuildAbortException: Build of instance 29ce4c98-77eb-4675-9150-1d58e228e143 aborted: Unexpected error while running command. 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf lvcreate -L 10G -n instance-0000039f-ephemeral0 lxd_vg 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Exit code: 5 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Stdout: u'' 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] Stderr: u' Volume group "lxd_vg" has insufficient free space (0 extents): 2560 required.\n' 2017-06-19 07:47:41.658 21769 ERROR nova.compute.manager [instance: 29ce4c98-77eb-4675-9150-1d58e228e143] root@spare002:/var/log/nova# pvs PV VG Fmt Attr PSize PFree /dev/sda1 vgroot lvm2 a-- 557.85g 0 /dev/sdb lxd_vg lvm2 a-- 557.86g 0 root@spare002:/var/log/nova# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert 23c7ab49074c068634e924a16f589bfca5603c960fd0a1ef43f4a2764df9819d lxd_vg Vwi-a-tz-- 10.00g LXDPool 8.73 LXDPool lxd_vg twi-aotz-- 555.86g 0.21 0.28 a97c1128127a861789b78069dfdcd4615dcd59f91d320ccbc0848d18e943af23 lxd_vg Vwi-a-tz-- 10.00g LXDPool 2.87 lvroot vgroot -wi-ao---- 557.85g